Data Engineer (1 - 2 years of experience in Big Data, Spark, Hadoop, Shell scripting)
Is this job for you?
Build my CV Build your CV and see how well you match this role — and every other one.
The role
Design, develop, and maintain automated test frameworks and scripts for large-scale Big Data applications using PySpark.
Collaborate with data engineering, development, and DevOps teams to understand data pipelines, transformation logic, and CI/CD integration.
Build robust tests to validate data ingestion, processing, and transformation in Hadoop and Spark ecosystems; perform data validation and data integrity checks.
Apply deep Spark knowledge (RDDs, DataFrames, Datasets, Spark SQL) to create reusable test automation components and optimize performance.
Leverage Shell scripting and Unix tools for automation, log analysis, environment setup, and troubleshooting.
Maintain documentation, conduct manual and automated testing for critical financial data, and collaborate across time zones to meet SLAs.
Collaborate with data engineering, development, and DevOps teams to understand data pipelines, transformation logic, and CI/CD integration.
Build robust tests to validate data ingestion, processing, and transformation in Hadoop and Spark ecosystems; perform data validation and data integrity checks.
Apply deep Spark knowledge (RDDs, DataFrames, Datasets, Spark SQL) to create reusable test automation components and optimize performance.
Leverage Shell scripting and Unix tools for automation, log analysis, environment setup, and troubleshooting.
Maintain documentation, conduct manual and automated testing for critical financial data, and collaborate across time zones to meet SLAs.
See the full job post
Responsibilities, requirements, skills and benefits — create your free account.
or
Already have an account?
Log inSimilar openings
Other roles that could suit you.
? Supply Chain Data Analyst ? Sr. Data Scientist (Advanced SQL & Python + Spark + ML End-to-End Workflow + Power BI/Tableau) ? Data Science, AI and Automation Specialist ? Data Engineer (1 - 2 years of experience as SDET, Big Data Testing, PySpark) ? Senior Software Engineer, Open Banking (AI-first team, Java, AWS, GenAI) ? SW Engineer (1-2 years of experience in Full Stack Development - Java, Angular/React, GenAI)