Data Engineer (1 - 2 years of experience as SDET, Big Data Testing, PySpark)
Is this job for you?
Build my CV Build your CV and see how well you match this role — and every other one.
The role
Develop and maintain PySpark-based test automation scripts for big data applications.
Collaborate with data engineers and developers to understand data processing workflows and requirements.
Design automated tests for data ingestion, processing, and transformations within a Hadoop ecosystem.
Perform data validation, data integrity checks, and performance testing for Spark jobs.
Implement Spark-specific testing using RDDs, DataFrames, Datasets, and Spark SQL.
Set up and maintain CI/CD pipelines for automated testing in a big data environment and ensure high-quality releases.
Collaborate with data engineers and developers to understand data processing workflows and requirements.
Design automated tests for data ingestion, processing, and transformations within a Hadoop ecosystem.
Perform data validation, data integrity checks, and performance testing for Spark jobs.
Implement Spark-specific testing using RDDs, DataFrames, Datasets, and Spark SQL.
Set up and maintain CI/CD pipelines for automated testing in a big data environment and ensure high-quality releases.
See the full job post
Responsibilities, requirements, skills and benefits — create your free account.
or
Already have an account?
Log inSimilar openings
Other roles that could suit you.
? Supply Chain Data Analyst ? Sr. Data Scientist (Advanced SQL & Python + Spark + ML End-to-End Workflow + Power BI/Tableau) ? Data Science, AI and Automation Specialist ? Senior Software Engineer, Open Banking (AI-first team, Java, AWS, GenAI) ? SW Engineer (1-2 years of experience in Full Stack Development - Java, Angular/React, GenAI) ? Staff Engineer GenAI Developer