Data Engineer (1 - 2 years of experience as SDET, Big Data Testing, PySpark)
仕事内容
Develop and maintain PySpark-based test automation scripts for big data applications.
Collaborate with data engineers and developers to understand data processing workflows and requirements.
Design automated tests for data ingestion, processing, and transformations within a Hadoop ecosystem.
Perform data validation, data integrity checks, and performance testing for Spark jobs.
Implement Spark-specific testing using RDDs, DataFrames, Datasets, and Spark SQL.
Set up and maintain CI/CD pipelines for automated testing in a big data environment and ensure high-quality releases.
Collaborate with data engineers and developers to understand data processing workflows and requirements.
Design automated tests for data ingestion, processing, and transformations within a Hadoop ecosystem.
Perform data validation, data integrity checks, and performance testing for Spark jobs.
Implement Spark-specific testing using RDDs, DataFrames, Datasets, and Spark SQL.
Set up and maintain CI/CD pipelines for automated testing in a big data environment and ensure high-quality releases.
求人の全文を見る
業務内容、求める人物像、スキル、待遇 — 無料アカウントの作成で閲覧できます。
または
すでにアカウントをお持ちですか？
ログイン類似の求人
あなたに合いそうな他の職種。
? Supply Chain Data Analyst ? Sr. Data Scientist (Advanced SQL & Python + Spark + ML End-to-End Workflow + Power BI/Tableau) ? Data Science, AI and Automation Specialist ? Senior Software Engineer, Open Banking (AI-first team, Java, AWS, GenAI) ? SW Engineer (1-2 years of experience in Full Stack Development - Java, Angular/React, GenAI) ? Staff Engineer GenAI Developer