Data Engineer (1 - 2 years of experience in Big Data, Spark, Hadoop, Shell scripting)
仕事内容
Design, develop, and maintain automated test frameworks and scripts for large-scale Big Data applications using PySpark.
Collaborate with data engineering, development, and DevOps teams to understand data pipelines, transformation logic, and CI/CD integration.
Build robust tests to validate data ingestion, processing, and transformation in Hadoop and Spark ecosystems; perform data validation and data integrity checks.
Apply deep Spark knowledge (RDDs, DataFrames, Datasets, Spark SQL) to create reusable test automation components and optimize performance.
Leverage Shell scripting and Unix tools for automation, log analysis, environment setup, and troubleshooting.
Maintain documentation, conduct manual and automated testing for critical financial data, and collaborate across time zones to meet SLAs.
Collaborate with data engineering, development, and DevOps teams to understand data pipelines, transformation logic, and CI/CD integration.
Build robust tests to validate data ingestion, processing, and transformation in Hadoop and Spark ecosystems; perform data validation and data integrity checks.
Apply deep Spark knowledge (RDDs, DataFrames, Datasets, Spark SQL) to create reusable test automation components and optimize performance.
Leverage Shell scripting and Unix tools for automation, log analysis, environment setup, and troubleshooting.
Maintain documentation, conduct manual and automated testing for critical financial data, and collaborate across time zones to meet SLAs.
求人の全文を見る
業務内容、求める人物像、スキル、待遇 — 無料アカウントの作成で閲覧できます。
または
すでにアカウントをお持ちですか？
ログイン類似の求人
あなたに合いそうな他の職種。
? Supply Chain Data Analyst ? Sr. Data Scientist (Advanced SQL & Python + Spark + ML End-to-End Workflow + Power BI/Tableau) ? Data Science, AI and Automation Specialist ? Data Engineer (1 - 2 years of experience as SDET, Big Data Testing, PySpark) ? Senior Software Engineer, Open Banking (AI-first team, Java, AWS, GenAI) ? SW Engineer (1-2 years of experience in Full Stack Development - Java, Angular/React, GenAI)