Lead Data Engineer - (Datawarehouse) - Apache Nifi, Python, PySpark, Hadoop, Cloudera platforms, and Airflow
Is this job for you?
Build my CV Build your CV and see how well you match this role — and every other one.
Track your applications on mobile The free Whileresume app, on iPhone and Android.
The role
Lead the design and delivery of secure, scalable data pipelines and data warehouse solutions in a Big Data environment using PySpark (Python/Scala) on Hadoop or object storage.
Build and optimize end-to-end data pipelines and petabyte-scale data architectures, leveraging Nifi pipelines when available and Airflow for workflow orchestration.
Work across on-premises and cloud platforms (AWS, Azure, Databricks) and manage Hive external tables, partitions, and multiple file formats.
Collaborate in Agile/Scrum teams, create well-defined user stories, perform SQL tuning (Oracle, Netezza), and implement code quality via Git and CI/CD (Jenkins).
Drive automation and efficiency in data ingestion, movement and access workflows; diagnose production issues and implement robust mitigations.
Demonstrate initiative and strong communication while juggling multiple projects in distributed teams and upholding security standards.
Build and optimize end-to-end data pipelines and petabyte-scale data architectures, leveraging Nifi pipelines when available and Airflow for workflow orchestration.
Work across on-premises and cloud platforms (AWS, Azure, Databricks) and manage Hive external tables, partitions, and multiple file formats.
Collaborate in Agile/Scrum teams, create well-defined user stories, perform SQL tuning (Oracle, Netezza), and implement code quality via Git and CI/CD (Jenkins).
Drive automation and efficiency in data ingestion, movement and access workflows; diagnose production issues and implement robust mitigations.
Demonstrate initiative and strong communication while juggling multiple projects in distributed teams and upholding security standards.
See the full job post
Responsibilities, requirements, skills and benefits — create your free account.
or
Already have an account?
Log inYou might also like these jobs
No closely matching jobs yet — here are the most recent ones.