Senior Data Engineer - Product
Is this job for you?
Build my CV Build your CV and see how well you match this role — and every other one.
Track your applications on mobile The free Whileresume app, on iPhone and Android.
The role
Re-architect and scale the DSF-based big data processing components powering production risk analytics.
Analyze workload patterns across Spark jobs, notebooks, and DS API usage to drive performance, reliability, and cost improvements.
Ensure stability of Spark jobs running on EMR or Kubernetes and operate Hadoop ecosystem components (HDFS, YARN) and Spark runtimes.
Maintain and evolve ingestion pipelines between Runtime and DSF (Firehose, Glue → S3) and improve Spark runtimes.
Enhance developer and power-user experience across JupyterLabs and the DS API.
Own services throughout their lifecycle following DevOps practices, collaborating with data scientists and platform teams to deliver a reliable, scalable data platform.
Analyze workload patterns across Spark jobs, notebooks, and DS API usage to drive performance, reliability, and cost improvements.
Ensure stability of Spark jobs running on EMR or Kubernetes and operate Hadoop ecosystem components (HDFS, YARN) and Spark runtimes.
Maintain and evolve ingestion pipelines between Runtime and DSF (Firehose, Glue → S3) and improve Spark runtimes.
Enhance developer and power-user experience across JupyterLabs and the DS API.
Own services throughout their lifecycle following DevOps practices, collaborating with data scientists and platform teams to deliver a reliable, scalable data platform.
See the full job post
Responsibilities, requirements, skills and benefits — create your free account.
or
Already have an account?
Log inYou might also like these jobs
No closely matching jobs yet — here are the most recent ones.