Senior Data Engineer - Product
Is this job for you?
Build my CV Build your CV and see how well you match this role — and every other one.
The role
Re-architect and scale the DSF-based big data processing components powering production risk analytics.
Analyze workload patterns across Spark jobs, notebooks, and DS API usage to drive performance, reliability, and cost improvements.
Ensure stability of Spark jobs running on EMR or Kubernetes and operate Hadoop ecosystem components (HDFS, YARN) and Spark runtimes.
Maintain and evolve ingestion pipelines between Runtime and DSF (Firehose, Glue → S3) and improve Spark runtimes.
Enhance developer and power-user experience across JupyterLabs and the DS API.
Own services throughout their lifecycle following DevOps practices, collaborating with data scientists and platform teams to deliver a reliable, scalable data platform.
Analyze workload patterns across Spark jobs, notebooks, and DS API usage to drive performance, reliability, and cost improvements.
Ensure stability of Spark jobs running on EMR or Kubernetes and operate Hadoop ecosystem components (HDFS, YARN) and Spark runtimes.
Maintain and evolve ingestion pipelines between Runtime and DSF (Firehose, Glue → S3) and improve Spark runtimes.
Enhance developer and power-user experience across JupyterLabs and the DS API.
Own services throughout their lifecycle following DevOps practices, collaborating with data scientists and platform teams to deliver a reliable, scalable data platform.
See the full job post
Responsibilities, requirements, skills and benefits — create your free account.
or
Already have an account?
Log inSimilar openings
Other roles that could suit you.