(USA) Principal, Data Scientist - Gen AI Vision
Is this job for you?
Build my CV Build your CV and see how well you match this role — and every other one.
Track your applications on mobile The free Whileresume app, on iPhone and Android.
The role
Own the evaluation architecture for AI agents, designing how success is measured across non-deterministic outputs.
Lead the brain that critiques agents with a mix of LLM-based judging, human benchmarks, and automated pipelines.
Develop evaluation pipelines, calibrate prompts, and distill smaller judge models balancing accuracy, latency, and cost.
Curate large-scale conversation data to build reliable Golden Set datasets and standardized ground truth for subjective tasks.
Collaborate with engineering to integrate quality signals into CI/CD pipelines for automated testing and monitoring.
Drive insights to identify systemic weaknesses, mentor data scientists, and advance best practices in AI evaluation.
Lead the brain that critiques agents with a mix of LLM-based judging, human benchmarks, and automated pipelines.
Develop evaluation pipelines, calibrate prompts, and distill smaller judge models balancing accuracy, latency, and cost.
Curate large-scale conversation data to build reliable Golden Set datasets and standardized ground truth for subjective tasks.
Collaborate with engineering to integrate quality signals into CI/CD pipelines for automated testing and monitoring.
Drive insights to identify systemic weaknesses, mentor data scientists, and advance best practices in AI evaluation.
See the full job post
Responsibilities, requirements, skills and benefits — create your free account.
or
Already have an account?
Log inYou might also like these jobs
No closely matching jobs yet — here are the most recent ones.