(USA) Principal, Data Scientist - Gen AI Vision
Questa offerta fa per te?
Crea il mio CV Crea il tuo CV e scopri la tua percentuale di corrispondenza con questa posizione — e con tutte le altre.
La posizione
Own the evaluation architecture for AI agents, designing how success is measured across non-deterministic outputs.
Lead the brain that critiques agents with a mix of LLM-based judging, human benchmarks, and automated pipelines.
Develop evaluation pipelines, calibrate prompts, and distill smaller judge models balancing accuracy, latency, and cost.
Curate large-scale conversation data to build reliable Golden Set datasets and standardized ground truth for subjective tasks.
Collaborate with engineering to integrate quality signals into CI/CD pipelines for automated testing and monitoring.
Drive insights to identify systemic weaknesses, mentor data scientists, and advance best practices in AI evaluation.
Lead the brain that critiques agents with a mix of LLM-based judging, human benchmarks, and automated pipelines.
Develop evaluation pipelines, calibrate prompts, and distill smaller judge models balancing accuracy, latency, and cost.
Curate large-scale conversation data to build reliable Golden Set datasets and standardized ground truth for subjective tasks.
Collaborate with engineering to integrate quality signals into CI/CD pipelines for automated testing and monitoring.
Drive insights to identify systemic weaknesses, mentor data scientists, and advance best practices in AI evaluation.
Vedi l'annuncio completo
Mansioni, profilo, competenze e vantaggi — crea il tuo account gratuito.
Hai già un account? Accedi
Offerte simili
Altre posizioni che potrebbero interessarti.