(USA) Principal, Data Scientist | Conversational AI

As a Principal Data Scientist for Quality & LLM Judging Systems, you will define and measure the success of conversational AI agents. Design state-of-the-art evaluation architectures using LLM-as-a-judge and hybrid scoring frameworks. Develop high-precision prompts for evaluator models and calibrate them to align with human judgments. Lead model distillation and optimization to create scalable, cost-efficient judge models with acceptable latency. Curate Golden Set datasets and write annotation instructions to standardize ground truth for subjective tasks, plus conduct failure mode analyses. Collaborate with engineering and product teams to integrate quality signals into CI/CD, enable automated regression testing and production monitoring, and influence cross-functional improvements with data-driven insights, while mentoring peers.