AI Evaluation Engineer

Mindrift seeks an AI Evaluation Engineer with over five years in software development to create and refine tasks for evaluating AI coding agents in real-world scenarios. The role involves designing developer environments, crafting tasks, and writing tests that assess AI solutions, ensuring they handle multiple valid approaches. Candidates should have experience with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis, along with functional and integration testing abilities. The position requires English proficiency at B2+ level and offers a project-based structure with flexible scheduling. Responsibilities include reviewing AI agent solutions, analysing failures, and improving evaluation frameworks without developing code from scratch. The work environment is specifically geared towards testing frontier models' capabilities and limitations.