AI Engineer, Product
仕事内容
Contribute to the product team's LLM evaluation framework by designing reference tests, heuristics, and model-graded checks.
Define and monitor metrics such as task success, helpfulness, safety flags, latency, and cost to drive data-driven decisions.
Plan and execute A/B tests for prompts, models, and system prompts, with clear rollout or rollback recommendations.
Implement end-to-end observability for LLM calls, including structured logging, tracing, dashboards, and alerts.
Manage model release processes with canary/shadow traffic, sign-offs, SLO-based rollback criteria, and regression detection.
Improve core behaviors (memory policies, intent classification, follow-ups, routing, and tool-call reliability) and create reusable eval templates for teams.
Define and monitor metrics such as task success, helpfulness, safety flags, latency, and cost to drive data-driven decisions.
Plan and execute A/B tests for prompts, models, and system prompts, with clear rollout or rollback recommendations.
Implement end-to-end observability for LLM calls, including structured logging, tracing, dashboards, and alerts.
Manage model release processes with canary/shadow traffic, sign-offs, SLO-based rollback criteria, and regression detection.
Improve core behaviors (memory policies, intent classification, follow-ups, routing, and tool-call reliability) and create reusable eval templates for teams.
求人の全文を見る
業務内容、求める人物像、スキル、待遇 — 無料アカウントの作成で閲覧できます。
または
すでにアカウントをお持ちですか？
ログイン類似の求人
あなたに合いそうな他の職種。