Measure and improve · Add-on
Monitor AI quality in production
AI quality can change after launch as users, data and models change. Monitor quality, cost and safety continuously, and act quickly when they slip.
About 2-4 daysFast-movingLast reviewed 5 November 2026
Who this applies to
- Skills it builds
- Data and measurementTechnical and AI fluency
- PM types
- AI
- Product types
- Data or AI product
- Starting points
- All
- Ways of working
- All
- Experience
- All
- Size of organisation
- All
Steps, by experience
Beginner (under 1 year)
- Pick a few measures: answer quality, user corrections, complaints and cost per use.
- Sample real outputs every week and check them by hand.
- Re-run your evaluation set whenever the model or prompts change.
- Set alerts for sudden changes in quality or cost.
- Have a way to roll back to a previous version.
Intermediate (1 to 4 years)
- Track quality, safety and cost measures.
- Review samples and re-run evaluations on change.
- Alert on drift and keep rollback ready.
Experienced (5+ years)
- Automate evaluation on live samples.
- Feed failures back into the evaluation set.
- Report AI quality and cost to leaders like any other product health measure.
Who can do this
Owner: Product manager
Can be delegated to
- Data Scientist / ML Engineer or Backend Engineer: Set up monitoring and evaluation
- QA / Tester: Review samples of live outputs
Reviewed by: Data Scientist / ML Engineer, Tech Lead
Tool examples
- Claude
- Use a grading prompt to score samples of outputs against your criteria, and check a share by hand.
- Looker
- Chart quality and cost per use over time.
Definition of done
- Quality, safety and cost measures tracked
- Weekly sample review
- Evaluations re-run on change
- Rollback ready
Common pitfalls
- Assuming launch quality lasts
- Only watching cost
- Changing models without re-testing
If you lead the team
What good looks like: AI quality, cost and safety are monitored continuously after launch.
Ask your PM
- How do we know if AI quality drops, and who is alerted?
- What's our cost per user, and how is it changing?
Warning signs
- AI quality is only checked when users complain.
- Costs grow unnoticed.