01
A successful demonstration proves very little
A prototype can show that a model produces a compelling answer under curated conditions. Production must handle incomplete data, adversarial input, provider failure, changing behavior, access control, cost, latency, and human adoption.
Architecture connects the probabilistic capability to a reliable operating system.
02
Evaluation is release infrastructure
Production teams need representative test cases, failure categories, acceptance thresholds, human review samples, and comparison across prompt, retrieval, model, and policy changes.
Model quality is only one measure. The system must also track task success, escalation, user correction, latency, cost, and downstream impact.
03
Design the operating model with the software
Someone must own exceptions, review drift, approve changes, investigate incidents, manage provider and cost decisions, and communicate limits to users.
Observability, security, fallback behavior, documentation, and change control are product capabilities—not launch cleanup.
- Identity and authorization
- Data handling and retention
- Evaluation and production feedback
- Model and provider fallback
- Cost attribution and budgets
- Human escalation and incident response
Decision guide
Questions to carry into the work
- Define production success in business and user terms.
- Build evaluation before scaling exposure.
- Capture enough evidence to investigate decisions.
- Specify degraded and no-model behavior.
- Assign operational ownership and change authority.