Success criteria definition
Define what "good enough to proceed" means for each phase in quantifiable terms: precision, recall, F1, BLEU score, hallucination rate, human preference rating, or business-metric proxy.
Method
Five-step evaluation framework design
Evaluation-first milestones shift the conversation from "are we on schedule?" to "does the model meet the bar required to proceed?" This requires defining that bar before development starts — not after the first demo.
Benchmark task design
Create a representative evaluation set drawn from real production inputs. Tasks must reflect the actual distribution of the use case — not cherry-picked examples that flatter early results.
Evaluation protocol specification
Define who evaluates, how evaluation is conducted, and how disagreements are resolved. Establish whether evaluation is automated, human-reviewed, or a hybrid — with documented rubrics for each.
Phase gate criteria formalization
Document the specific evaluation scores required to pass each milestone gate. Include minimum thresholds, acceptable variance, and conditions under which a phase may be extended rather than failed.
Stakeholder alignment and sign-off
Obtain documented agreement on evaluation criteria from all stakeholders before development begins. This eliminates post-hoc goalposts and scope creep driven by subjective expectations.
Outputs
Artifacts produced by the process
Evaluation criteria document
Formal specification of success criteria for each phase, agreed before development begins.
- Metrics and minimum thresholds per phase
- Evaluation method and scoring rubric
- Stakeholder sign-off record
Benchmark task set
Representative evaluation dataset drawn from production inputs with documented sampling methodology.
- Input distribution and coverage analysis
- Edge case and failure mode representation
- Ground truth labeling and review log
Phase gate scorecard
Structured template for recording evaluation results at each milestone gate.
- Score vs. threshold comparison
- Pass/fail determination with notes
- Extension criteria and conditions
Evaluation run history
Audit trail of all evaluation runs with scores, configurations, and decisions recorded.
- Model version and configuration per run
- Score progression across iterations
- Decision rationale for gate outcomes
Engagement Cadence
How the process runs in practice
Typical timeline: 1-2 weeks (setup); ongoing per phase
- Days 1–4: success criteria definition and benchmark task design
- Days 5–8: evaluation protocol specification and phase gate criteria formalization
- Days 9–10: stakeholder alignment, sign-off, and evaluation infrastructure setup
Output: an evaluation framework that governs phase advancement objectively, eliminating calendar-driven decisions and protecting against premature production deployment.