What you can commission
Measure the task, not the model.
Work starts from the job the AI feature actually performs and the decision its output feeds. A general benchmark score says nothing about whether your workflow is safe to run.
01 · Datasets
Golden datasets and task definition
Define the task precisely enough to be gradeable, then build a versioned evaluation set from real cases: the ordinary ones, the edge cases that caused past incidents, and the adversarial ones that should be refused. Each item carries an expected output or a rubric, and the dataset is versioned so a score is always attributable to a specific set.
- You receive a versioned dataset with documented provenance and coverage, the task definition it grades, and the rule for adding a case when production produces a new failure.
02 · Grading
Human review and automated graders
The two are different instruments and are treated separately. Human review uses a written rubric with more than one reviewer on a sample, and agreement between reviewers is measured — a rubric two people apply differently is not yet a rubric. Automated graders (exact match, structural checks, retrieval-grounding checks, or a model-as-judge) are calibrated against those human labels, and their disagreement rate is reported rather than assumed away.
- You receive a review rubric, an inter-reviewer agreement figure, the automated grader set, and the measured agreement between each grader and human judgement.
03 · Metrics
Task metrics and regression gates
Choose metrics the task justifies — extraction accuracy per field, grounding and citation correctness, refusal correctness, tool-call validity, resolution rate — and define each one in writing so two people compute it the same way. Those metrics become a gate in the delivery pipeline: a prompt, model or retrieval change runs the suite and a regression beyond the agreed threshold blocks the release.
- You receive defined metrics with their formulas and thresholds, a runnable evaluation suite, and a pipeline gate that fails a release on regression rather than reporting it afterwards.
04 · Adversarial
Red teaming
Probe the workflow deliberately: prompt injection through retrieved documents and tool output, instruction override, data exfiltration through the model's context, jailbreaks against refusal behaviour, and tool misuse where an agent has permissions it can be talked into using. Findings are reproducible cases added to the regression set, not anecdotes.
- You receive reproducible adversarial cases with severity and the control that mitigates each, folded into the regression suite so a fix cannot silently regress.
05 · Operations
Tracing, versioning, latency and cost
Instrument the workflow so a single output can be reconstructed: the prompt version, model and parameters, retrieved context, tool calls and the intermediate steps. Version prompts, models, retrieval configuration and datasets together, so a quality change can be attributed to what actually changed. Track latency at the percentiles users experience and cost per task, because a quality gain that triples cost or breaks a response-time budget is a trade-off someone must decide, not a win.
- You receive Trace instrumentation, a versioning scheme covering prompt, model, retrieval and dataset, and latency-percentile and cost-per-task reporting alongside quality.
06 · Feedback
Production feedback loop
Capture what production tells you: explicit user signals, implicit ones such as edits, retries and escalations, and the cases a human corrected. Route them into triage, and promote confirmed failures into the evaluation set so the suite reflects the system's real failure surface rather than the one imagined at design time.
- You receive a feedback capture and triage route with a named owner, and a documented path from a production failure to a permanent regression case.