Skip to main content
AI evaluations & LLMOps

Know whether it works before your users do.

For teams running an AI feature they cannot currently measure — where a prompt change ships on someone's judgement, and nobody can say whether last week's release made the output better or worse.

Bring the workflow, the outputs people already dispute, and the decision the output feeds.

What you can commission

Measure the task, not the model.

Work starts from the job the AI feature actually performs and the decision its output feeds. A general benchmark score says nothing about whether your workflow is safe to run.

01 · Datasets

Golden datasets and task definition

Define the task precisely enough to be gradeable, then build a versioned evaluation set from real cases: the ordinary ones, the edge cases that caused past incidents, and the adversarial ones that should be refused. Each item carries an expected output or a rubric, and the dataset is versioned so a score is always attributable to a specific set.

  • You receive a versioned dataset with documented provenance and coverage, the task definition it grades, and the rule for adding a case when production produces a new failure.
02 · Grading

Human review and automated graders

The two are different instruments and are treated separately. Human review uses a written rubric with more than one reviewer on a sample, and agreement between reviewers is measured — a rubric two people apply differently is not yet a rubric. Automated graders (exact match, structural checks, retrieval-grounding checks, or a model-as-judge) are calibrated against those human labels, and their disagreement rate is reported rather than assumed away.

  • You receive a review rubric, an inter-reviewer agreement figure, the automated grader set, and the measured agreement between each grader and human judgement.
03 · Metrics

Task metrics and regression gates

Choose metrics the task justifies — extraction accuracy per field, grounding and citation correctness, refusal correctness, tool-call validity, resolution rate — and define each one in writing so two people compute it the same way. Those metrics become a gate in the delivery pipeline: a prompt, model or retrieval change runs the suite and a regression beyond the agreed threshold blocks the release.

  • You receive defined metrics with their formulas and thresholds, a runnable evaluation suite, and a pipeline gate that fails a release on regression rather than reporting it afterwards.
04 · Adversarial

Red teaming

Probe the workflow deliberately: prompt injection through retrieved documents and tool output, instruction override, data exfiltration through the model's context, jailbreaks against refusal behaviour, and tool misuse where an agent has permissions it can be talked into using. Findings are reproducible cases added to the regression set, not anecdotes.

  • You receive reproducible adversarial cases with severity and the control that mitigates each, folded into the regression suite so a fix cannot silently regress.
05 · Operations

Tracing, versioning, latency and cost

Instrument the workflow so a single output can be reconstructed: the prompt version, model and parameters, retrieved context, tool calls and the intermediate steps. Version prompts, models, retrieval configuration and datasets together, so a quality change can be attributed to what actually changed. Track latency at the percentiles users experience and cost per task, because a quality gain that triples cost or breaks a response-time budget is a trade-off someone must decide, not a win.

  • You receive Trace instrumentation, a versioning scheme covering prompt, model, retrieval and dataset, and latency-percentile and cost-per-task reporting alongside quality.
06 · Feedback

Production feedback loop

Capture what production tells you: explicit user signals, implicit ones such as edits, retries and escalations, and the cases a human corrected. Route them into triage, and promote confirmed failures into the evaluation set so the suite reflects the system's real failure surface rather than the one imagined at design time.

  • You receive a feedback capture and triage route with a named owner, and a documented path from a production failure to a permanent regression case.
What evaluation is, and is not

Measurement is not a guarantee.

This distinction decides whether an evaluation programme is useful or misleading, so it is stated before scope rather than in a footnote.

Not a guarantee of accuracy

A score is a sample

An evaluation measures performance on the cases in a dataset. It supports a judgement about inputs that resemble them, and says nothing about inputs unlike anything the set contains. No score here is a warranty of correctness on future inputs, and generative systems can produce a confident, incorrect answer on an input that looks routine.

Not a safety certificate

Red teaming finds, it does not clear

Adversarial testing shows that specific attacks succeed or are mitigated. It cannot show that no attack exists. A clean red-team run is evidence about the attacks that were tried, by the people who tried them, at that time — not a statement that the system is safe.

Not regulatory compliance

Evidence for a reviewer, not a verdict

Evaluation produces evidence your risk, legal or compliance function can use. It is not a compliance assessment, a conformity assessment, a certification, or legal advice on any AI regulation, and Codefremics does not issue those. Where a regime applies, your accountable reviewer decides what the evidence supports.

Example evaluation report

What the output actually looks like.

The structure of a report this engagement produces, shown against a worked example.

  • Synthetic demonstration — not customer evidence.Every element below is fictional and constructed for this page. There is no client, no engagement and no system behind it.
  • Illustrative valuesThe figures are chosen to show the report's shape; they were not produced by running an evaluation, and they are not measurements of any model, product or delivery by Codefremics.
  • ProvenanceAuthored 8 September 2026 as a documentation artifact. Source data: none — inputs are invented.
  • Review statusTechnical review: pending. Publication approval: pending. This example is not evidence that Codefremics has delivered this service.
  1. 1Task definitionFictional scenario. Extract five fields — supplier name, invoice number, currency, total, due date — from a supplier invoice PDF, and refuse rather than guess when a field is absent or illegible. The refusal behaviour is part of the task: a system that always answers cannot be graded on knowing when not to.
  2. 2Dataset and versionFictional. invoice-extraction-v3: 240 invented documents — 160 routine, 50 degraded scans and unusual layouts, 30 with a deliberately missing or unreadable field. Frozen; changes create v4. Two document languages and four supplier templates; handwritten annotations are absent, so the set supports no claim about them.
  3. 3Rubric and expected outputsFictional. Each field is correct, incorrect or correctly refused. Totals must match to the minor unit; dates are normalised before comparison; a confident wrong value on an unreadable field is the most severe failure class. Illustrative review plan: two reviewers would independently grade a 60-item sample. No reviewers performed this demonstration.
  4. 4Metric definitionsFictional definitions, illustrative values. Field accuracy is correct fields over gradeable fields. Refusal correctness is correct refusals over items where refusal was the right answer. Confident-error rate is wrong values given without hedging, over all answered fields. Illustrative, not measured: field accuracy 0.94; refusal correctness 0.71; confident-error rate 0.03.
  5. 5Fictional failure examplesFictional. The weakest area is refusal: on degraded scans the system supplies a plausible due date rather than declining. Two invented adversarial cases show instruction text inside a PDF being followed as a command. Illustrative grader agreement: 91%, an invented value. No automated grading or human labeling was performed.
  6. 6Limitations and recommended actionsFictional. The scenario assumes one model version, one prompt version and a 240-item set without handwritten documents; no run occurred, so the figures support no confidence interval. Recommended: route low-confidence extractions to human review, add the injection cases to the regression gate, re-run per prompt change, and widen the set before any claim about handwritten input.
Scope and boundaries

What is included, and what is not.

Evaluation depends on access to the workflow and to people who can say what a good answer is, so the responsibilities are agreed before delivery.

Prerequisites

Before we start

A named owner for the AI feature; access to a non-production environment and the model or API behind it; representative, lawfully usable cases; and a subject-matter reviewer who can decide what a correct output is.

Your responsibilities

What stays with you

Model and API costs during evaluation; the lawful basis for using real cases as evaluation data, and any redaction it requires; the business decision on acceptable thresholds; and accountability for operating the system and acting on findings.

Exclusions

Not in this engagement

Model training or fine-tuning; certification, conformity assessment or legal advice under any AI regulation; round-the-clock monitoring of your AI system; and any warranty that a model will not produce an incorrect or harmful output.

  • Evidence boundaryThe example above is a synthetic demonstration, labelled as such. This page publishes no customer evaluation, no measured production result and no delivery record.
  • Real artifactsAny real artifact is shown only after sanitisation, technical review and publication approval.
  • Where this fitsStart at Applied AI. Evaluation applies directly to Enterprise RAG & knowledge systems and AI workflow agents.
  • Build or notIf the question is whether to build at all, begin with AI readiness.
Start with one feature

Make the AI feature measurable.

A first assessment defines the task, the evidence that would count and the gate a release must pass. It starts from the outputs your team already disputes.

What to bring

  1. 01The workflowThe AI feature, who uses it and where its output goes.
  2. 02The disputed outputsExamples people already question, sanitised where needed.
  3. 03The decision it feedsWhat happens when the output is wrong, and who is accountable.
WhatsApp us