7. Operating & Looking Forward
An evaluation system that has been validated once has been validated for a single moment. The preceding chapters establish a rubric, a qualified annotator pool, an LLM judge that passes its stress tests, and evidence that the judge agrees with human annotators approximately as well as the annotators agree with one another. From the day the system enters production, however, each of its components begins to change: the questions users ask, the people who annotate, the model behind the judge’s API, and the team’s own definition of a good answer. Change in the underlying model is not hypothetical: Chen, Zaharia, and Zou documented substantial changes in the behavior of the same named commercial models over a period of a few months[1], and a judge built on such a model inherits those changes.
This final chapter concerns the operational layer that surrounds all three alignments of the three-way model: keeping a validated evaluation valid over time. Section 7.1 describes how to determine which component changed when a metric moves. Section 7.2 covers dashboards and launch gates. Section 7.3 describes a staged procedure for tasks that have no labels at all. Section 7.4 addresses how to store labels so that they remain useful after the rubric changes. Section 7.5 closes the handbook with open problems and a summary of its central recommendations.
Running example. The handbook uses Mise, a fictional cooking assistant that answers questions about recipes, techniques, and food safety (Section 1.5). This chapter refers to three of its evaluation tasks:
- the answer-quality task: does the response fully answer the user’s question? Labels: Fully / Partially / Does not answer;
- the food-safety task: does the response contain unsafe food-handling advice? Labels: Safe / Unsafe. Unsafe responses are rare (about 3% of traffic) but severe;
- the pairwise preference task: which of two responses is preferable? Labels: A / B / Tie.
Helpful background. The three-way alignment model (Section 1.4); Cohen’s κ, a chance-corrected agreement statistic (Section 3.2); the judge test suite (Section 5.5); the human ceiling (Section 6.1); selective judging (Section 6.4); and prediction-powered inference (Section 6.5). Each concept is briefly restated where it is used.
Five kinds of drift
Consider a typical incident. In week 8, the share of Mise responses that the judge labels Unsafe rises from 2% to nearly 4%. The natural first hypothesis is that the assistant has become less safe; the second is that the judge has broken. Either may be correct, but neither should be assumed, because the reported number is the product of several components, any of which can change independently. Separating these components is the first step of every investigation. At least five can be distinguished:
| Drift type | What changed | Mise example | How it is detected |
|---|---|---|---|
| Input distribution | What users send | A December increase in holiday-baking and leftovers questions | Topic mix, length, and language shares (no labels required) |
| Concept / target | How inputs relate to the correct label | Mise launches an “air fryer” mode; a correct cooking time now depends on the appliance | Fresh human labels on new traffic; errors concentrate in the new slice |
| Human annotation | How annotators apply the rubric | After several weeks on the task, annotators become lenient and assign more “Fully” labels | Accuracy on hidden gold items; agreement on blind repeated items |
| Judge | The evaluator itself | The provider updates the model snapshot behind the judge’s API alias without notice | Judge accuracy on a fixed anchor set, re-run on a schedule |
| Evaluation standard | What “good” means | The team decides that a full answer must include a safety caveat where relevant | Rubric version history; scores shift exactly at the version change |
The five types are not interchangeable, and each calls for a different response. Input drift may require nothing beyond reweighting the report, since the judge may perform equally well on the new mix. Judge drift calls for pinning or re-validating the judge. Annotator drift calls for retraining or recalibrating the annotators. A change in the evaluation standard means that old and new numbers are no longer comparable, and trend charts should state this explicitly.
Changes to the standard are more common than teams expect, because criteria often evolve once people inspect real outputs. Shankar et al. term this phenomenon criteria drift: in their study, grading actual outputs helped users define their criteria, and those criteria changed as more outputs were graded[2]. Criteria drift is not a failure of process; it is a reason to version the rubric and to plan for its revision.
Leading, lagging, and anchored signals
Distinguishing the five drift types requires signals that respond differently to each. Monitoring signals fall into three categories, and a robust configuration uses all three:
- Leading (unlabeled) signals are available immediately: input topic mix, the distribution of judge labels, score histograms, and the judge’s abstention rate. They indicate that something changed, not whether quality changed.
- Lagging (labeled) signals require fresh human labels: judge precision and recall on a recent random audit, or the rate at which unsafe advice is missed. They indicate whether quality changed, but only after days or weeks.
- Anchored signals re-run the judge (or re-present to annotators) a fixed, frozen set of items with trusted labels. Because the items never change, any movement reflects the instrument rather than the world.
Anchors are the least expensive means of separating “the world changed” from “the measuring instrument changed.” The four simulated dashboards below can be read with that distinction in mind.
When human–judge agreement drops
The scenarios above illustrate why the order of investigation matters. Suppose the weekly audit shows judge–human agreement on the answer-quality task falling from κ = 0.62 to 0.48 (κ is Cohen’s kappa, the proportion of the agreement possible beyond chance that is actually achieved; Section 3.2). A disciplined investigation examines the least expensive and most mechanical explanations first:
- Did the human side change? A new rubric version, a new annotator cohort, an interface update, or a change in who adjudicates. Check the annotators’ accuracy on gold items.
- Did the judge side change? The prompt version, the model snapshot, decoding settings, or a preprocessing step that now truncates long conversations. Re-run the anchor set.
- Did the inputs change? Compare the topic and length mix of the current audit with that of the previous month.
- Is the drop concentrated? Break agreement down by slice. A global drop that is confined to one new topic is a very different problem from a uniform drop.
- Which side is correct? Sample the new disagreements and have an expert adjudicate them, following the triage procedure in Section 6.2, before changing anything.
A frequent failure pattern proceeds as follows: agreement drops, the judge prompt is revised until agreement recovers, and the team later discovers that the annotators were the component that drifted. The judge has then been tuned to match a lenient annotator pool, and both instruments are now biased in the same direction, which no agreement statistic can reveal.
With millions of judged items, almost every week-over-week difference is statistically significant. A 0.3-point shift in the unsafe rate may have a negligible p-value and still be irrelevant to any decision. Alert thresholds are better expressed in units that matter to the decision (for example, “unsafe rate above 2% for two consecutive weeks”), with formal tests reserved for small labeled audits, where sampling noise is the principal risk. The detection of distribution shift has its own literature; Rabanser et al. provide a useful empirical comparison of methods[3].
Detecting drift is only useful if the signals are presented in a form that supports decisions. The next section turns to the dashboard and to the gate that controls changes to the judge itself.
Dashboards and launch gates
A dashboard that reports only the headline judge–human agreement can appear healthy while the quantity that matters most is failing. In the mock dashboard below, overall agreement is close to the human ceiling, yet the judge’s recall on unsafe advice has fallen below its floor. A dashboard for an evaluation system should therefore answer one question at a glance: can the numbers it produces still be trusted? This suggests grouping signals by the component of the system they describe, following the three-way alignment model (Section 1.4), with additional groups for drift and cost.
Three terms on the dashboard are defined elsewhere in the handbook. The human ceiling is the agreement that human annotators achieve with one another; a judge cannot be expected to exceed it meaningfully (Section 6.1). Automated coverage is the share of items for which the judge’s verdict is accepted without human review (Section 6.4). The PPI correction is the adjustment that prediction-powered inference applies to the judge’s aggregate rate, estimated from a random human-labeled sample (Section 6.5).
Placing the human ceiling beside every judge–human agreement figure, and the worst slice beside every average, changes how readers interpret the entire dashboard at the cost of two additional tiles.
A launch gate for a new judge version
The dashboard monitors a judge that is already deployed; a separate control is needed for the moment the judge itself is replaced. Changing the judge (a new prompt, a new model, or new output parsing) changes the instrument with which every downstream number is measured. Such a change warrants the same discipline as a model launch: the acceptance criteria are written before results are examined, and the old and new judges are compared on the same items. A paired comparison removes the variation due to item difficulty, so it can detect smaller differences with fewer labels than a comparison of two independent samples.
How many test cases constitute a pass?
The second criterion in the checklist raises a quantitative question. A statement such as “the judge caught every unsafe example in the test set” appears reassuring until one asks how many unsafe examples the set contained. If zero failures are observed in n independent tests, the one-sided 95% upper confidence bound on the true failure rate is approximately 3/n, a result known as the rule of three[4].
upper bound ≈ 3n (exact: 1 − 0.051/n)
Launch gates and dashboards presuppose that a validated judge and a body of labeled data already exist. The next section addresses the situation in which neither does.
Cold start: a new task with no labels
Suppose Mise introduces the air-fryer mode mentioned in Section 7.1. No rubric covers it, no annotator has been qualified on it, and no labeled examples exist. The same situation arises with a new language or a new type of output. The tempting response is to deploy an LLM judge immediately, because it requires no labels. The judge may ultimately be the appropriate tool, but without a verified subset there is no means of estimating how often it is wrong. A staged approach acquires that knowledge at modest cost, and each stage reuses a procedure described earlier in the handbook.
The role of synthetic data
Cold start creates strong pressure to generate data rather than collect it. LLMs are effective at producing stress cases that a team might not anticipate: a question about reheating rice phrased in five different ways, a question that mixes two languages, or a response that sounds confident while being subtly wrong. Such generation is valuable for coverage. It is considerably less valuable as a source of labels, as the following comparison illustrates.
An LLM writes 500 “unsafe” responses, its own label is retained for each, and a second LLM checks the labels. The judge’s recall on this set is reported as 97%.
- The generator’s notion of “unsafe” defines the label; no human has verified it.
- Generated failures tend to be obvious and stylistically uniform.
- The verifier may share the generator’s blind spots, so their agreement establishes little.
An LLM drafts candidate stress cases from the experts’ failure-mode list, and experts label (and often edit) a sample before any item counts as reference data.
- Synthetic items are tagged as synthetic and reported separately.
- Labels come from the same verified process as labels on real items.
- Audits of real traffic remain the source of rate estimates.
Whether the seed set, pilot labels, and adjudications produced during cold start remain useful later depends on how they are stored, which is the subject of the next section.
Keeping data reusable
A year after collection, a team may need to validate a new judge against last year’s labels, to re-score data after a rubric change, or to explain why a particular boundary ruling was made. The labels most likely to be needed again are also the most expensive: expert adjudications and boundary-case rulings. Whether such reuse is possible depends almost entirely on what was stored alongside each label at the time of collection.
Store the context, not only the answer
A label without provenance cannot be re-interpreted once the circumstances of its collection are forgotten. A reasonable record for one Mise item in the answer-quality task might take the following form:
{
"item_id": "mise-aq-000417",
"input_sha256": "9f2c…e81a", // detects silent edits to the item
"task": "answer_quality",
"rubric_version": "2.3",
"ui_version": "annot-ui-1.8", // the interface is part of the instrument
"sample": { "frame": "random_audit_wk42", "inclusion_prob": 0.0012 },
"annotations": [
{ "annotator": "r-118", "cohort": "2026-07", "label": "PARTIAL",
"rationale": "Crisping advice is fine; ignores the overnight-storage risk.",
"seconds_spent": 74, "submitted_at": "2026-10-12T14:03Z" },
{ "annotator": "r-052", "cohort": "2026-03", "label": "FULL",
"rationale": "Answers both parts.", "seconds_spent": 31,
"submitted_at": "2026-10-12T15:40Z" }
],
"adjudication": { "by": "expert-07", "label": "PARTIAL",
"ruling": "Answering a different situation is not answering.",
"rubric_change": "Added boundary example B-14 in v2.4" },
"judge_outputs": [
{ "judge": "answer-quality-judge", "prompt_version": "p11", "model": "<pinned snapshot>",
"label": "FULL", "evidence": "\"Day-old rice is actually perfect…\"" }
]
}Several fields in this record prove more valuable than their storage cost suggests:
- Individual annotations, not only the final label. Once labels are collapsed to a majority vote, it is no longer possible to recompute agreement, re-weight annotators, or study disagreement.
- Inclusion probability. Without it, one cannot later determine whether an item came from a random audit (usable for rate estimates) or from a targeted challenge set (not usable for rate estimates).
- Rubric and interface versions. When a number shifts, these are the first quantities to check, and they cannot be reconstructed retroactively.
- Adjudication rulings with rationale. These become the boundary examples in the next rubric version and the most difficult items in the next judge test suite.
At the dataset level, a short datasheet (recording who labeled the data, how annotators were recruited and compensated, what was sampled and excluded, and known limitations) answers questions that future users would otherwise have to guess at[5].
When the label set changes
Provenance becomes essential when the label set itself evolves, as it does when the evaluation standard drifts (Section 7.1). Categories may be split, merged, or redefined without a change of name. Each change leaves three options for existing data: remap it mechanically, relabel it, or retire it from comparisons. Which option is safe depends on the kind of change.
The practices described so far address problems for which reasonably settled methods exist. The final section turns to problems for which they do not.
Open challenges and looking forward
Most of this handbook describes practice that is reasonably well established. This section does not. It identifies problems on which the evidence is still developing and on which practitioners should hold their methods provisionally. Each is framed as a question rather than a prediction.
How should long trajectories be judged, rather than single responses?
Assistants increasingly plan, call tools, and act over many steps. A trajectory can reach the correct end state for the wrong reasons, or fail at step 3 of 40 in a way that matters only later. Human annotators struggle to read long traces carefully, and judges given an entire trace face the same difficulty with greater apparent confidence. Early work uses agents to judge other agents step by step[6], but what a sound rubric, unit of analysis, or human ceiling looks like for trajectories remains largely open.
Whose judgment serves as the reference?
For subjective tasks, Section 1.3 argued for preserving disagreement rather than voting it away. In practice, most pipelines still collapse judgments to a single label, and most judges are trained or prompted toward a single view. Pluralistic approaches, which represent the range of reasonable views, steer toward a specified perspective, or match the distribution of views in a target population, are an active area of research[7]. The practical question for teams is simple to state but difficult to answer: who are the annotators, and which users do they represent?
What happens when annotators see AI suggestions?
Pre-filling labels with judge outputs can make annotation considerably faster. It can also anchor annotators toward the judge, which erodes the independence that makes human labels useful for validating that same judge. In a clinical-annotation study, Levy et al. found that experts given model suggestions mitigated some model errors but took less initiative to add what the model had missed[8]. A reasonable default is to maintain a stream of fully independent, unassisted labels for validation, even when assisted labeling handles most of the volume.
Is the measuring instrument made of the thing being measured?
Judges and the models they evaluate are increasingly trained on overlapping data, sometimes including each other’s outputs. Judges can favor text that resembles their own generations[9], and training models on recursively generated data can erode the tails of the data distribution[10]. How much these effects matter for evaluation in practice is not yet known. The defensive measure is the one to which this handbook repeatedly returns: an independent, human-verified anchor set that no model has influenced.
Does the evaluation hold in every language and culture served?
Judges are usually validated predominantly in English, and rubrics written within one culture transfer imperfectly to others: what counts as a complete recipe answer, or an appropriately direct tone, varies. Per-language human ceilings and per-language judge validation are expensive, which is precisely why they are often omitted.
What happens to human labeling budgets?
As judges become cheaper and more capable, the pressure to reduce human labeling grows. The position of this handbook is that human labels do not become less important; they become more targeted: fewer, better-designed labels on random audits, boundary cases, new slices, and anchor sets. The risk is that budgets shrink faster than design improves.
Conclusion
This handbook has treated evaluation as measurement. Chapter 1 argued that every evaluation should begin from the decision it supports and the construct it measures. Chapters 2 and 3 addressed alignment within humans: designing the annotation task and measuring whether annotators apply it consistently. Chapters 4 and 5 addressed alignment within the judge: building an LLM judge and testing whether it is stable and free of irrelevant biases. Chapter 6 addressed alignment between the two, and this chapter addressed keeping all three alignments valid over time. The central recommendations can be summarized as three commitments.
- Design for the decision. Specify the construct, unit of analysis, population, estimand, and decision rule before collecting any data (Section 1.2).
- Test the whole system. Pilot the sample, rubric, interface, training, and judge together, and measure each of the three alignments separately.
- Validate, and continue validating. Compare against the human ceiling, inspect disagreements before correcting them, audit at random, and retain the audit trail that allows others to verify the work.
Further reading
- Foundations of LLM-as-a-judge: Zheng et al. (2023) on MT-Bench and Chatbot Arena, which documented position, verbosity, and self-enhancement biases.
- Aligning evaluators with people in practice: Shankar et al. (2024), on criteria drift and evaluator alignment workflows[2].
- Agreement statistics: Artstein & Poesio (2008), a thorough and accessible survey of inter-coder agreement for computational linguistics.
- Disagreement as signal: Aroyo & Welty (2015) and Plank (2022).
- Correcting aggregate estimates: Angelopoulos et al. (2023), prediction-powered inference.
- Documentation: Gebru et al. (2021), datasheets for datasets[5].
Summary & quiz
- Five components drift independently: the inputs, the relationship between inputs and labels, the human annotators, the judge, and the evaluation standard. Each requires a different response.
- Effective monitoring combines leading (unlabeled), lagging (labeled), and anchored (frozen-set) signals. Anchors separate a change in the world from a change in the instrument.
- When agreement drops, examine the annotators, the judge, and the inputs before editing prompts, and determine which side is correct before correcting anything.
- Changes to the judge warrant the same gate as a model launch: criteria written in advance, paired comparisons, critical-class recall, and enough test cases to be informative (the rule of three).
- Cold start proceeds in stages; synthetic data supports coverage rather than labeling; and sufficient provenance keeps labels reusable through changes to the rubric and the label set.
The judge’s “Unsafe” rate on live traffic rises from 2% to 4% overnight. The input topic mix is unchanged, and the judge’s accuracy on the frozen anchor set dropped on the same day. What is the most likely cause?
Judge v5 correctly flagged all 3 unsafe examples in the launch test set. Which summary is accurate?
The team redefines the answer-quality label “Fully” to require a safety caveat where relevant. How should the previous year of answer-quality trend data be treated?
To save time, the annotation interface is pre-filled with the judge’s label, and annotators are asked to confirm or correct it. What is the principal risk?
A team generated 500 stress cases with an LLM, retained its labels, and verified them with a second LLM. How should the set be used?
References
- Chen, L., Zaharia, M., & Zou, J. (2024). How is ChatGPT’s behavior changing over time? Harvard Data Science Review, 6(2).
- Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. UIST 2024.
- Rabanser, S., Günnemann, S., & Lipton, Z. C. (2019). Failing loudly: An empirical study of methods for detecting dataset shift. NeurIPS 2019.
- Hanley, J. A., & Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA, 249(13).
- Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12).
- Zhuge, M., Zhao, C., Ashley, D., et al. (2025). Agent-as-a-Judge: Evaluate agents with agents. ICML 2025.
- Sorensen, T., Moore, J., Fisher, J., et al. (2024). A roadmap to pluralistic alignment. ICML 2024 (position paper).
- Levy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021). Assessing the impact of automated suggestions on decision making: Domain experts mediate model errors but take less initiative. CHI 2021.
- Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. NeurIPS 2024.
- Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631.