3. Measuring Human Judgments
A well-designed annotation task does not by itself guarantee trustworthy labels. The same rubric, applied by different people, can yield very different datasets: when the SummEval benchmark collected ratings of summary quality from both crowdworkers and domain experts, the two groups diverged substantially, and the authors chose to rely on the expert ratings[1]. Before human labels can serve as a reference for anything else, including an LLM judge, the team needs evidence that the people producing them apply the rubric consistently and as intended.
This chapter addresses that evidence. In the three-way alignment model of Section 1.4, it covers the within-humans side: whether annotators agree with each other, with themselves over time, and with the intent of the rubric. Section 3.1 discusses who should annotate and how to qualify and train them. Section 3.2 introduces agreement statistics and their pitfalls. Section 3.3 examines consistency over time and systematic rater bias. Section 3.4 covers how to combine several labels into one reference label, and Section 3.5 describes the quality controls that keep these properties stable during production labeling.
The examples use Mise, a fictional cooking assistant that answers questions about recipes, techniques, and food safety (described in Section 1.5). Three Mise evaluation tasks appear here:
- Answer-quality task: does the response fully answer what the user asked? Labels: Fully / Partially / Does not answer (an ordered, rubric-interpreted judgment).
- Food-safety task: does the response contain unsafe food-handling advice? Labels: Safe / Unsafe. Unsafe responses are rare (about 3% of traffic) but severe.
- Pairwise preference task: which of two responses would the user rather receive? Labels: A / B / Tie.
Helpful background: the three kinds of reference (Section 1.3), the executable rubric (Section 2.2), and the pilot study (Section 2.5). All numbers in the interactive figures are simulated for illustration.
Recruit, qualify, train
Two well-known results appear to point in opposite directions. Snow et al. found that, for several simple natural language processing (NLP) labeling tasks, averaging a small number of non-expert labels per item could approach expert-level quality[2]. The SummEval study described above found that crowd ratings of summary quality diverged substantially from expert ratings[1]. Both findings hold. Non-expert annotation works well when the judgment is one that most attentive people can make; it fails when the judgment requires knowing what to look for.
The first staffing question is therefore not how many annotators to recruit, but what kind. The answer follows from the kind of reference the task requires (Section 1.3): a verifiable question calls for people who can check against an authoritative source, a rubric-interpreted question calls for people trained on the rubric, and a subjective question calls for people who represent the intended users.
| Pool | Suitable for | Risky when |
|---|---|---|
| Open crowd | Clear, short tasks with little rubric interpretation; large volume; many labels per item. | The task requires domain knowledge, or the difficult cases are rare and easy to overlook. |
| Trained raters | Rubric-interpreted tasks such as the answer-quality task, performed by people who have completed calibration and know the documented edge-case rulings. | Training diverges from the written guide, or a small, long-standing team develops undocumented conventions. |
| Domain experts | High-stakes, nearly verifiable judgments such as the food-safety task; adjudication; authoring gold items. | High volume is required, or the question concerns what ordinary users experience. |
| Target users | Subjective tasks such as the pairwise preference task or tone judgments, where user reactions constitute the construct. | Their disagreement is treated as noise and removed by majority vote. |
Qualification tests: the easy-item trap
Once a pool is chosen, most teams screen candidates with a short qualification test. A common mistake is to build that test from the clearest examples in the guideline, because those items have unambiguous answers. A test composed only of easy items largely measures whether a candidate read the instructions. It says little about how the candidate will handle the boundary cases that later account for most disagreements. The simulation below makes this concrete.
A strong source of qualification items is the set of pilot disagreements (Section 2.5): items on which pilot raters split and an expert subsequently adjudicated. These are precisely the decisions new raters must get right. Retaining a few easy items is still useful, because it distinguishes candidates who did not read the guide from candidates who read it but misjudged a boundary.
Calibration sessions turn a rubric into shared practice
Even qualified raters will encounter cases that the written guide does not cover. In a calibration session, raters label the same small batch independently and then discuss the items on which they disagreed. When conducted well, the session produces rulings: short, citable decisions that are written back into the guide so that every rater applies them in the same way.
- The “correct” label is shown for each practice item, without discussion.
- Disagreements are settled by the most senior person present.
- Rulings are recorded only in a chat thread that becomes difficult to locate.
- Every rater labels independently first and then sees the distribution of labels.
- The group examines the evidence and names the rule that decides the case.
- The ruling is added to the guide together with the example, e.g. “Advice to ‘reheat until hot’ does not make counter-stored rice Safe.”
Gold items, attention checks, and working conditions
Calibration establishes shared practice; the team still needs a way to verify that individual raters follow it. Gold items serve this purpose: they carry an expert-established label and are mixed, without identification, into the regular queue. Unlike agreement, they measure accuracy. Two cautions apply. First, gold items that differ visibly from real items (shorter, cleaner, evidently planted) tend to be recognized and receive extra care, so they overestimate accuracy; they should be drawn from real traffic. Second, gold labels require maintenance: when the guide changes, some gold answers change as well.
Attention checks (for example, “select ‘Partially’ for this item”) detect automated or inattentive responses, but they are a blunt instrument. Crowdsourcing research has long recommended questions with verifiable answers that require engaging with the content[3], rather than trick instructions that primarily measure whether a rater noticed the trick.
Fatigue, compressed deadlines, and per-item pay that rewards speed all appear in the data as disagreement. Realistic time per item (measured during the pilot), payment for that time, scheduled breaks, and rotation of task types in long sessions are matters of ethics and also of measurement quality.
With a qualified and calibrated pool in place, the next question is how to quantify whether these raters in fact agree, which is the subject of the next section.
Agreement beyond chance
Consider two raters who label 200 Mise responses on the food-safety task. They assign the same label to 90% of the responses, and the team reports the task as reliable. However, if 95% of the responses are in fact Safe, two raters who label every response Safe without reading it would agree on 100% of items. A high percent agreement can therefore arise without any skill at detecting the class the task exists to catch.
Percent agreement, the fraction of items on which raters choose the same label, remains easy to explain and worth reporting. Chance-corrected statistics address its weakness by subtracting the agreement that would be expected if each rater labeled at random according to that rater’s own label frequencies. For two raters, Cohen’s κ (kappa)[4] is defined as:
κ = Po − Pe1 − Pe
Po = observed agreement. Pe = agreement expected by chance from each rater’s marginal label rates. κ = 1 indicates perfect agreement; κ = 0 indicates agreement no better than chance.
The preset “Rater 2 always says Safe” is the most instructive. Percent agreement appears respectable, while κ is zero and agreement on the Unsafe class is zero. For the food-safety task, Unsafe is the class the task was built to detect. For this reason, class-specific agreement should be reported alongside any single summary statistic.
The prevalence paradox
Chance correction introduces its own difficulty, which Feinstein and Cicchetti described as “high agreement but low kappa”[5]. Suppose two raters have fixed skill: each detects the same fraction of truly unsafe responses and correctly passes the same fraction of safe ones. If only the prevalence of Unsafe changes, percent agreement barely moves, whereas κ falls sharply as the class becomes rarer, even though the raters themselves have not changed.
Neither statistic is correct in isolation. κ answers a narrower question (how much better than chance the raters agree, given their marginal label rates), and that question becomes genuinely harder when one class is rare. A sound practice is to report percent agreement, a chance-corrected statistic, the label distribution, and per-class agreement together, and then to read the disagreeing items. The statistics indicate where to look; the items explain why.
Choosing a statistic
Cohen’s κ is only one member of a family of agreement statistics. The appropriate choice depends on the number of raters, whether every rater labels every item, and the measurement level of the labels.
| Setup | Common choice | Note |
|---|---|---|
| Two raters, nominal labels | Cohen’s κ | Each rater has separate marginal rates. |
| Many raters, a fixed number per item | Fleiss’ κ[6] | Raters need not be the same people across items. |
| Any number of raters, missing labels, nominal, ordinal, or interval data | Krippendorff’s α[7] | The most flexible default for real-world annotation designs. |
| Ordinal scale (answer-quality: Fully / Partially / Does not) | Weighted κ or ordinal α | “Fully vs. Partially” is treated as a smaller disagreement than “Fully vs. Does not.” |
| Continuous or many-point scores | Intraclass correlation coefficient (ICC)[8] | Several ICC forms exist; the form should match the study design. |
| Pairwise preferences (A / B / Tie) | Pairwise agreement, with and without ties | The treatment of ties should be reported, as it changes the value considerably. |
The statement “κ above 0.8 is good” is often quoted as a rule. The commonly cited bands (0.41–0.60 “moderate,” 0.61–0.80 “substantial,” and so on) originate with Landis and Koch, who presented them while acknowledging that the divisions were arbitrary[9]. Artstein and Poesio’s survey for computational linguistics discusses why no single cutoff suits all tasks[10]. A κ of 0.6 may be satisfactory for a judgment of which reply is more humorous and inadequate for a judgment of whether advice is unsafe. The threshold should be derived from the risk of the decision.
Agreement between raters is one aspect of reliability. The next section examines two further aspects that inter-rater statistics do not reveal.
Consistency and rater bias
Suppose a team compares two model versions and finds that the newer version scores half a point lower on the answer-quality scale. Inter-rater κ is acceptable. On inspection, however, most of the newer version’s responses were rated by a single recently onboarded rater who scores every response about a point lower than colleagues do. Inter-rater agreement did not reveal the problem, because it measures whether raters agree on the same items, not whether one rater is systematically shifted. This section covers two such properties: whether each rater agrees with themselves over time, and whether each rater has a systematic tilt.
Intra-rater consistency: blind repeats
To measure self-consistency, a small random set of previously labeled items is reinserted into a rater’s queue, separated by enough time and enough other items that the rater is unlikely to recognize them. The rate at which a rater changes their own answer directly indicates how stable that rater’s decision boundary is. A rater who changes the label on 25% of repeated items is signaling a problem with the task, with fatigue, or with a part of the rubric that has not been internalized. It is informative to examine which items change: if they cluster around one rule, that rule requires a clearer example.
Severity and leniency
On scored tasks, some raters are systematically harsher than others. A strict rater who assigns a 3 where a lenient rater assigns a 4 may be entirely consistent; the two simply use the scale differently. A simple additive model of each rating makes this explicit:
yij = μ + qi + bj + εij
The rating of item i by rater j equals the overall mean plus item quality plus a rater offset plus noise. A strict rater has a negative offset bj.
Severity is harmless when every item is rated by the same raters, because the offsets cancel. It becomes a problem when raters see different items, which is the usual situation at scale. In that case an item’s raw average partly reflects which raters happened to rate it.
Statistical adjustment has limits. It requires sufficient overlap (raters sharing items) to estimate the offsets, and with only two ratings per item the estimates are noisy. Its principal value is diagnostic: once it is evident that Rater C scores about 0.6 points below the average rater and more than a point below Rater A, the team can decide whether to recalibrate, rebalance assignments, or model the effect. A rater who scores lower is not necessarily less accurate; accuracy on gold items should be checked before concluding which rater is miscalibrated.
Assignment design can manufacture a result
The opening example of this section illustrates why assignment design matters as much as the statistics. The following comparison contrasts the two designs.
Responses from model v3 were labeled in week 1 by the original team. Responses from model v4 were labeled in week 2, mostly by one new, strict rater.
Result (hypothetical): v4 averages 3.4 compared with 3.9 for v3, and the team nearly abandons v4. Most of the gap is attributable to the rater, not the model.
The v3 and v4 responses to each prompt are shuffled into a single queue, every rater receives a mix of both, and the producing model is hidden.
Rater offsets now affect both models equally and cancel in the difference. The comparison measures the models.
Further reading: many-facet Rasch and hierarchical models
The additive model above is a starting point. For ordinal rating scales, many-facet Rasch measurement[11] models the probability of each rating category as a function of item difficulty, rater severity, and other facets (such as the criterion being rated) on a common logit scale. Mixed-effects (hierarchical) regression with random effects for items and raters is closely related and can be fitted with standard statistical software. Both approaches yield rater-severity estimates with uncertainty, and both require a connected design in which every rater is linked to every other rater through shared items.
Measuring agreement, consistency, and severity characterizes the raters. Most downstream uses, however, require a single label per item, which raises the question of how to combine the labels.
From many labels to a reference
Suppose each Mise response in the food-safety task has received five independent Safe/Unsafe labels, and on one response the vote is three Safe to two Unsafe. One of the two Unsafe votes comes from a rater who labels nearly everything Unsafe; the other comes from the most accurate rater in the pool. Majority vote returns Safe and treats the two Unsafe votes as equivalent. Downstream uses (a launch metric, a validation set for an LLM judge) usually require one reference label per item, and the method used to derive it encodes a specific belief about what disagreement means.
| Strategy | Appropriate when | Limitation |
|---|---|---|
| Majority vote | Largely objective tasks, comparably skilled raters, independent errors. | Treats a careless rater the same as the best rater. |
| Expert adjudication | High-stakes items; small sets; constructing gold items. | Expensive; experts also err, and their errors become the reference. |
| Reliability-weighted vote | Per-rater gold accuracy is available. | Weights become outdated; a single accuracy figure hides class-specific skill. |
| Dawid–Skene[12] | Many noisy raters with enough overlap to estimate each rater’s error pattern. | Assumes one latent true label and errors that are independent given that label. |
| Retain the distribution | Subjective tasks; genuine ambiguity. | Downstream metrics and models must handle soft labels. |
The Dawid–Skene model warrants closer examination because it captures what majority vote cannot. It treats each rater as a noisy observer with an individual confusion matrix: how often the rater says Unsafe when the item is truly Unsafe (sensitivity), and how often the rater says Safe when the item is truly Safe (specificity). Using the expectation–maximization (EM) algorithm, it alternates between estimating the true labels from weighted votes and estimating each rater’s confusion matrix from those labels. A rater who flags everything ends up with high sensitivity, near-zero specificity, and votes that carry almost no information.
Item 21 (starting slow-cooker chicken from frozen) illustrates the limit. The generalist raters A, B, and C all failed to recognize that this practice is generally advised against, because the food can remain too long at unsafe temperatures. Both methods therefore label it Safe with confidence. Statistical aggregation assumes that rater errors are independent; a shared misconception is indistinguishable from agreement. Only an independent check, such as an expert audit or an authoritative reference, can detect it. Audits should therefore include a sample of items on which all raters agreed, not only the disagreements.
Triage disagreements before adjudicating
Whichever aggregation method is used, some items will require expert review, and expert time is the scarcest resource in the pipeline. It should not be spent on mechanical noise. A practical order of operations is as follows:
- Remove mechanical noise. Normalize casing, whitespace, aliases, and empty values (“unsafe”, “Unsafe ”, and “UNSAFE” are one label). Drop rows that have become invalid, for example an answer-quality label on a response that a rater flagged as not being in English, and record the reason.
- Recompute agreement. Some apparent disagreements disappear at this step.
- Categorize the remaining cases. Each case should be read and assigned a cause, because each cause calls for a different remedy:
| Cause | Mise example | Remedy |
|---|---|---|
| Missed evidence | One rater did not scroll to the line recommending raw-egg mayonnaise for a toddler. | Training; interface changes such as evidence highlighting |
| Rubric conflict | Raters split on whether “consult a doctor” counts as answering a health-related question. | A new ruling in the guide |
| Missing context | An earlier user turn mentioned an allergy, but the task displayed only the final turn. | Change what the task displays |
| Genuine ambiguity | “Fully answers” applied to a question with two reasonable interpretations. | Preserve the uncertainty; possibly add a label |
| Legitimate preference | In the pairwise preference task, some raters prefer terse answers and others prefer explanations. | Retain the distribution |
Regardless of the final decision, the raw, independent labels should be retained alongside the reference label. The disagreement history is what allows a ruling to be revisited, uncertainty to be modeled, or rater drift to be studied later. A substantial literature now addresses learning directly from disagreement rather than discarding it[13], motivated in part by evidence that some disagreement in tasks such as natural language inference is inherent and reproducible rather than noise[14].
Restricting evaluation to items on which all raters agreed removes the most difficult items, inflates every downstream metric, and makes an LLM judge appear more accurate than it is in precisely the region where accuracy matters most. If such a filter is applied, it should be reported, together with results on the full set.
The procedures above produce a reference dataset at one point in time. The final section addresses how to keep its quality stable while labeling continues.
Always-on quality control
Consider a rater whose accuracy on gold items is 92% during onboarding. Three months later, after a guideline revision and a shift toward holiday-baking questions, the same rater’s accuracy is 81%, but no one has checked since onboarding. Annotation quality is not a property certified once: raters drift, guidelines change, new kinds of content arrive, and fatigue accumulates. The controls below are inexpensive when designed in from the start and costly to add later.
| Control | Measures | Typical trigger for investigation |
|---|---|---|
| Hidden gold items | Accuracy | A rater’s gold accuracy falls below a task-specific threshold |
| Blind repeats | Self-consistency | The rate of changed labels on repeats increases |
| Double annotation on a random subset | Inter-rater reliability | κ or α declines relative to its history |
| Random expert audits | Error rate of the whole pool | The audited error rate increases |
| Timing and behavior signals | Process anomalies | Implausibly fast items; long runs of identical answers |
| Slice monitoring | Subgroup quality | Quality declines in one language, topic, or source |
Random audits and targeted audits
Targeted audits (reviewing all work by a flagged rater, or every item on which the judge and humans disagreed) are effective for finding errors. Because they select unusual items, however, their error rate does not estimate the error rate of the whole dataset. If 30% of targeted items are wrong, it does not follow that 30% of all labels are wrong. Estimating pool-wide quality requires a small random audit sample alongside the targeted one. The two samples should be labeled separately in the records so that they are not combined later.
Monitoring one rater over time
Gold-item accuracy is typically tracked per rater per week. The figure below shows such a series, including a guideline update, and illustrates how easily noise can be mistaken for decline.
The chart carries two lessons. First, 20 gold items per week is a small sample: a single week at 85% is well within the expected variation for a rater whose long-run accuracy is 90%. Alerts should be based on sustained declines or pooled windows rather than single points; otherwise investigation effort is spent on noise and raters come to regard alerts as arbitrary. Second, the decline begins immediately after the guideline update in week 10. When accuracy falls after a guideline change, the change itself is the first suspect: either training on the new rule was ineffective, or the gold labels were not updated to reflect it.
Labeling speed is a screening signal rather than a verdict. A rater who works twice as fast as the median may be skimming or may be highly skilled; a sample of that rater’s items should be sent to expert audit, and accuracy should decide. Whenever the guide changes, a short recalibration batch for all raters should precede the resumption of production labeling, and the guide should be versioned so that each label records the rules under which it was produced.
This completes the within-humans side of the alignment model: the team now has a reference dataset whose reliability has been measured and whose limits are documented. Chapter 4 turns to the automated side, describing how to build an LLM judge that applies the same rubric at scale. The human labels produced here become its validation set in Chapter 6.
Summary & quiz
- Match the annotator pool to the kind of reference: open crowds for clear judgments, trained raters for rubric-interpreted tasks, experts for high-stakes verifiable judgments, and target users for subjective ones.
- Qualify with boundary cases, calibrate through discussion, and record rulings in the guide.
- Report percent agreement, a chance-corrected statistic, and per-class agreement together. No universal κ threshold exists; the threshold follows from the decision.
- Rater severity is harmless under balanced assignment and harmful without it. Intra-rater consistency is measured with blind repeats.
- Aggregation methods encode assumptions. Dawid–Skene accounts for uneven rater quality, but only independent audits detect shared misconceptions. Retain the raw labels.
- Quality control is continuous: gold items, repeats, random audits, and recalibration after guideline changes.
Two raters agree on 96% of food-safety items, but Cohen’s κ is 0.15. What is the most likely explanation?
A qualification test has a 92% pass rate, but newly qualified raters disagree frequently on production work. What should be checked first?
Model v4 scored lower than v3, but most v4 responses were rated by a strict new rater. What is the most appropriate change for the next round?
All five raters label an item Safe, and Dawid–Skene agrees with high confidence. An expert determines that it is Unsafe. What does this illustrate?
A targeted audit of 100 items on which humans and the judge disagreed finds 40 human errors. What can be concluded about the human error rate on the whole dataset?
References
- Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). SummEval: Re-evaluating summarization evaluation. TACL, 9.
- Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast — but is it good? Evaluating non-expert annotations for natural language tasks. EMNLP 2008.
- Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. CHI 2008.
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1).
- Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6).
- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5).
- Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1).
- Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2).
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1).
- Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4).
- Linacre, J. M. (1989). Many-facet Rasch measurement. MESA Press.
- Dawid, A. P., & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C, 28(1).
- Uma, A. N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., & Poesio, M. (2021). Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72.
- Pavlick, E., & Kwiatkowski, T. (2019). Inherent disagreements in human textual inferences. TACL, 7.