CT annotation quality control: measure reader agreement and review disagreements

Keep each reader's annotation

Four radiologists can outline the same finding and produce four different masks. A quality process has to keep those differences long enough for someone to understand them.

The case in this guide is LIDC-IDRI-0003, a public chest CT with published reader outlines. The picture at the top of this guide shows the four outlines of one nodule in white, on the slice the scanner numbered 64. They overlap along much of the edge but differ in some regions. Reader 4 drew the largest outline and marked the nodule on 10 slices. The other three marked it on 7.

Store an individual mask for every reader before building any combined reference. Keep the reader identifier, the source series, the slice geometry and the version of the instructions each reader followed. Without those, a disagreement score is hard to read and harder to repeat.

A combined mask answers a different question. A common rule for LIDC keeps every pixel that at least two of four readers included. On this slice that gives 479.1 mm², against 411.1 to 580.0 mm² for the individual readers. Record that rule with the combined mask so the next team can reproduce it.

Choose the comparison unit

Decide what you compare before you calculate anything. One slice, one object across all its slices, and a whole volume give three different answers, and a report that mixes them is hard to interpret.

This guide starts with one slice because readers can compare all four outlines by eye. That comparison does not describe agreement across the whole object. One object across its slices is usually the unit that matters for training, since the model learns the whole shape. A whole volume adds empty slices, and those need a rule of their own: two readers who both leave a slice empty agree completely, and averaging that in makes every case look better.

Table 1 already shows why the unit matters. The readers disagree about area on this slice by up to 169 mm², and reader 4 also outlines the nodule on three more slices than the others. A single-slice score cannot see that second disagreement at all.

Calculate overlap without hiding disagreement

With four readers there are six pairs. On this slice their IoU runs from 0.71 to 0.85.² Keep all six beside the masks. A reader can then see which pair produced each value and where the outlines diverge. The average of 0.78 does not identify the pairs that disagree most.

In Table 2, every pair that includes reader 4 has lower overlap than the three pairs among readers 1, 2 and 3. At adjudication, a radiologist should check whether reader 4 included tissue the others left out, then ask whether the difference came from the instructions or from judgement about the edge.

Look at where the pixels agree as well as how much. Of the 880 pixels any reader marked on this slice, 547 were marked by all four, and 168 by only one. Most of those 168 lie within three pixels of the outer edge (Figure 1). Inspect that band to see where the readers disagree.

Boundary measurements

A two-pixel boundary shift changes IoU less on a large mask than on a small nodule. For small findings, add a boundary measure such as the largest distance between two outlines, and report it in millimetres at the recorded spacing.

Table 2Overlap between each pair of reader outlines on InstanceNumber 64, highest first.

Readers	IoU	Dice
1 and 2	0.85	0.92
2 and 3	0.82	0.90
1 and 3	0.81	0.89
3 and 4	0.75	0.86
1 and 4	0.71	0.83
2 and 4	0.71	0.83

Decide how a disagreement gets resolved

Write down how adjudication resolves a disagreement before the first study arrives. Overlap scores identify differences but cannot determine which outline to use.

Your written procedure could define four outcomes, separate from Nitsor's verdicts. Accept one reader's outline as the reference. Correct it into a new outline and record the reason. Return the case to the original reader with a note about the instruction. Or keep the disagreement as data, because some findings have no single agreed edge. An evaluation should account for that uncertainty.

Decide which cases need review. Your procedure might require review below an agreement threshold and a random sample of the remaining cases. Sampling can reveal errors even where readers agree. The agreement calculations in this guide ran outside Nitsor. Routed review applies the same rule to model output: a threshold and a sampling rate you set, with the rule kept on each task.

Record which revision the reviewer judged

A review record should separate the original reader annotation from any later correction. Keep both, name the person who made the change, and attach the verdict to the revision that was inspected.

For a case like reader 4's outline above, send the study to a consensus read. Two to sixteen readers label it, blind to each other by default, and Nitsor scores their agreement: Dice or IoU for masks, distance in millimetres for points and exact match for each field.

Your team sets a criterion for each metric, a minimum overlap or a maximum distance, whether readers work independently or check and correct, who may adjudicate and what each reader sees. When reads miss a criterion, the study goes to an adjudicator. The adjudicator takes one reader's labels, keeps the original or edits their own, and History records the decision with a name and a time.

A single second read is recorded the same way. Automatic routing sends the work to someone other than its author; if an operator names the author instead, the record shows it was a self-review. The reviewer records accept, correct or reject with a name and a time. A disagreement your team keeps as data carries no verdict; write that decision into the task instead.

At a review stage, a reviewer records accept, reject or correct per annotation row, bound to the annotation hash judged. The reviewer sees who authored each annotation, and finishing the review records a commit under the reviewer's name. Editing an annotation changes its hash, so an earlier verdict stops applying; the verdict remains in the history. The agreement numbers in this guide were calculated from the published outlines with a short script outside Nitsor. Keep those outlines and the calculation inputs to trace each result.

An annotation QA worksheet

Fill this in once per project, before the first comparison. The answers belong in the task instructions, where every reader and reviewer can see them.

1. Name the finding or structure, and link the instruction version readers follow.
2. Keep one mask per reader, with the reader, series and slice geometry.
3. Choose the comparison unit: one slice, one object across its slices, or a volume.
4. Say how empty slices count before averaging anything.
5. Report every pairwise score, not only the mean.
6. Add a boundary measure in millimetres for small findings.
7. State the rule behind any combined reference, such as at least two of four readers.
8. List the four procedure outcomes above and who owns each.
9. Set the agreement threshold for review and a sampling rate for the rest.
10. Attach every verdict to the revision the reviewer saw.

If your team reviews CT or MRI labels with more than one reader, we can walk through this worksheet on a public case close to your data. Request a walkthrough.

In the product: Routed reviewRead the annotation guide (opens in a new tab)