Notes from the build · 0228 August 2026 · 6 minutes
What a label diff in millimetres looks like
A folder called final_v3 does not tell you what changed. On the slice in Figure 1, moving every boundary out by a single pixel changes the labelled area by 14 per cent. That is the size of the thing people argue about, and it is why a change here is reported in millimetres and counts.
Written by the people building it
Figure 1 · slice 192 · one pixel of disagreementThe folder called final_v3
Every team that labels scans has one. A shared drive with masks_v2, masks_v2_fixed, final, final_v3, and one folder whose name is somebody's initials and a date. Ask which one trained the model that is running in production and you get a pause, then a guess, then a message to a person who has left.
The pause is not carelessness. It is what happens when the only thing the tool records is a file, and a file has no memory of the argument that produced it.
So the question we started from was small and concrete. When two versions of a label disagree, what is the smallest useful sentence you can say about the difference?
Not "the mask changed". Not a percentage overlap score, which is a number two people can both accept while still disagreeing about the part. Something an inspector can act on.
One pixel, measured
Start with the smallest disagreement two annotators can have: one of them puts the boundary one pixel further out.
On the slice in Figure 1 that pixel is 0.33 mm across.1 It sounds like nothing. So we measured it. The slice carries 16 labelled instances covering 3,690 mm². Grow every one of those boundaries outwards by exactly one pixel and the labelled area becomes 4,216 mm².
The difference is 526 mm², which is 14 per cent of what was there.3
Fourteen per cent is not a rounding error. It is larger than most of the effects people are trying to detect. If your acceptance rule is written as a percentage of area, two careful annotators who never disagree about what a defect is can still land on opposite sides of it.
The reason is geometry, not sloppiness. Labelled shapes on a slice have long thin boundaries, and a thin ring around a long boundary is a lot of area. The measured ring here is about 1,594 mm of boundary, one pixel wide.
One pixel out on every edge is fourteen per cent of the labelled area. That is the size of the argument.
Measured on slice 192, not estimatedSo a system that reports "the mask changed" is telling you nothing, and a system that reports a percentage overlap is telling you something that hides this. The useful sentence has millimetres in it.
Volume, boundary, class
A label change is one of three things, and each one deserves a different sentence.
Volume. The object got bigger or smaller. Report it as a change in cubic millimetres, per object, with the sign.
Boundary. The object is in the same place and the edge moved. Report how many boundaries moved and by how much, in millimetres, as a median rather than a mean, because one badly drawn outlier should not set the number.
Class. The object is the same shape and it is now called something else. Report the old name, the new name, and the rule change that caused it.
Those three cover almost every real disagreement we have seen, and each one lands as a plain line an inspector can read without a manual. A reviewer accepts or rejects the change, and both names stay attached to it.
What that looks like in practice is Table 1. The shapes of those lines are real. The specific numbers and names in the table are made up, because these are not two versions of a published dataset, they are an example of the sentence.
| Kind | The question it answers | The line you get |
|---|---|---|
| Volume | Did the object get bigger or smaller? | −1.8 mm² median across 6 revised boundaries |
| Boundary | Did the edge move, and by how much? | 6 boundaries revised, 4 accepted, 2 sent back |
| Class | Is it now called something else? | 2 voids rejected as an imaging artefact |
| Class, by rule | Did the naming rules change under us? | taxonomy v3 to v4: porosity split into gas and shrinkage |
A rollback has to bring the rules back
The fourth row of Table 1 is the one that catches teams out.
Half the label changes in an inspection programme are not drawing at all. Somebody rewrites the rule about what counts as porosity, and every drawing made under the old rule now means something slightly different. The outlines on disk did not move. Their meaning did.
So a rollback that only restores masks restores a lie. Going back to last March has to bring back the list of names that was in force last March, and the review rules that went with it, or the labels you get back do not mean what they meant.
That is why the naming list and the workflow rules are versioned in the same record as the drawings, and why a rollback takes all three.
Branching a dataset and reading a change as volume, boundary and class works today.4
What this does not fix
Measuring a disagreement is not the same as settling it.
It does not tell you who is right.A diff in millimetres says the two people disagree by 1.8 mm². Which boundary belongs on the part is still a judgement, and it still needs the person whose judgement counts.
It does not make a bad rule good. If the naming rules are vague, versioning them only means you have a dated record of the vagueness. That record is worth having. It is not a fix.
It does not remove the cost of review. Someone still has to look. What we can do is make sure that person looks at the slices where the model is unsure rather than at all of them, which is a different post.4
What it does fix is the pause. Ask which version trained the model, and the answer is a commit, with a name and a time and a list of what changed since the one before it.
Notes and sources
- Fraunhofer EZRT XXL-CT Instance Segmentation Me 163, subvolume V5. doi:10.5281/zenodo.10651746, CC BY 4.0. 512 slices, 0.33 mm per pixel in the plane of the slice, 0.6 mm between slices, 169 mm across the frame. Slice 192 is the frame shown in Figure 1. The label colours are ours, drawn at one perceived lightness so no instance shouts louder than another; the labels themselves are the dataset's. back to text
- Instance counts come from the dataset's own labels: 168 labelled instances across the 64 sampled slices, 16 of them on slice 192. back to text
- Areas measured on the label image this page loads, on 28 August 2026: 33,888 labelled pixels, which is 3,690 mm² at 0.1089 mm² a pixel; growing the set by one pixel in each of the four directions adds 4,830 pixels, which is 526 mm², or 14.3 per cent. That image is a compressed copy of the dataset's labels, so read the numbers as accurate to about a per cent, not to the square millimetre. Your browser may land a few square millimetres from those figures, because image decoding is not identical on every machine. The measurement is repeated in your browser when the page loads, and the caption of Figure 1 is rewritten with what it finds. back to text
- Working on a copy of a dataset, and reading a change as volume, boundary and class, both run on the hosted instance at app.nitsor.com. Sign-in is by invitation. Product status lists what does and does not work. back to text