Why the scan never leaves your bucket

Two ways to run the job

There are two ways to get a set of scans labelled by somebody else's software. The first is the one everyone knows. You copy the scans to the vendor. Their annotators draw on them. A folder of masks comes back a few weeks later.

The second is the one this post is about. The scans stay in the storage you already pay for. The software reads the slices a person is looking at, over the network, and sends back a record of what was drawn. No copy is made.

We built the second one. Not because the first is hard. Because of the question somebody asks two years later, when a part fails or a model behaves oddly in the field: who drew this mask, and on which version of the scan? If the pixels went to a vendor and the masks came back in a folder, that question is an archaeology project. If the scan never moved, it is a query.

What a range read moves

The volume in Figure 1 is a published scan of a Me 163 airframe from Fraunhofer EZRT.¹ As we hold it, it is 512 slices of 512 by 512 samples, at two bytes a sample. Multiply that out and the whole volume is 268 MB. One slice is 524 kB.

When you open slice 192 in the browser, the browser asks your storage for the bytes of slice 192 and stops there. The web has had a header for this since 1999, called Range, and every storage service that copies the S3 protocol answers it. So 524 kB crosses the network, straight from your bucket to the screen of the person looking at it.

Scroll through ten slices and you have moved about 5 MB. Upload the same volume to a vendor and you have moved 268 MB, and then it is somewhere else, forever, on a machine you do not control.

The difference gets larger, not smaller, with the size of the study. A 2,000-slice industrial scan is a bad upload and an ordinary read.

The pictures on this page are much smaller

Every frame in this post is a 512-pixel AVIF image of about 36 kB, because a web page does not need 16-bit samples to show you the shape of a wing spar. All 64 frames come to 2.0 MB, and the label overlays add 0.3 MB.³ The viewer inside the product reads the real samples. This is a blog post.

What we keep instead of a copy

Three things, and none of them is a picture.

A list. Where each series lives, how many slices it has, how big they are, and the technique settings that came with the file. The viewer needs this to know what to ask for.

A hash. A checksum of each file, taken when we first index it. If the file changes under us, the next read says so instead of quietly showing you something else. A scan that changed after it was labelled is a fact you want on the record, not a surprise.

The record. Every label event, with the person or model that made it, the person who accepted it, the parent it came from, and the time. This is the actual product. The rest is plumbing.

So if somebody took our whole database and none of your storage, they would have a list of file paths, a column of checksums, and a set of outlines in millimetres with names attached. We are not going to tell you that is nothing. It is not nothing. It is also not your scans, and it is a great deal less than a full copy of them sitting in a vendor's account.

What an auditor gets

This is the part that makes the whole arrangement worth the trouble.

A release is a JSON file. It lists the label events in order, names the people and the model versions behind them, and carries a chain of hashes so each entry commits to the one before it. Change any entry after the fact and the chain stops matching.

An auditor can check that file on their own laptop. They do not need an account with us. They do not need the scans. What the chain proves is that the list has not been edited since we chained it. It is not proof that we could not rechain it, and we are not going to claim otherwise.

Compare that with the usual artefact, which is a PDF report saying the work was done. A PDF is a claim. A chain of hashes over a list of events is a claim you can test.

Releases with a log and a hash chain work on the hosted instance today.⁴

What the arrangement costs you

It is not free, and a post that only listed the good parts would not be worth reading.

You run the storage. Its uptime is yours. If the bucket is unreachable, so is the slice, and we cannot serve you a cached copy because we do not have one.

Your network is in the path. A slice read from a bucket two regions away arrives later than a slice read from a disk in the same room. Annotators feel that as a small delay when they jump around a series. Putting the storage near the people is the fix, and it is your decision to make, not ours.

Signed reads have a clock. The browser gets a link that works for a short window and then stops working. That is deliberate, and it means clock skew on a laptop can produce a confusing error. We would rather have that error than a permanent public link.

You have to know where the scans are. More teams than you would expect cannot answer that on the first call. The answer is usually worth having anyway.

None of these are surprises we spring later. They are the price of the scan staying yours, and we would rather you read them here than discover them in week three.