Three capture screens in ninety minutes: what we deleted to make label photos readable

When Munchable does not have a product, it asks the person in the aisle to photograph the ingredients panel, and that photo becomes the data every later scanner sees. So the capture screen is the most important screen nobody designs on purpose. On 12 September it was redesigned three times in ninety minutes, and most of the work was deletion. Here is what went, why, and what the two additions were.

Deleted: the zoom slider and the crop canvas

The first version of the day had a full-screen camera, a vertical zoom slider, a contain-fitted crop canvas, and a separate preview screen with "Use this photo" and "Crop it again". About 550 lines. All of it went an hour later, and the reasoning is one sentence in the screen docs:

There is no zoom control on purpose. Digital zoom gives the reader nothing a crop does not, and a zoomed preview over an unzoomed still is the "it zooms out after the photo" jump that has to never happen.

The underlying problem was soft corners. A cramped viewfinder pushes people to within a couple of inches of the pack, inside the lens's minimum focus distance. Shooting wide and cropping afterwards gives the reader more pixels per letter than a close-up of the same panel, because the crop happens before the downscale. Digital zoom is the opposite: fewer pixels, same blur.

The frozen still now uses the same cover fit as the live preview, so when the shutter fires the picture simply stops. Nothing moves, letterboxes or appears to zoom out.

Deleted: the framing guide as a default crop

The middle version drew a target rectangle on the camera that doubled as the default crop on the still, so a user who framed the panel inside it needed one tap. That also went, an hour later, in favour of this:

THE SCREEN IS AN ORDINARY MUNCHABLE PAGE. Heading, explanation and buttons are on the cream page in the app's own type, like every other screen, and the camera is a card in the middle of it. Nothing is drawn over the picture on a dark band. This is how the scanner apps people already know lay out a capture step, and it is what lets the page say, in a real heading, why the camera is open.

The user draws the crop box themselves. What remains of the guide is four white corner marks, and the comment beside them is precise about what they are: "Four corners, not a box: where the panel should go. The whole frame is still captured, and the user draws the real box on the next step."

Photos are then reviewed on their own page, full size, with a labelled Remove under each. Checking whether words are readable is a job for a big picture and a calm page, not a 72-point tile next to a shutter.

The one piece of geometry that survived

The drawn box gets breathing room before it is cropped, and the amount is a percentage rather than a fixed number of points:

/**
 * Percentage rather than a fixed number of points because the boxes people
 * draw here differ by an order of magnitude: a two line "Ingredients: oats,
 * water" on a porridge pot against the full wrap of a shampoo sized smoothie
 * bottle. A fixed 12pt pad is a third of the height of the small box and
 * invisible on the big one. Six percent is the number because a drag ends
 * where the finger lifts, which on a 4.5in class thumb is a couple of
 * millimetres inside where the user thought they were pointing.
 */
export const CROP_PAD_FRACTION = 0.06;

/**
 * Floor under the percentage pad, in view points. A box only just over
 * MIN_SELECTION would otherwise get 44 * 0.06 = 2.6pt of margin, which is less
 * than the width of the box's own border and cuts the ascenders off the top
 * line of text.
 */
export const MIN_CROP_PAD = 8;

Added: tap to focus, by patching the camera library

expo-camera does not ship tap to focus. The repo patches the dependency to add it, and the Android half is a dozen lines of Kotlin against CameraX:

fun focusAt(x: Float, y: Float) {
  val cam = camera ?: return
  val point = previewView.meteringPointFactory.createPoint(x, y)
  val action = FocusMeteringAction.Builder(point, FocusMeteringAction.FLAG_AF or FocusMeteringAction.FLAG_AE)
    .setAutoCancelDuration(4, java.util.concurrent.TimeUnit.SECONDS)
    .build()
  cam.cameraControl.startFocusAndMetering(action)
}

Four seconds later it cancels back to continuous focus. On the JavaScript side a gold ring appears where the finger landed and fades over 1.2 seconds. The patch is applied through the package manager's patched-dependencies mechanism, so it survives installs.

The comment next to the autofocus prop is the kind that saves a future contributor an afternoon:

/**
 * FOCUS MODE. DO NOT "FIX" THIS TO 'on'. expo-camera's FocusMode
 * reads backwards from its name, on both platforms: 'on' means
 * focus ONCE and then lock, 'off' means keep focusing as the
 * scene moves. So native gets 'off'.
 */
autofocus={Platform.OS === 'web' ? 'on' : 'off'}

Deleted: the confidence gate

The server used to refuse a capture whose OCR confidence was below 0.75, with a "not clear enough" error and nothing saved. That gate is gone, and the replacement comment explains why it was never doing what it looked like it was doing:

if the reader found an ingredients list, the capture worked and it is saved. A read the reader itself doubted still enters as unverified, still needs a second independent read to reach consensus, and still fails closed on the device if the words are unknown, so nothing about trust rested on this number. What the number bought was a user who had done everything right being told their photo was not good enough and a product Munchable then did not have.

The 0.75 survives in exactly one place: as the floor below which a capture does not earn a contributor reward. A reward floor, not a save floor. The only thing refused now is a value that is not a confidence at all.

Added: a question before discarding work

The close button asks first, but only when there is something to lose:

async function leave() {
  const hasWork = raw !== null || ingredientsShots.length > 0;
  if (!hasWork) { goBack(); return; }
  const ok = await confirm({
    title: 'Leave without reading the label?',
    message: 'The photos you have taken will be discarded. Nothing has been saved yet.',
    confirmLabel: 'Leave',
    cancelLabel: 'Keep going',
    destructive: true,
  });
  if (ok) goBack();
}

A tap that lands on the X by accident would otherwise throw away a label the user spent a minute framing. With nothing taken yet there is nothing to ask about, so it simply closes. Adding a confirmation to the empty case would be a popup for its own sake.

The capture flow is in the Android app and in the browser version, where the shutter works but the torch and tap to focus do not, because the web camera is a getUserMedia preview with none of that wired to it. Sign in, scan a barcode the catalogue does not have, and the screen this post describes is the one that opens. The trust rules that decide what happens to the photo afterwards are in an earlier post, crowdsourcing data into a health app with no moderation queue.