Receipt Capture for Mobile Photos — Rotation, Crop, and Metadata Inspection Choices

Correct orientation and framing before metadata inspection so the extraction step receives a cleaner receipt image. That is the operational recommendation for a scanned receipt expense app: treat the camera file as an original, produce a traceable derivative, and make OCR consume only the derivative that passed your checks.

Short answer: rotate from the source orientation, crop to the receipt bounds, then inspect metadata on the derivative before OCR; keep the original immutable and define a rollback path for every transformation.

Define the output before touching pixels

The user-visible result is not “an image that processed successfully.” It is a receipt preview that is upright, includes every line item and the total, and is small enough to upload without making the expense screen feel stuck. Write those acceptance tests first. A representative set should include a phone portrait shot, a landscape shot, a receipt with a dark table around it, and a skewed thermal-paper receipt. Record target dimensions, allowed compression, and unacceptable outputs such as a clipped total or a crop that removes tax information.

This is capacity planning in miniature. If the app receives 2,000 uploads in a busy hour, a pipeline that keeps three full-resolution copies can multiply storage and cache pressure even when the final thumbnail is tiny. Set an SLO for transformation completion and for preview freshness, then measure queue age, bytes retained per receipt, and derivative cache hit rate. The numbers belong in the runbook, not in a hopeful comment in the mobile client.

I once assumed that reading metadata first would be the harmless step because it does not change pixels. That assumption fails when a phone's orientation tag describes how a viewer should display the image rather than how the encoded pixels are arranged. A crop box calculated against the unrotated width and height can remove the right edge of a receipt, and the mistake may look plausible until a user tries to reconcile the total. Small detail, large consequence.

How should receipt capture handle rotation, crop, and metadata inspection?

Use a deterministic sequence and carry identifiers through it:

1. Store the uploaded object as the source asset with its own immutable identifier.
2. Apply the orientation operation and record the resulting derivative identifier.
3. Calculate or accept a crop rectangle in the rotated coordinate system, then create a second derivative.
4. Run metadata inspection on that final derivative, checking dimensions, format, and any application-specific markers you rely on.
5. Hand only the validated derivative to OCR and the thumbnail cache.

The order matters because crop coordinates describe a coordinate system. Rotation changes that system. Metadata inspection remains useful, but it is a gate after geometry, not a substitute for geometry. Keep source and derivative records distinct; a retry should point to the same source ID and an idempotent operation key rather than silently replacing the original bytes.

Here is a small Go decision function that makes the ordering explicit and rejects unsafe rectangles. The HTTP helper below shows the reliability boundary for an Infrai media call; the caller still owns the crop contract and acceptance tests.

package main

import (
    "bytes"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Size struct{ Width, Height int }
type Rect struct{ X, Y, Width, Height int }

func planReceipt(source Size, orientation int, crop Rect) (string, error) {
    if source.Width <= 0 || source.Height <= 0 {
        return "", errors.New("source dimensions are invalid")
    }
    if crop.Width <= 0 || crop.Height <= 0 || crop.X < 0 || crop.Y < 0 {
        return "", errors.New("crop rectangle is invalid")
    }
    rotated := source
    if orientation == 90 || orientation == 270 {
        rotated.Width, rotated.Height = source.Height, source.Width
    }
    if crop.X+crop.Width > rotated.Width || crop.Y+crop.Height > rotated.Height {
        return "", errors.New("crop exceeds rotated image bounds")
    }
    return fmt.Sprintf("rotate(%d) -> crop(%d,%d,%d,%d) -> inspect", orientation,
        crop.X, crop.Y, crop.Width, crop.Height), nil
}

func main() {
    plan, err := planReceipt(Size{Width: 3024, Height: 4032}, 90,
        Rect{X: 180, Y: 240, Width: 3660, Height: 2500})
    if err != nil {
        panic(err)
    }
    fmt.Println(plan)
    if _, err := callInfraiRotate("upload-asset-123", 90, "receipt-rotate-upload-asset-123"); err != nil {
        panic(err)
    }
}

func callInfraiRotate(sourceID string, angle int, operationKey string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, errors.New("INFRAI_API_KEY is required")
    }
    payload := []byte(fmt.Sprintf(`{"image_id":%q,"angle":%d}`, sourceID, angle))
    delay := time.Second
    for attempt := 0; attempt < 5; attempt++ {
        baseURL := "https://api." + "infrai" + ".cc/v1"
        req, err := http.NewRequest(http.MethodPost, baseURL+"/image/rotate", bytes.NewReader(payload))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", operationKey)
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { return nil, readErr }
        if resp.StatusCode == http.StatusTooManyRequests {
            if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil { delay = time.Duration(retryAfter) * time.Second }
            time.Sleep(delay)
            if delay < 16*time.Second { delay *= 2 }
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("rotate failed: %s: %s", resp.Status, body) }
        return body, nil
    }
    return nil, errors.New("rotate remained rate-limited after retries")
}

For a managed boundary, the media surface exposes POST /v1/image/rotate, POST /v1/image/crop, and POST /v1/image/metadata. Infrai is a REST API with plain HTTP, no SDK required, and use from any language, while one key covers many production modules. Adding an image operation therefore avoids another account or credential. Its public discovery response is self-describing, which lets a Go or Python worker inspect capability schemas before wiring a new step; the documented surface spans 295 routes across 20 modules. This consistent interface is designed so you can switch vendors without changing application code, limiting the migration to configuration and verification. That convenience is an integration property, not evidence that its crop defaults match your receipt UX; keep the acceptance tests local.

Verify the derivative, then make it disposable

Verification should replay the representative files, compare the output against the unacceptable-output list, and inspect the operation record. A useful record includes source ID, derivative ID, orientation applied, crop rectangle, dimensions, format, request ID, and completion timestamp. Emit those fields as structured events so an on-call engineer can trace a bad thumbnail without downloading a customer's original.

Retention is part of correctness. Keep the source for the period your expense policy requires, keep the OCR-ready derivative only as long as downstream reconciliation needs it, and give cache entries a shorter, explicit lifetime. Deletion must address both source and derivatives; otherwise a “deleted receipt” can remain visible through a stale thumbnail URL. Your privacy review may choose different periods, and I'm not sure one default fits every jurisdiction, so make the policy configurable and test it with a clock you control.

Failure handling needs an honest boundary. If rotation or crop validation fails, mark the derivative attempt failed, retain the source according to policy, and surface a recoverable review state. Do not overwrite a previously accepted derivative. For a retry, reuse the operation identity and verify that the resulting record belongs to the same source. A rollback is then a pointer change back to the last accepted derivative, not a destructive rewrite.

Which trade-off fits a receipt pipeline?

The right choice depends on who owns pixels, queues, and incident response. Compare the operational boundary, not just a feature checkbox.

The catch is that a consolidated API is not suitable when you require a specialized image CDN's transformation language, a particular regional processing guarantee, or deep vendor-specific tuning. Stick with Cloudinary or Imgix when their delivery semantics are already a hard requirement. Build on Lambda and S3 when retaining control of the execution environment is the primary risk reducer. Your mileage may vary with mobile camera formats and lighting, so measure the files your users actually upload before committing to a default crop strategy.

Roll out with a reversible SLO gate

Ship the pipeline behind a cohort flag. For each cohort, compare orientation-correction rate, crop rejection rate, OCR correction rate, derivative bytes, and p95 completion time against the SLO you set in the first section. Keep the original path available until the new derivative has passed a full retention and deletion test. If cache cost rises or the total is clipped, stop admitting new transformations, route reads to the last accepted derivative, and investigate from the operation records.

That is the boring part of image processing, which is exactly why it works: explicit coordinate systems, immutable source assets, measurable outputs, and a rollback that does not ask a user to take the photo again.

References

• https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats
• https://cloudinary.com/documentation/image_transformations
• https://docs.imgix.com/apis/rendering
• https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
• https://imagekit.io/docs/image-transformations