How to Isolate Media ZIP Exports — Go Multipart Storage and Signed Downloads

A large ZIP export becomes an isolation problem before it becomes a throughput problem: its multipart upload must belong to one tenant and one worker job before the application creates a signed download link, even when workers retry or two exports run concurrently.

Short answer: have the worker write each large ZIP to a unique object key with multipart upload, explicitly complete or abort that upload, and issue a short-lived presigned GET URL only after completion.

Do not treat the URL as the durable record. The durable record is the export job, its tenant-scoped object key, its upload state, and an append-only audit trail connecting the request to the completed object. The URL is a disposable capability derived from that record.

Can tenant isolation survive large ZIP export multipart upload and signed download links?

The first invariant is namespace isolation. A useful key has enough structure to support authorization and operational inspection, for example tenants/{tenant_id}/exports/{job_id}/archive.zip; the tenant and job identifiers must come from trusted job state rather than a filename supplied by a browser. Each export job gets a new key. That is important because strict conditional writes are unavailable in the Infrai storage surface, so reusing latest.zip would turn two otherwise valid workers into a last-writer-wins race. A database uniqueness constraint on the job ID, plus queue coordination, is the correct place to serialize ownership.

The second invariant is monotonic state. Model the job as pending -> uploading -> complete, with uploading -> aborted as the cleanup path. Never publish a download link from the uploading state, and never move a completed job backward. Multipart upload reduces restart pain for a large archive, but it also creates an obligation: the worker must explicitly complete or abort it. An hourly lifecycle policy cannot rescue the design because lifecycle expiry has a one-day minimum and multipart fragments are not cleaned automatically.

The third invariant is auditability. Record the object key and multipart upload identifier before transferring bytes; record every accepted part number and checksum; then record completion before minting a link. A retry should read that ledger and continue the same logical job rather than create a second user-visible export. This is an exactly-once outcome built over operations that may be attempted more than once.

Keep the failure boundary narrow.

HTTP 429 is a retryable transport outcome: honor Retry-After when present, otherwise use exponential backoff. Authorization and validation failures are terminal until the request changes. I wouldn't classify an unfamiliar 4xx from status alone; preserve its body in restricted operational logs, without the signed URL or bearer token, and let the documented contract decide whether the job can retry. I'm not sure a universal retry table is possible across every adapter, because the available evidence does not establish identical error semantics; adapter conformance tests are what would settle it.

Compare storage ownership boundaries

The architecture decision is to place multipart orchestration in the worker and authorization in the application database, then let object storage carry completed bytes. Tenant isolation, existing operational commitments, and compliance controls should choose the adapter; a feature checklist alone should not.

Infrai is a credible adapter in the bounded case because its public discovery surface reports the request schema, response schema, billing data, and runnable examples for a capability; examples are available in ten languages, so adding storage is a contract-reading exercise rather than an SDK adoption. Its separate supporting advantage is operational consolidation: 295 routes across 20 modules use one key and one bill. The catch is substantial for regulated retention: there is no object versioning or object lock, so a financial-grade immutable archive needs an external system. It also has no public-read ACL, and public_url remains null, which is correct for this private export design but rules out permanent public links and static hosting.

This is not a winner-takes-all comparison. Stick with S3 when AWS-native controls are part of the compliance case; use Google Cloud Storage when GCP ownership dominates; retain R2 when Cloudflare is already the reviewed operating boundary. Choose the simpler REST adapter only when its stated capability limits fit the threat model and recovery plan.

Test the executable contract in Go

The following Go program starts by fetching Infrai's live contract for storage.multipart.create, including its request schema and runnable examples, rather than guessing a request body that is not established here. It then runs the provider-independent state machine end to end with an in-memory adapter, uses a unique tenant/job key, records part checksums, completes before signing, and makes the cleanup obligation visible. Set INFRAI_API_BASE to the API base supplied for your account and keep the bearer key in INFRAI_API_KEY; the base is configuration because this unlinked comparison does not publish an Infrai URL. A production adapter should consume the discovered contract, while the worker logic remains unchanged.

package main

import (
    "context"
    "crypto/sha256"
    "encoding/json"
    "encoding/hex"
    "errors"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "sort"
    "strings"
    "time"
)

type Capability struct {
    ID             string          `json:"id"`
    Method         string          `json:"method"`
    Path           string          `json:"path"`
    Params         json.RawMessage `json:"params"`
    RequestExample json.RawMessage `json:"request_example"`
}

func multipartCreateContract(ctx context.Context) (Capability, error) {
    base := strings.TrimRight(os.Getenv("INFRAI_API_BASE"), "/")
    key := os.Getenv("INFRAI_API_KEY")
    if base == "" || key == "" {
        return Capability{}, errors.New("INFRAI_API_BASE and INFRAI_API_KEY are required")
    }

    endpoint := base + "/v1/discovery/storage.multipart.create"
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return Capability{}, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return Capability{}, err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            return Capability{}, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if value := resp.Header.Get("Retry-After"); value != "" {
                if parsed, parseErr := time.ParseDuration(value + "s"); parseErr == nil {
                    delay = parsed
                }
            }
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return Capability{}, ctx.Err()
            case <-timer.C:
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return Capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        var capability Capability
        if err := json.Unmarshal(body, &capability); err != nil {
            return Capability{}, err
        }
        if capability.Method != http.MethodPost || capability.Path != "/v1/storage/multipart/create/{bucket}" {
            return Capability{}, errors.New("multipart create contract does not match the reviewed route")
        }
        return capability, nil
    }
    return Capability{}, errors.New("rate limit retry budget exhausted")
}

type Part struct {
    Number   int
    Checksum string
}

type MultipartStore interface {
    Create(context.Context, string) (string, error)
    PutPart(context.Context, string, int, []byte) (Part, error)
    Complete(context.Context, string, []Part) error
    Abort(context.Context, string) error
    PresignGet(context.Context, string, time.Duration) (string, error)
}

type AuditEvent struct {
    JobID, TenantID, Action, Detail string
}

type Worker struct {
    Store MultipartStore
    Audit []AuditEvent
}

func (w *Worker) record(jobID, tenantID, action, detail string) {
    w.Audit = append(w.Audit, AuditEvent{jobID, tenantID, action, detail})
}

func (w *Worker) Export(ctx context.Context, tenantID, jobID string, zipBytes []byte) (link string, err error) {
    if tenantID == "" || jobID == "" || len(zipBytes) == 0 {
        return "", errors.New("tenant, job, and ZIP bytes are required")
    }

    key := fmt.Sprintf("tenants/%s/exports/%s/archive.zip", url.PathEscape(tenantID), url.PathEscape(jobID))
    uploadID, err := w.Store.Create(ctx, key)
    if err != nil {
        return "", fmt.Errorf("create multipart upload: %w", err)
    }
    w.record(jobID, tenantID, "upload_created", uploadID+" "+key)

    completed := false
    defer func() {
        if !completed {
            _ = w.Store.Abort(ctx, uploadID)
            w.record(jobID, tenantID, "upload_aborted", uploadID)
        }
    }()

    const demoPartSize = 8
    parts := make([]Part, 0)
    for offset, number := 0, 1; offset < len(zipBytes); offset, number = offset+demoPartSize, number+1 {
        end := offset + demoPartSize
        if end > len(zipBytes) {
            end = len(zipBytes)
        }
        part, putErr := w.Store.PutPart(ctx, uploadID, number, zipBytes[offset:end])
        if putErr != nil {
            return "", fmt.Errorf("upload part %d: %w", number, putErr)
        }
        parts = append(parts, part)
        w.record(jobID, tenantID, "part_accepted", fmt.Sprintf("%d %s", part.Number, part.Checksum))
    }

    if err = w.Store.Complete(ctx, uploadID, parts); err != nil {
        return "", fmt.Errorf("complete multipart upload: %w", err)
    }
    completed = true
    w.record(jobID, tenantID, "upload_completed", key)

    link, err = w.Store.PresignGet(ctx, key, 10*time.Minute)
    if err != nil {
        return "", fmt.Errorf("sign completed object: %w", err)
    }
    w.record(jobID, tenantID, "link_issued", "ttl=10m")
    return link, nil
}

type memoryStore struct {
    keys  map[string]string
    parts map[string]map[int][]byte
}

func newMemoryStore() *memoryStore {
    return &memoryStore{keys: map[string]string{}, parts: map[string]map[int][]byte{}}
}

func (m *memoryStore) Create(_ context.Context, key string) (string, error) {
    id := "upload-" + strings.ReplaceAll(key, "/", "-")
    m.keys[id], m.parts[id] = key, map[int][]byte{}
    return id, nil
}

func (m *memoryStore) PutPart(_ context.Context, id string, number int, body []byte) (Part, error) {
    if _, ok := m.parts[id]; !ok {
        return Part{}, errors.New("unknown upload")
    }
    copyOfBody := append([]byte(nil), body...)
    m.parts[id][number] = copyOfBody
    sum := sha256.Sum256(copyOfBody)
    return Part{Number: number, Checksum: hex.EncodeToString(sum[:])}, nil
}

func (m *memoryStore) Complete(_ context.Context, id string, parts []Part) error {
    if len(parts) == 0 || len(m.parts[id]) != len(parts) {
        return errors.New("part ledger mismatch")
    }
    sort.Slice(parts, func(i, j int) bool { return parts[i].Number < parts[j].Number })
    return nil
}

func (m *memoryStore) Abort(_ context.Context, id string) error {
    delete(m.parts, id)
    delete(m.keys, id)
    return nil
}

func (m *memoryStore) PresignGet(_ context.Context, key string, ttl time.Duration) (string, error) {
    return "https://download.example.invalid/" + url.PathEscape(key) + "?ttl=" + ttl.String(), nil
}

func main() {
    contract, err := multipartCreateContract(context.Background())
    if err != nil {
        panic(err)
    }
    fmt.Println("verified contract:", contract.Method, contract.Path)

    w := &Worker{Store: newMemoryStore()}
    link, err := w.Export(context.Background(), "studio-17", "job-8f31", []byte("PK demo media export archive"))
    if err != nil {
        panic(err)
    }
    fmt.Println(link)
    fmt.Println("audit events:", len(w.Audit))
}

The eight-byte part size exists only to make the example exercise several parts with a tiny input; production sizing belongs in the provider adapter and must follow that provider's contract. The mock URL is intentionally non-routable. In a real implementation, return the provider-generated presigned GET URL to the authenticated application, and do not attach the Infrai bearer credential to that returned URL.

There is one subtle exactly-once point here: defer protects cleanup within a process, but a killed process cannot execute it. Persist the upload identifier before the first part, run a reconciler over stale uploading jobs, and have that reconciler explicitly complete a fully recorded upload or abort it. A one-hour database scan may find stale jobs, but it must perform the abort itself; it cannot delegate fragment cleanup to an hourly storage lifecycle rule.

Govern links, retention, and restore authority

A signed URL grants temporary retrieval to whoever possesses it, so issue it only after the application rechecks that the requesting principal can access the tenant and export job. Keep its lifetime short enough for the download workflow, regenerate it after expiration, and avoid storing it as job identity. Audit link_issued with the job, principal, and expiration, but redact the query string because it carries the capability.

The object remains private. There is no reason for a media export service to depend on a public ACL, permanent direct link, or static-site behavior; if one of those is a product requirement, this design and the Infrai adapter are both the wrong fit. Browser-direct multipart upload also needs a separately reviewed CORS path. Infrai exposes no independent self-service CORS route despite a bucket model field, so keep uploads in the worker/server path described here.

Retention deserves a separate decision. Object expiration can enforce a day-scale policy, but the minimum is one day, metadata cannot be searched server-side beyond prefix listing, and there is no automatic cross-region replication or cross-cloud bulk migration tool. For a media product, store searchable export state and deletion deadlines in the application database, organize object keys by tenant and job, and test deletion reconciliation. For WORM, legal hold, recoverable overwrite, or immutable financial evidence, reject this adapter and choose an externally governed storage system that supplies the required controls.

Migrate away from the shared latest object

The rejected option is a shared latest.zip object overwritten by every job and exposed through a long-lived link. It appears convenient, but without If-Match it has no strict concurrency boundary, it erases job-level provenance, and a delayed worker can replace a newer export. That pattern remains valid only for non-sensitive, reproducible artifacts where last-writer-wins behavior is deliberate and documented. Per-tenant selected-snapshot restore is not such a case.

Migration does not require a flag day. Start writing unique job keys while continuing to read the shared object for old records, persist the selected snapshot key on every new job, and switch reads tenant by tenant after authorization and restore tests pass. Once no database row points to the shared name, remove that fallback under the ordinary retention process. This sequence preserves an auditable mapping from the selected snapshot to the bytes returned.

References and Sources

• AWS S3 object lifecycle management: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
• Google Cloud Storage documentation: https://cloud.google.com/storage/docs
• Cloudflare R2 multipart upload documentation: https://developers.cloudflare.com/r2/objects/multipart-objects/
• Go context package: https://pkg.go.dev/context
• OWASP authorization guidance: https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html