Your database transaction rolled back. Did your webhook fire anyway?

You've probably shipped this bug without knowing it.

Your app processes an order. It inserts a row into the database, fires a webhook to your partner's API, and both succeed. Except then something goes wrong — a constraint violation, a network error, your own code throwing an exception — and the database transaction rolls back.

The order never existed. But the webhook already fired.

Your partner received a notification for a ghost order. They shipped a package to a customer who never paid. You have no record that any of this happened.

This is the phantom webhook problem. It happens silently, in production, and most webhook implementations have it.

Why it happens

The standard webhook implementation looks like this:

1. Begin transaction
2. INSERT order INTO orders
3. Commit transaction
4. POST /webhook to partner  ← happens AFTER commit

Seems safe, right? The commit succeeded before the webhook fires.

But there are three failure modes nobody thinks about:

Failure mode 1: The crash window

1. INSERT order ──────────────────→ success
2. COMMIT ────────────────────────→ success
3. [process crashes / pod restarts]
4. POST /webhook ─────────────────→ never happens

The order exists in your database. The webhook never fired. Your partner never knows about it.

Failure mode 2: The timeout race

1. INSERT order ──────────────────→ success
2. COMMIT ────────────────────────→ success
3. POST /webhook ─────────────────→ timeout (partner slow)
4. Retry POST /webhook ───────────→ success (duplicate!)

You sent the webhook twice. Your partner charged the customer twice.

Failure mode 3: The silent rollback

1. INSERT order ──────────────────→ success
2. INSERT order_items ────────────→ constraint violation ← second insert fails
3. ROLLBACK ──────────────────────→ order gone from DB
4. [but you already called POST /webhook in step 1.5]

This one happens when the webhook call is buried inside a larger transaction that later rolls back. The webhook fired for data that doesn't exist.

The transactional outbox pattern

The fix is to stop treating the webhook as a separate side effect and make it part of the same atomic operation as your business data.

❌ What you're doing now:

  ┌─────────────────────────────┐
  │  Transaction                │
  │  ├── INSERT order           │
  │  └── COMMIT                 │
  └─────────────────────────────┘
         │
         └── POST /webhook  ← outside the transaction
                               can fail independently
                               can never happen
                               can happen twice


✅ Transactional outbox:

  ┌─────────────────────────────────────────┐
  │  Transaction                            │
  │  ├── INSERT order                       │
  │  ├── INSERT webhook_event  ← same tx   │
  │  └── COMMIT (or ROLLBACK atomically)    │
  └─────────────────────────────────────────┘
         │
         └── Background worker (always running)
                │
                ├── reads webhook_events table
                ├── POST /webhook
                ├── retries on failure (exponential backoff)
                └── dead letter queue after N failures

Either both the order and the webhook event exist — or neither does. The background worker handles the actual HTTP call separately, with retry logic and a full audit log.

This pattern has a name — the transactional outbox — and it's been standard practice in Java and Go shops for years. The Rust ecosystem didn't have a library for it until now.

webhooksmith: drop-in transactional outbox for Rust

I built webhooksmith to solve this for Rust + Postgres apps. Add it to your project:

[dependencies]
webhooksmith = "0.1"
tokio = { version = "1", features = ["full"] }
serde_json = "1"

Setup takes about 10 lines:

use webhooksmith::WebhookEngine;
use serde_json::json;

let engine = WebhookEngine::builder()
    .database_url("postgres://user:pass@localhost/mydb")
    .build()
    .await?;

engine.migrate().await?;  // creates 3 tables, safe to call on startup

let endpoint = engine
    .register("https://partner.example.com/webhooks", "your-secret-key")
    .await?;

Here's the outbox pattern with webhooksmith:

let mut tx = engine.pool().begin().await?;

// Your business logic
sqlx::query!("INSERT INTO orders (id, total) VALUES ($1, $2)", order_id, total)
    .execute(&mut *tx)
    .await?;

// Webhook in the SAME transaction
engine
    .send_in_tx("order.created", json!({"id": order_id, "total": total}), endpoint.id, &mut tx)
    .await?;

tx.commit().await?;
// If this commit succeeds: order exists AND webhook is queued
// If this commit fails: order doesn't exist AND webhook was never queued
// No phantom webhooks. No silent drops.

Start the background worker at the end of your main():

// Delivers events, retries failures, moves exhausted events to DLQ
engine.run().await;

// Or gracefully drain before shutdown:
engine.run_graceful(async {
    tokio::signal::ctrl_c().await.ok();
}).await;

What happens when delivery fails

The worker retries with exponential backoff. If a partner endpoint is down for 6 hours, events wait in the database and are delivered when it comes back — without any code from you.

Attempt 1  ──→ partner returns 503 ──→ retry in ~2s
Attempt 2  ──→ partner returns 503 ──→ retry in ~4s
Attempt 3  ──→ partner returns 503 ──→ retry in ~8s
...
Attempt 10 ──→ partner returns 503 ──→ moves to dead letter queue

You can inspect the DLQ and requeue events:

// See what failed
let dead = engine.dead_events(endpoint.id).await?;
for event in &dead {
    println!("{}: {} attempts", event.event_type, event.attempts);
}

// Requeue all of them
let requeued = engine.retry_all_dead(endpoint.id).await?;
println!("Requeued {} events", requeued);

There's also a delivery log for every event — you can see exactly what was sent, when, the HTTP response status, and how long each attempt took.

Sending to multiple partners at once

If you have multiple endpoints (multiple partners, multiple environments), broadcast delivers to all of them atomically:

// One call = one event per endpoint, all in one SQL statement
let events = engine.broadcast_in_tx("order.created", json!({"id": order_id}), &mut tx).await?;
println!("Queued {} webhooks across {} endpoints", events.len(), events.len());

Idempotency for retrying callers

If your own code retries on network errors, you might call send twice for the same event. Idempotency keys prevent duplicates:

// Safe to call multiple times — only one webhook is ever created
engine
    .send_idempotent("order.created", payload, endpoint.id, "order-1001-created")
    .await?;

The second call returns the same event that was created on the first call. The partner receives exactly one webhook.

What gets sent to the partner

The worker sends an HTTP POST with HMAC-SHA256 signatures that partners can verify:

POST https://partner.example.com/webhooks
Content-Type: application/json
x-webhooksmith-signature: v1,a3f4b2...
x-webhooksmith-timestamp: 1735689600
x-webhooksmith-event-id: 550e8400-e29b-41d4-a716-446655440000
x-webhooksmith-event-type: order.created

{"id": 1001, "total": 49.99}

If you're also building the receiving side with axum, webhooksmith-axum verifies signatures automatically:

use axum::{Router, routing::post, http::StatusCode};
use webhooksmith_axum::{WebhookSecretLayer, VerifiedWebhook};

async fn receive(VerifiedWebhook(payload): VerifiedWebhook) -> StatusCode {
    println!("{}: {:?}", payload.event_type, payload.body);
    StatusCode::OK
}

let app: Router = Router::new()
    .route("/webhooks", post(receive))
    .layer(WebhookSecretLayer::new("your-secret-key"));

The extractor automatically rejects replayed requests (5-minute timestamp window), requests with wrong signatures, and bodies over 1 MB.

Monitoring queue health

let stats = engine.queue_stats().await?;
println!("pending={} delivering={} failed={} dead={}",
    stats.pending, stats.delivering, stats.failed, stats.dead);

Set up an alert if dead > 0 — that means events have exhausted all retries and need manual attention.

What it requires

Rust 1.75+
An existing Postgres 14+ database (the one your app already uses — no new infrastructure)
No Redis, no SQS, no external queue service

Get started

cargo add webhooksmith

→ webhooksmith on crates.io

→ Source + examples on GitHub

→ webhooksmith-axum for receiving side

If you're building a SaaS with partner webhooks, or any system where "fire and forget" isn't acceptable, this gives you the outbox pattern in about 10 lines of code.

The ghost order problem is real — I've seen it in production. The outbox pattern is the right fix. This is the implementation I wished existed when I first ran into it.