Launch HN: HyperProbe (YC S26) – Agents that do read-only debugging in prod
High error rate on order status. 23% of requests failing.
PagerDuty fires. GET /api/orders/{id}/status is returning 500 for nearly a quarter of requests. 847 failures in the last 10 minutes. No exception in the logs.
GET /api/orders/{id}/status · 500 · 23% error rate
847 failures / 10 min · threshold exceeded
HyperProbe follows the trace chain. Identifies a silent write failure upstream.
HyperProbe reads the distributed traces and follows the failure chain. Order service is healthy. Payment service downstream is returning 404. Payments exist in the payment gateway but are not in the system.
|
└── GET payment-service/api/getPaymentsByOrder/{orderId} 404
Payments exist in the payment gateway. Not found in the system. A write failed silently somewhere upstream.
HyperProbe places a virtual breakpoint on the webhook handler.
HyperProbe identifies payments are recorded when the payment gateway calls a webhook. A virtual breakpoint placed on the webhook handler at /src/api/webhooks.ts line 78. No redeploy. Service keeps running.
file: /src/api/webhooks.ts · line 78
Non-blocking · Read-only · No redeploy
Snapshot captures live request at the exact moment the webhook fires.
Gateway is sending PENDING. The code has no case for it. Idempotency check marks payment as processed before confirming state. Payment never written to DB. No exception fires.
duplicate = null ← first time seen, passes through
db.insert → never called
redis.set → called anyway, payment locked out permanently
Payment gateway started sending PENDING, a status your code never handled. Idempotency key written before state is checked. Payment marked processed, never recorded.
Root cause confirmed. Fix ready. 5 minutes from alert.
Payment gateway started sending PENDING, a status your code never handled. Idempotency key written before state is checked. Payment marked processed, never recorded.
Source: hackernews