The Context
I wasn't part of the original team. I didn't know their codebase, their business logic, or their deployment process. I was brought in because the internal team had tried everything and the system was still failing daily.
A legacy PHP application served as the backend for a marketing dashboard. Business users would select a large segment of customers and click a button (e.g., "Send Campaign" or "Mark as Processed").
The Original Flow:
- User clicks the dashboard button.
- PHP receives the HTTP request.
- PHP executes a
SELECT * FROM users WHERE status = 'pending'(fetching up to 1 million rows). - PHP attempts to process these rows and update a
statusflag synchronously. - The HTTP request times out (30s), the database CPU spikes to 100%, and the system fails daily.
The application had no message queue and no asynchronous worker. The database crash caused partial updates, leading to duplicate actions on subsequent retries.
Why Re-Architecture Wasn't Possible
When you're staring at a burning production system, the ideal solution—a full rewrite, microservices, Kubernetes, event sourcing—is a luxury you simply don't have. Here is why we couldn't just "do it properly":
1. The System Was 8 Years Old
The PHP monolith had accumulated years of business logic, edge cases, and undocumented behavior. No one on the team fully understood every code path. A rewrite would have taken months and introduced new bugs.
2. No Automated Test Coverage
There were no integration tests, no unit tests, no staging environment that mirrored production. Any change to the core business logic was a gamble. We had to minimize the surface area of our changes.
3. The Business Couldn't Wait
The system was failing daily at 3 PM. Marketing campaigns were delayed, users were getting duplicate SMS, and support tickets were piling up. The business needed a fix now, not in 6 months.
4. Infrastructure Was Constrained
- The database and application ran on the same 64GB server.
- There was no budget for additional RDS instances, read replicas, or load balancers.
- The hosting environment was locked down—installing new tools required weeks of approvals (if they were even allowed).
5. Team Skills Were Specialized
The existing team was PHP-heavy. Introducing a new language or a complex orchestration tool would have created a knowledge gap and maintenance burden. We needed a solution that was simple enough to hand over.
6. Risk Aversion
The company had been burned by previous "big bang" rewrites that failed. Management was highly risk-averse. Any solution had to be incremental, reversible, and low-risk.
The Bottom Line
We couldn't rewrite the monolith. We couldn't introduce Kubernetes. We couldn't wait for a greenfield project. We needed a minimally invasive, high-impact fix that could be deployed in days, not months.
The Fix: Decoupling the Request Lifecycle
To prevent the HTTP timeout and stabilize the database, we decoupled the request:
-
PHP (Producer): Accepts the request, selects the user IDs, publishes them to a queue, and returns
202 Acceptedimmediately. - RabbitMQ: Acts as a persistent buffer, absorbing the load spike.
- Go (Consumer): A stateless, dedicated worker that consumes messages in batches and performs atomic database updates.
┌─────────────┐ ┌──────────────┐ ┌──────────────────────┐
│ Dashboard │───HTTP─▶│ PHP Backend │──────▶│ RabbitMQ │
│ (User) │◀──202──│ (Producer) │ │ (Buffer) │
└─────────────┘ └──────────────┘ └──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Go Worker │
│ (Batch Updater) │
└──────────┬───────────┘
│
▼
┌──────────────┐
│ PostgreSQL │
│ (Stable) │
└──────────────┘
Why this was the right minimal intervention:
- PHP changes: Minimal (~50 lines to publish to RabbitMQ).
- Go service: A single binary, easy to deploy, no dependencies.
- RabbitMQ: Already available in the infrastructure (used for other purposes).
- Rollback plan: If it failed, we could simply stop the Go service and revert the PHP changes in minutes.
The Initial Implementation (Static)
Here is the core logic that was deployed to stop the daily crashes. It is intentionally simple, stateless, and idempotent.
package main
import (
"database/sql"
"time"
)
func processBatch(messages []StatusMessage) error {
// 1. Fixed batch size of 50,000 messages.
batch := messages[:50000]
// 2. Begin an atomic transaction. (ALL or NOTHING)
tx, err := db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
// 3. Process updates within the transaction.
for _, msg := range batch {
// CRITICAL: The WHERE clause enforces idempotency.
_, err := tx.Exec(`
UPDATE users
SET status = 'processed'
WHERE id = $1
AND status = 'pending' -- << Guard Rail
`, msg.UserID)
if err != nil {
// Rollback the entire 50k batch on any error.
return err
}
}
// 4. Commit the batch atomically.
if err := tx.Commit(); err != nil {
return err
}
// 5. Acknowledge messages to RabbitMQ ONLY after commit.
for _, msg := range batch {
msg.Ack()
}
// 6. Static Backpressure: Sleep for 2 seconds.
// Without this, the DB CPU saturates at 100%.
time.Sleep(2 * time.Second)
return nil
}
Why This Worked (The Principles)
1. Idempotency
RabbitMQ delivers at-least-once. The WHERE status = 'pending' ensures replaying the same batch is a no-op. Zero duplicate updates.
2. Atomicity
Wrapping 50k updates in a single transaction guarantees consistency. If the service crashes mid-batch, the DB rolls back, and the messages remain unacked for safe redelivery.
3. Backpressure
The static time.Sleep(2s) gave the PostgreSQL wal_writer time to flush, dropping CPU from 100% to a stable 68%.
4. Separation of Concerns
The Go worker does not read the database and does not know about business logic. It is a pure "flag updater," making it highly resilient.
Performance Outcomes
| Metric | Before (Sync) | After (Async Static) |
|---|---|---|
| HTTP Response Time | ~30s (Timeout) | < 200ms (202 Accepted) |
| DB CPU | 100% (Crash) | 65-70% (Stable) |
| Failures | Daily | Zero failures (18 months) |
Future Iterations: Moving from Static to Adaptive
The system worked perfectly for 18 months. However, if I were to build this today, I would move from a static configuration to an adaptive one.
Here is exactly how I would improve it for the next iteration:
1. Dynamic Backpressure (CPU-Aware Sleep)
Instead of a hard-coded time.Sleep(2s), we can query the database CPU utilization and adjust the throttling in real-time.
type AdaptiveConfig struct {
TargetCPU float64 // e.g., 0.7 (70%)
MaxBatchSize int // 50000
MinBatchSize int // 10000
CurrentBatch int // Starts at 50000
SleepDuration time.Duration // Starts at 2s
}
func (a *AdaptiveConfig) Adjust() {
cpu := getDBCPU() // Read pg_stat_database or /proc/stat
if cpu > a.TargetCPU {
// Back off: Smaller batch, longer sleep
a.CurrentBatch = int(float64(a.CurrentBatch) * 0.9)
if a.CurrentBatch < a.MinBatchSize {
a.CurrentBatch = a.MinBatchSize
}
a.SleepDuration = time.Duration(float64(a.SleepDuration) * 1.2)
} else if cpu < a.TargetCPU - 0.1 {
// Speed up: Larger batch, shorter sleep
a.CurrentBatch = int(float64(a.CurrentBatch) * 1.05)
if a.CurrentBatch > a.MaxBatchSize {
a.CurrentBatch = a.MaxBatchSize
}
a.SleepDuration = time.Duration(float64(a.SleepDuration) * 0.9)
}
}
Why this matters: Peak vs. off-peak hours. The system becomes self-tuning—processing faster when the DB is idle, and slowing down automatically when it gets busy.
2. Sub-Batching Inside the Transaction
A 50k-row transaction holds locks for a long time. To reduce contention, we can break the work into smaller chunks within the same atomic transaction:
func processBatch(messages []StatusMessage) error {
tx, _ := db.Begin()
defer tx.Rollback()
// Sub-batch of 1000 rows inside the 50k transaction
for i := 0; i < len(messages); i += 1000 {
end := min(i+1000, len(messages))
chunk := messages[i:end]
// Single statement for 1000 rows (faster than 1000 individual updates)
tx.Exec(`
UPDATE users
SET status = 'processed'
WHERE id = ANY($1::int[])
AND status = 'pending'
`, chunk.IDs)
}
tx.Commit() // All 50k still commit atomically
// ... ACK messages ...
}
Result: Locks are held for 1000 rows at a time, but the atomicity of the overall 50k is preserved.
3. Observability (Prometheus Metrics)
Without metrics, we were flying blind. Next time, I would export:
var (
batchDuration = prometheus.NewHistogram(...) // How long per batch
batchSize = prometheus.NewGauge(...) // Current batch size (dynamic)
dbCPU = prometheus.NewGauge(...) // DB CPU %
queueLag = prometheus.NewGauge(...) // Time since message creation
)
4. Graceful Shutdown
To avoid losing in-flight batches during deployments:
func main() {
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT)
go func() {
<-sigChan
log.Println("Shutting down gracefully...")
consumer.Stop() // Stop pulling new messages
worker.Wait() // Finish current 50k batch
os.Exit(0)
}()
}
5. Dead Letter Queue (DLX)
If a message is malformed (e.g., missing user_id), retrying it forever is wasteful. Next time, I'd configure RabbitMQ to route bad messages to a DLX for manual inspection.
The Takeaway
The initial static solution was the perfect "firefighter" tool—it stopped daily outages in 3 days. It worked within the constraints of a legacy system, a risk-averse business, and a team that needed a simple handover.
Sometimes, being an outsider is an advantage. The internal team had been fighting this problem for weeks—they were too close to see the simple solution. I walked in, looked at the data flow, identified the bottleneck, and built a minimal intervention.
No heroics. No complex architecture. Just 200 lines of Go, a 50k batch limit, and a 2-second sleep.
Simplicity works for immediate rescue. Adaptability works for long-term scale.
Key Lessons
Don't let perfection be the enemy of stability. A simple fix deployed today beats a perfect solution delivered next year.
Understand your constraints. Re-architecting wasn't possible, but isolating the bottleneck was.
Idempotency is your best friend. When dealing with message queues, always design for at-least-once delivery.
Backpressure saves databases. Sometimes, slowing down is the fastest way to finish.
Build for handover. Your fix isn't done until someone else can maintain it.
Sometimes, the outsider sees the solution first. Familiarity breeds blindness. Fresh eyes can spot the obvious bottleneck that everyone else has been walking past for weeks.