How VarMQ Processes Millions of Messages with an Event Loop and Worker Pool

Processing a million messages is easy if time and memory don't matter. The hard part is doing it without paying a big dispatch cost per message, without letting concurrency run wild, and without spinning up a fresh goroutine or a fresh allocation every time something arrives.

VarMQ tackles that with two pieces that cooperate:

• One event loop decides when the next job may start.
• A reusable worker pool runs jobs concurrently, up to a limit you configure.

The queue sits between the producers and these two pieces. Producers can add work as fast as they want, but the event loop controls how quickly that work reaches the pool. That separation keeps the submission path cheap, and it means a burst of messages never becomes an equally large burst of goroutines.

The processing path

Here's what a message goes through inside an in-memory VarMQ worker:

Adding a message never creates a goroutine. It builds a job, enqueues it, and sends a wake-up notification to the worker. Only the event loop can admit queued jobs into execution inside the worker pool.

The event loop dispatches; it does not execute

The core loop is small enough to understand in one pass. This is a shortened version of the event loop

for {
    select {
    case <-ctx.Done():
        // Wait for in-flight work, clean up the pool, and stop.
        return

    case <-signal:
        for w.IsActive() &&
            w.curProcessing.Load() < w.concurrency.Load() &&
            w.queues.Len() > 0 {
            cont, err := w.processNextJob()
            if err != nil {
                w.sendError(err)
            }
            if !cont {
                break
            }
        }
    }
}

When there's nothing to do, the loop just blocks in select. It doesn't poll the queue, and it doesn't burn a CPU core while idle.

Once signaled, it keeps dispatching until one of three conditions changes:

1. The worker is paused or stopping.
2. The configured concurrency limit has been reached.
3. Every bound queue is empty.

So a single wake-up can dispatch many jobs. There's no one-to-one relationship between messages and event-loop wake-ups.

The signal channel has a capacity of one, and notifications use a non-blocking send:

select {
case w.eventLoopSignal <- struct{}{}:
default:
}

If 10,000 messages arrive while a wake-up is already waiting, the extra notifications collapse into the pending signal. The event loop wakes up, inspects the real queue length, and fills every available processing slot. That's how a burst of submissions avoids becoming thousands of blocked signal senders.

Workers send the same notification when they finish. Releasing a processing slot decrements the in-flight counter and wakes the loop if more capacity is available. The result is a short feedback cycle:

message arrives -> event loop fills pool -> worker finishes -> slot released -> event loop fills slot

The pool bounds concurrency

The event loop dispatches serially, but handlers run concurrently inside pool nodes. Each node owns a small channel and a goroutine that serves it. When the event loop dispatches a job, it removes an idle node from the pool and sends the job to it.

If no idle node exists, VarMQ creates another one. The event loop's curProcessing < concurrency check keeps this from running past the configured limit. A queue may hold a million messages, but a worker configured with concurrency 32 will execute at most 32 at a time.

When a handler returns, its node follows one of two paths:

• It goes back into the idle list when more work is waiting or the configured idle-worker policy wants to keep it warm.
• It stops, and its linked-list node goes into a sync.Pool cache for later reuse.

You can tune concurrency while the worker is running. Increasing it wakes the event loop immediately. Decreasing it changes admission for new jobs and lets excess idle workers disappear without interrupting jobs already in flight.

This is where VarMQ's concurrency comes from. The event loop does a small amount of coordination; the pool spends its time inside application handlers.

Where the memory savings come from

"Low memory usage" here means low overhead per message and bounded execution resources. It does not mean an unlimited backlog somehow consumes constant memory.

VarMQ trims overhead in a few places.

Chunked queue storage (Default standard Queue)

The standard FIFO queue stores jobs in linked buffer chunks instead of repeatedly growing and copying one large slice or a linked list node for each message. It starts with space for 1,024 entries. When a chunk fills, the next chunk grows by 50 percent, up to the configured maximum chunk size.

Enqueue and dequeue remain amortized O(1). Crossing a chunk boundary advances a pointer rather than copying unread messages into a new array. A dequeue also clears the consumed slot, so the queue doesn't keep a reference to a completed job and its payload.

The payoff: no large copy spikes when a single backing array grows under a heavy backlog.

Reused worker nodes

Pool nodes move between the busy state, the idle linked list, and a sync.Pool cache. Reusing those nodes means the bookkeeping around a worker doesn't have to be rebuilt every time traffic rises after an idle period.

The optional idle expiry policy also lets an application pick its memory and latency tradeoff. Keeping idle workers warm reduces dispatch latency. Expiring excess workers reduces retained pool state during quiet periods. VarMQ always keeps at least the configured minimum idle ratio, with one idle worker as the default minimum.

Small coordination state

The worker tracks its status, concurrency, and current processing count with atomics. Locks still protect queue chunks, pool lists, and condition-variable transitions, so the design is not lock-free. But the hot checks in the event loop don't need a large shared scheduler object or a goroutine for every queued message.

VarMQ also offers different job types for different needs. A fire-and-forget Worker carries less result state than ErrWorker or ResultWorker. Choosing the smallest useful abstraction shows up directly in the allocation numbers.

VarMQ under load: a head-to-head benchmark against PondV2

The repository's own microbenchmark - a trivial handler, one CPU, concurrency one - measures VarMQ's scheduler overhead in isolation (roughly 1.01 million round trips per second for single adds, 1.54 million for batches). To see how the design behaves under realistic load - many producers, different handler shapes, and different CPU counts - we ran a head-to-head comparison against PondV2, a popular Go worker pool with a feature set close to VarMQ's. VarMQ's design was partly inspired by PondV2, which makes the comparison fair and informative.

The full benchmark suite lives in the varmq-benchmarks repository. Interactive charts for every result, grouped by workload and CPU configuration, are published at goptics.github.io/varmq-benchmarks.

Each run exercises the complete path from submitting tasks to waiting for completion. The suite covers:

• three task shapes - Sleep10ms (I/O simulation), Loop10 (CPU-bound math), and Sleep5Loop5 (mixed);
• five producer patterns - from 1 user submitting 1M tasks (1u-1Mt) to 1M users submitting 1 task each (1Mu-1t);
• pool sizes of 50k, 100k, and 300k workers;
• three CPU configurations - 4, 8, and 24 (the i7-13700's default GOMAXPROCS).

All runs used the same machine as the microbenchmark: an Intel Core i7-13700 on Linux/amd64. The tables below show representative results; the charts hold the full set.

4 CPUs

8 CPUs

24 CPUs

What the numbers show:

• VarMQ won every workload at every CPU count. The typical throughput advantage is roughly 2–2.5x, and the widest gap is 6.2x - the CPU-bound Loop10 workload with one million producers at 8 CPUs.
• Memory use stays about 2.4–2.5x lower and allocations about 3.4–3.5x lower regardless of CPU count, which matches the queue chunking and node reuse described above.
• The 1Mu-1t pattern many producers, one task each - is where PondV2 degrades most. That is exactly the case VarMQ's design targets: Add never creates a goroutine, and producer wake-ups coalesce into one event-loop signal.
• Both libraries slow down at 24 CPUs - the i7-13700 has 16 physical cores and 24 threads, so hyper-threading adds scheduling pressure - but VarMQ keeps its lead.
• The one near-parity result is Sleep5Loop5 at 24 CPUs with a million producers, where the two libraries land within a few percent of each other. Everything else is a clear VarMQ win.

As with the microbenchmark, these are derived rates from ns/op on one machine, with the benchmark harness overhead included. Throughput on your hardware and handler shape will differ. The charts let you compare any workload across all three CPU configurations, and the repository documents how to re-run the suite yourself.

Backlogs still need a memory budget

Bounded concurrency protects execution, not queue length. If producers submit work faster than handlers complete it, pending messages accumulate - and every pending job and its payload has to live somewhere.

For a service that may receive sustained bursts, set a queue capacity and decide what the producer should do when Add returns false. Depending on the application, that may mean retrying with backoff, shedding low-priority work, writing to a durable queue, or applying backpressure earlier in the request path.

That's how low per-message overhead turns into predictable memory usage. Without a capacity, even an efficient queue can exhaust memory when the arrival rate stays above the completion rate long enough.

Graceful shutdown keeps concurrency manageable

The same event loop owns shutdown coordination. Cancelling the worker context moves the worker to stopping, waits for in-flight jobs, removes pool workers, broadcasts the final state, and exits the event-loop goroutine.

Pause works differently. It stops admission of new jobs while allowing current handlers to finish. Resume returns the worker to idle and signals the event loop so queued work starts again. These state transitions let an application control traffic without abandoning jobs that are already running.

The useful idea is separation

VarMQ gets high throughput by keeping responsibilities narrow:

• Producers enqueue and send a cheap, coalesced notification.
• The event loop admits work only while capacity exists.
• Pool workers run handlers concurrently and report completion.
• Chunked queues and reusable pool nodes keep allocation pressure under control.

The event loop doesn't make handlers faster; it makes dispatch predictable. The pool doesn't make the queue smaller; it keeps active concurrency bounded. Together, they let VarMQ process large message volumes without creating execution resources in proportion to the backlog.

That is how a small Go scheduler can move more than a million trivial messages per second in a microbenchmark while still giving production applications explicit control over concurrency, memory, and shutdown behavior.