Have you ever watched a memory graph climb and known, with absolute certainty, that a queue was the culprit? That was me last month. I built a real-time translation pipeline that chained two streaming AI models, and the first run crashed with an out-of-memory error.
Model A used MonkeyCode's free server option to stream English tokens, and Model B translated them into Chinese on my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The models weren't slow; they were too fast for my architecture, and the unbounded queue between them swallowed every byte until the process died.
The Symptom: A Memory Curve That Only Went Up
The pipeline looked simple on paper. Model A streamed English sentences, I pushed each chunk into an asyncio.Queue, and Model B consumed those chunks and translated them into Chinese. I expected the queue to act as a smooth buffer between two uneven speeds, and for the first few seconds, it did exactly that. Then the memory graph started climbing, and it never came back down.
queue = asyncio.Queue() # unbounded: the silent killer
The unbounded queue was the entire problem. When Model B paused to think or hit its own rate limit, Model A kept producing, and every chunk landed in the queue. Python queues don't have a natural size limit, so the process just kept allocating memory until the OS stepped in and killed it. I checked the logs, and the last line was a MemoryError, followed by the sound of my container restarting.
The Diagnosis: Producer Speed vs. Consumer Speed
I ran the pipeline with memory_profiler and watched the allocation pattern in real time. The queue size grew by roughly 1.5 MB per second while Model B was translating, and it only stopped growing when the producer finished its turn. The asymmetry was obvious: the producer emitted a token every 80 milliseconds, but the consumer needed 250 milliseconds per token. Why did the queue grow by 170 milliseconds of work every second? Because the producer never waited for the consumer.
Here's the diagnostic workflow I should have used from the start:
- Log the queue size on every put and get, then graph it over time.
- Measure the producer's tokens-per-second and the consumer's latency distribution.
- If the queue grows monotonically, you have a producer-consumer mismatch, not a memory leak.
I didn't notice this in testing because my test prompts were short. A 50-word English sentence produces maybe 200 tokens, and the queue never had time to grow. The real workload was a 2,000-word article, and that gave the queue plenty of time to consume every available megabyte. This is the classic difference between testing for correctness and testing for sustained throughput.
The Fix: A Bounded Queue and a Blocking Producer
The solution came from an unexpected place: TCP's sliding window. TCP doesn't let the sender dump unlimited data; the receiver advertises a window size, and the sender must stop when the window is full. I applied the same idea to my pipeline with a single line of code:
queue = asyncio.Queue(maxsize=16)
That one parameter changed everything. When the queue is full, await queue.put(chunk) blocks, which pauses the producer's async generator, which stops reading from the network, which tells the upstream server to slow down. The backpressure propagates all the way from the consumer to the model API, and the memory usage flatlines.
import asyncio
async def produce(stream, queue):
async for chunk in stream:
await queue.put(chunk) # blocks when queue is full
print(f"[producer] queued, size={queue.qsize()}")
async def consume(queue):
while True:
chunk = await queue.get()
translated = await model_b_translate(chunk) # pseudocode
print(f"[consumer] {translated}")
queue.task_done()
async def main():
queue = asyncio.Queue(maxsize=16)
stream = model_a_stream() # pseudocode: async generator
producer = asyncio.create_task(produce(stream, queue))
consumer = asyncio.create_task(consume(queue))
await asyncio.gather(producer, consumer)
The beautiful part is that the fix didn't require explicit coordination. The async runtime handles the blocking and resuming automatically, and the backpressure chain works in both directions. If Model B slows down, the queue fills up, the producer pauses, and the upstream server eventually sees a slow reader and throttles its own output.
A Decision Table for Queue Sizing
Queue size is a tradeoff, not a constant. Here's how I think about it now:
| Queue size | What it protects | The cost |
|---|---|---|
| 1–4 | Minimal latency, strict ordering | Producer stalls often, low throughput |
| 8–32 | Smooth bursts, good throughput | Some latency under spikes |
| 64+ | High throughput, burst tolerance | Memory grows, backpressure weakens |
| Unlimited | Never blocks the producer | OOM, the original bug |
The right number depends on your consumer's speed variance. If your consumer is a model with a 10x latency range, you need a larger queue to absorb the spikes. If your consumer is a simple string transform, a tiny queue is fine. Measure the p95 consumer latency, multiply by the producer's tokens per second, and size the queue to cover that product.
Limitations and Who Should Skip This
Backpressure isn't a universal cure. Some streaming APIs don't respect slow readers; they buffer the entire response server-side and dump it when the connection closes. In that case, a bounded queue just shifts the memory problem from your process to the server's, and you need a different strategy: request smaller batches, use a non-streaming endpoint, or add a sampling layer.
This approach also assumes a single producer-consumer pair. If you have a fan-out pattern, where one model feeds three downstream tasks, the queue math gets more complicated, and you need per-consumer buffers plus a coordination layer. And if your consumer can crash and restart, an unbounded queue with a disk-backed store might actually be the right call for durability, even though it uses more memory.
The deeper lesson is that streaming pipelines are networks, and networks need flow control. TCP figured this out in 1974, and I rediscovered it in 2026 by watching a container die. If you've ever seen a memory graph climb during a streaming workload, I'd love to hear how you traced it back to the queue.
MonkeyCode provides free models that can run this workflow.