Fair Queue for a Shared Free AI Server: 5-Dev Postmortem

Five independent clients on one free AI server will produce 429s and a thundering herd unless you add a fair queue. We fixed it with a client-side asyncio queue that capped concurrency at two, prioritized interactive work, and dropped 429s from 23 to 0 on a 100-request mixed workload.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What Failed When Five Developers Shared One Server

We shared one MonkeyCode free server for code review and refactoring. Each of us ran our own scripts. Nobody coordinated. The first symptom was latency: requests that took two seconds started taking thirty. Then came the 429s. Then came the retries. Retries made everything worse. The server spent more time rejecting requests than answering them.

The timeline compressed quickly:

• Day 1: two developers, no issues
• Day 3: four developers, latency doubles
• Day 5: five developers, 429s appear
• Day 6: retries cause a thundering herd
• Day 7: the team stops using the server

The root cause was not the server. It was the absence of coordination. Five independent clients hammered one endpoint. Each client assumed it was the only user. The server had no way to prioritize. HTTP 429 is the standard “too many requests” signal; we treated it as a retry cue instead of backpressure. That is how a shared free endpoint turns into a retry storm.

The deeper problem was architectural. Each of us built a separate integration. Each integration had its own retry logic. Under load those retries multiplied. The server received about five times the intended traffic, not because we needed five times the work, but because five clients were guessing independently.

Contrast the two modes we actually ran:

• Uncoordinated: five scripts, five retry loops, unbounded in-flight calls, no shared view of queue depth.
• Coordinated: one process, one priority heap, two in-flight calls, explicit rejection when the queue is full.

The first mode failed in a week. The second mode is what we shipped.

How We Built a Client-Side Fair Queue

The fix is a client-side queue. All requests flow through one process. That process assigns priority, enforces concurrency, and tracks usage. The queue becomes the single point of coordination.

Design goals we actually shipped:

• Priority for interactive requests over batch jobs
• FIFO ordering within the same priority
• Maximum two concurrent requests
• Explicit rejection when the queue is full

Implementation

The queue wraps any model-calling function. Callers pass a coroutine and a priority. Interactive requests use priority 1. Batch jobs use priority 2. A min-heap guarantees ordering via Python’s heapq. An asyncio.Semaphore bounds concurrency. Together they give us fair order plus a hard cap on in-flight work.

# fair_queue.py — coordinated access to a shared AI server
import asyncio
import heapq
import time

class QueueFullError(Exception):
    pass

class FairQueue:
    def __init__(self, max_concurrency=2, max_queue=20):
        self.semaphore = asyncio.Semaphore(max_concurrency)
        self.heap = []
        self.counter = 0
        self.max_queue = max_queue
        self.metrics = {"submitted": 0, "completed": 0, "rejected": 0}

    async def submit(self, call_fn, priority=2, *args, **kwargs):
        if len(self.heap) >= self.max_queue:
            self.metrics["rejected"] += 1
            raise QueueFullError("Queue is full. Retry later.")
        self.counter += 1
        self.metrics["submitted"] += 1
        heapq.heappush(
            self.heap,
            (priority, self.counter, time.time(), call_fn, args, kwargs)
        )
        return await self._run_next()

    async def _run_next(self):
        async with self.semaphore:
            _, _, _, call_fn, args, kwargs = heapq.heappop(self.heap)
            try:
                result = await call_fn(*args, **kwargs)
                self.metrics["completed"] += 1
                return result
            except Exception:
                self.metrics["completed"] += 1
                raise

Read the tuple as (priority, counter, timestamp, ...). Lower priority numbers run first. The monotonic counter breaks ties so two priority-1 reviews stay FIFO. The semaphore is the only thing that talks to the network at once; everything else waits in the heap.

Integration pattern

The queue replaces direct model calls. A typical integration looks like this:

# usage.py — route all model calls through the queue
queue = FairQueue(max_concurrency=2, max_queue=20)

async def review_code(diff_text):
    # priority 1: interactive code review
    return await queue.submit(call_model, priority=1, prompt=diff_text)

async def refactor_batch(files):
    # priority 2: background refactoring
    for f in files:
        await queue.submit(call_model, priority=2, prompt=f"Refactor {f}")

The key rule is simple. Every model call goes through the queue. No exceptions. One direct call bypasses the queue, the queue loses its view of the traffic, and the thundering herd returns.

Practical rollout steps we used:

1. Put FairQueue in a shared module, not copied into each script.
2. Start with max_concurrency=2 and max_queue=20 for a five-person team.
3. Route interactive reviews at priority=1 and batch refactors at priority=2.
4. Catch QueueFullError and fail fast instead of retrying immediately.
5. Log submitted, completed, and rejected on every run.

Fail-fast matters. Immediate retries after a full queue recreate the herd we were trying to kill. Back off, or drop the batch job, before you retry.

What Changed After We Queued Every Call

We ran the same workload with and without the queue. The workload was 100 mixed requests. The results were stark:

The queue did not make the server faster. It made us slower to overwhelm it. Bounded concurrency prevented the thundering herd. Prioritization kept interactive requests responsive while batch refactors waited.

Three numbers explain queue health:

1. Wait time — time from submit to execution start
2. Queue depth — pending requests at any moment
3. Rejection rate — requests refused because the queue is full

Log these per request. A rising wait time means the team is outgrowing the server. A rising rejection rate means the queue size is too small. Both signals appear before total failure—unlike 429 storms, which appear as total failure.

A useful comparison while you watch those numbers:

• If wait time rises and depth stays low, concurrency is the bottleneck. Do not raise it blindly on a free server.
• If depth rises and rejections stay at zero, the queue is absorbing burst traffic. That is the intended behavior.
• If rejections rise, either shrink batch jobs or stop treating the free server as a cluster.

Who Should Copy This Pattern—and Who Should Not

Use this decision table as a starting point, not a promise:

The table encodes a simple rule. The more clients, the more coordination. A free server is not a production cluster. It is a shared resource with limits.

Limitations we will not paper over:

• The queue is client-side. It cannot fix server-side throttling.
• It cannot increase the token grant.
• It adds wait time for interactive requests when batch work is already queued.
• It does not improve model quality.
• The 10 million token grant and free server availability are operator-reported figures. Verify them in the official repository before planning around them.

Who should use this: teams sharing a free server; developers who have seen 429s from a shared endpoint; anyone who wants a single view of AI usage.

Who should not: solo developers with light usage; production workloads that need SLAs; teams that need model-level isolation. A queue cannot fix a server that is genuinely overloaded. It can only stop clients from making it worse.

The free server failed because five clients acted independently. The queue fixed the failure by making them act as one. Coordination is cheaper than retries. Metering is cheaper than surprises.

Next step: copy FairQueue into your shared tooling, route every model call through it for one week, and log wait time, queue depth, and rejection rate. If wait time climbs while concurrency is already at two, you have outgrown the free server—not the queue. Drop your three metrics in the comments so other small teams can compare.

MonkeyCode provides free models that can run this workflow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.