Imagine this: an AI agent answers correctly for the first ten prompts, then dies with a maximum context length exceeded error. The script was fine before. The conversation history wasn't.
If you've built anything with free-tier models, you've seen the pattern. The fix is rarely a bigger context window. It's a deliberate token budget, a memory strategy, and a retrieval plan.
This article gives you a glossary of the terms, a decision tree that picks a strategy, and a worked example for every leaf. You can test all of them on any free model provider; MonkeyCode's free model access and free server option are one convenient sandbox for exactly this workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Glossary First
Before the tree, we need precise definitions. Ambiguity here causes the worst bugs.
| Term | Definition | Why it matters |
|---|---|---|
| Token | A piece of text the model reads or writes | Every action costs tokens; every limit is token-based |
| Context window | The total tokens a model can see in one call | Hard ceiling for the entire agent |
| System prompt | Persistent instructions that never scroll away | Cheapest way to encode fixed rules |
| Conversation history | All prior messages sent to the model | Most common source of overflow |
| Truncation | Dropping the oldest messages from history | Simple, but loses information |
| Summarization | Compressing old messages into a shorter form | Keeps high-level facts at the cost of detail |
| Retrieval | Pulling only relevant documents for a query | Scales memory beyond the window |
| Vector store | A database of embeddings for similarity search | Backbone of retrieval-based agents |
The Decision Tree
This tree assumes a single model with a fixed context window. It does not care which provider you use—only how you spend your token budget.
Step 1: Is your task stateless?
A stateless task never needs to remember anything between calls. A one-shot translation, a single-document extraction, or a format conversion all qualify.
- If yes, go to Leaf A.
- If no, go to Step 2.
Step 2: Do you need the full conversation history?
Some tasks only need the latest user input plus a few fixed rules. Others require the model to see every previous exchange to stay coherent.
- If yes, go to Step 3.
- If no, go to Leaf B.
Step 3: Will the expected history fit inside the model's token limit?
Estimate: system_prompt_tokens + average_message_tokens * expected_turns. If the sum is under the limit, you can keep the history as-is.
- If yes, go to Leaf C.
- If no, go to Step 4.
Step 4: Are the important facts scattered across a large corpus?
If the agent must answer from a knowledge base, logs, or many documents, retrieval beats summarization on both token cost and answer quality.
Leaf A: Stateless Single-Shot
When to use: one request, no memory, deterministic output.
Worked example: parse an unstructured log line into JSON.
import json
import requests
def parse_log_line(line: str, api_key: str) -> dict:
prompt = (
"Extract timestamp, level, message, and service from this log. "
"Return only JSON."
f"\nLog: {line}"
)
resp = requests.post(
"https://api.freemodel.example/v1/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json={"model": "free-model", "messages": [{"role": "user", "content": prompt}]},
)
return json.loads(resp.json()["choices"][0]["message"]["content"])
The prompt is self-contained. There is no history to trim, no hidden state, and no overflow risk. If your task fits here, ignore every other memory technique.
Leaf B: Summarization
When to use: you need gist, not detail; conversation turns are long but few.
Worked example: a meeting note taker that condenses yesterday's transcript before today's call.
def summarize_history(history: list[str]) -> str:
combined = "\n".join(history)
return call_model(
"Summarize this transcript in 100 words or fewer. Keep decisions and action items.",
combined,
)
Store the summary as a single system-level message. On the next call, prepend it to fresh input. You lose exact quoting, but you retain every decision. Summarization is ideal when the conversation is a stream that eventually becomes a report.
Leaf C: Buffered Window
When to use: the full history will stay under the token limit, but you still want to cap growth.
Worked example: a code review bot that needs the last ten comments to understand the current discussion.
from collections import deque
class ContextBuffer:
def __init__(self, max_messages: int):
self.history = deque(maxlen=max_messages)
def add(self, message: dict):
self.history.append(message)
def messages(self) -> list[dict]:
return list(self.history)
A deque(maxlen=...) silently drops the oldest messages. This is the simplest memory policy that still respects a hard cap. The risk is that an important early detail scrolls away—mitigate it with a system prompt that restates the task invariants.
Leaf D: Retrieval (RAG)
When to use: your agent answers from a large, changing corpus; exact recall matters.
Worked example: a support agent that searches a product manual before answering.
from sentence_transformers import SentenceTransformer
import chromadb
client = chromadb.PersistentClient(path="./manual_db")
collection = client.get_or_create_collection("manual")
model = SentenceTransformer("all-MiniLM-L6-v2")
def retrieve(query: str, top_k: int = 3) -> str:
embedding = model.encode(query).tolist()
results = collection.query(query_embeddings=[embedding], n_results=top_k)
return "\n\n".join(doc for doc in results["documents"][0])
The vector store does the heavy lifting. You only send the retrieved chunks plus the query, so the context window stays tiny no matter how large the manual grows. Free-tier models handle this especially well because generation length stays short.
Leaf E: Hybrid (Summary + Window)
When to use: the task runs across many sessions; you need a rolling summary plus the last few raw messages.
Worked example: a data cleanup agent that processes files one by one. It keeps a running summary of what it has cleaned and a buffer of the last three file results.
class HybridMemory:
def __init__(self, buffer_size: int = 3):
self.summary = ""
self.recent = deque(maxlen=buffer_size)
def update(self, new_summary: str, message: dict):
self.summary = new_summary
self.recent.append(message)
def build_prompt(self, current_task: str) -> str:
return f"Summary: {self.summary}\nRecent: {list(self.recent)}\nTask: {current_task}"
The summary preserves global context; the buffer preserves local precision. This pattern is verbose to implement but survives arbitrarily long agent runs. It is what you reach for when step 3 fails and RAG is overkill because the corpus is actually the agent's own output.
Implementation Notes
- Token counting needs a real tokenizer. Free providers may charge the same rate for input and output, so measure both.
- Test with the longest conversation you realistically expect. A unit test that sends 50 turns is worth more than a guess.
- Log token usage per request. When the agent breaks, the ledger points at which strategy failed.
Limitations
This tree assumes a single model and no external memory system. It does not cover multi-model orchestration, fine-tuning, or distributed agent state. The code examples are illustrative pseudocode, not a production library.
Who Should Not Use This
Do not use this approach if your task fits comfortably inside the context window every time—the extra machinery is waste. Do not use it if you need transactional memory or guaranteed replay; a vector store is not a relational database. And if your agent's state must survive a server crash, the tree won't help—you need a real persistence layer.
For zero-cost experimentation, MonkeyCode's free server and free model access give you a place to run these exact patterns without touching your wallet. The decision tree above ensures you only pay for the complexity you actually need.