How I Audit Error Messages With Free AI Compute

Error messages are the most frequently rendered user interface in any application, yet they are almost never covered by tests. I audit them with a four-stage workflow on free AI compute—extract, score, rank, and regress—so unspecific, unactionable, or blaming copy is rewritten before it becomes a support ticket.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why I Treat Error Messages as the Last Untested UI

Every other UI I ship gets a design review, a usability pass, and a regression suite. Error messages usually get a string literal and a hope that nobody reads them. When something fails, that string is often the only UI a user sees, and it is the first input I read in a debugging session.

I treat a bad message as having three costs that never show up in coverage:

• Support load. Users who cannot understand the message ask a human.
• Debug time. Developers who cannot parse the message re-run the code with print statements.
• Churn. Users who feel blamed start looking for an alternative.

Those costs match established UX writing. Nielsen Norman Group's error-message guidelines still ask for a precise description of the problem, a constructive next step, and a tone that does not blame the user. Google's guidance on error messages adds a developer-facing bar: name the failure, include relevant values, and avoid vague verbs like "failed." I use those two sources as the human rubric, then let a model apply them at codebase scale.

The Four-Stage Audit I Run on Free Compute

I designed each stage to run on free infrastructure so I will actually run it. MonkeyCode's free model access and free server make that practical. I only send the error strings, not surrounding source, which keeps exposure small.

1. Extract every error message with a read-only static analysis script.
2. Audit the strings in batches against a structured prompt.
3. Rank the evaluations in a JSON report by severity.
4. Regress the worst rewrites with a keyword assertion so the old copy cannot silently return.

Extraction is the only stage that touches the repo, and it is read-only. The report is JSON aggregation. The regression is a unit test, not a new product feature.

Extract messages with a conservative AST walk

I walk a Python tree and collect string literals on raise calls. I stay conservative: a false positive is worse than a missed message. I skip exceptions built from variables because those strings cannot be audited without runtime context.

# extract_errors.py — collect error messages from a Python codebase
import ast
import json
import os
import sys

def extract(root_dir: str) -> list[dict]:
    """Extract error messages from raise statements in Python files."""
    messages = []
    for dirpath, _, filenames in os.walk(root_dir):
        for filename in filenames:
            if not filename.endswith(".py"):
                continue
            filepath = os.path.join(dirpath, filename)
            try:
                with open(filepath) as f:
                    tree = ast.parse(f.read())
            except (SyntaxError, UnicodeDecodeError):
                continue
            for node in ast.walk(tree):
                if not isinstance(node, ast.Raise):
                    continue
                exc = node.exc
                if not isinstance(exc, ast.Call) or not exc.args:
                    continue
                arg = exc.args[0]
                if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
                    messages.append({
                        "file": os.path.relpath(filepath, root_dir),
                        "line": node.lineno,
                        "message": arg.value,
                    })
    return messages

if __name__ == "__main__":
    root = sys.argv[1] if len(sys.argv) > 1 else "."
    print(json.dumps(extract(root), indent=2))

What this catches versus what it skips:

• Catches: raise ValueError("message"), raise TypeError("message"), and other raise SomeError("literal") forms where the first argument is a string constant.
• Skips: f-strings, concatenations, messages stored in variables, bare raise SomeError, and non-Python files.

The output is a JSON array that feeds the audit stage. Python's errors and exceptions tutorial is a useful reminder that users see the exception type plus this string—so the string is the UI.

Batch-audit against four quality dimensions

I send each message to a free AI endpoint and ask for 1–5 scores plus a rewrite when any score is below 4. The script fails open: if the endpoint is down or quota is exhausted, I record the error in the review field instead of aborting.

# audit_errors.py — send error messages to a free AI endpoint for evaluation
import json
import os
import sys
import urllib.request

def audit(messages: list[dict], endpoint: str, api_key: str) -> list[dict]:
    """Evaluate error messages against four quality dimensions."""
    results = []
    for item in messages:
        prompt = f"""Evaluate this error message from a developer experience perspective.

Message: "{item['message']}"
Location: {item['file']}:{item['line']}

Score it 1-5 on each dimension:
- specificity: does it name the actual failure?
- actionability: can the user fix it without guessing?
- tone: does it blame or shame the user?
- context: does it include relevant values?

If any score is below 4, provide a concrete rewrite.
Respond as JSON: {{"scores": {{...}}, "rewrite": "..."}}"""

        body = json.dumps({
            "model": "model-name-placeholder",  # verify current model in repo
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 400,
            "response_format": {"type": "json_object"},
        }).encode()

        req = urllib.request.Request(
            endpoint,
            data=body,
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
            },
        )
        try:
            with urllib.request.urlopen(req, timeout=60) as resp:
                data = json.loads(resp.read())
            review = json.loads(data["choices"][0]["message"]["content"])
            results.append({**item, "review": review})
        except Exception as exc:
            results.append({**item, "review": {"error": str(exc)}})
    return results

if __name__ == "__main__":
    messages = json.loads(sys.stdin.read())
    endpoint = os.environ["MONKEYCODE_ENDPOINT"]
    api_key = os.environ["MONKEYCODE_API_KEY"]
    print(json.dumps(audit(messages, endpoint, api_key), indent=2))

I score four dimensions because each maps to an outcome I can feel:

Scoring rules I actually use:

• Below 4 on any dimension → rewrite candidate
• Below 3 on tone → priority, because tone drives churn
• Below 3 on specificity → a support ticket waiting to happen

A rewrite I would accept: "Invalid input" becomes "email must contain '@'; got 'user.example.com'." Specificity, actionability, and context all move; tone stays neutral.

Which Messages I Audit First

Not every string deserves the same attention. I rank by how visible the message is to a human who is not already in a debugger.

The rule is simple: a message in a user-facing form is worth ten messages in a server log. The form generates tickets; the log generates a stack trace a developer can interpret anyway.

For high-priority rewrites I add a cheap regression: I assert the new message still contains the field name, the constraint, and the observed value. That is not a full UX test, but it stops the next refactor from reverting to "Invalid input".

Limitations, and Who Should Skip This

This approach has three honest limits:

1. The audit is only as good as the model's grasp of my domain. A generic model will miss context a human reviewer would catch.
2. The extractor only handles Python raise statements. A polyglot repo needs a different extractor per language.
3. The audit sends error strings to a remote endpoint. That is a non-issue for most projects and a hard blocker in regulated environments.

I skip this workflow when error copy is already owned by a dedicated developer-experience team, when compliance forbids sending any code artifacts to external services, or when the codebase is small enough that I can review every message in an afternoon. For everyone else, a free AI audit is a cheap way to find the messages that are costing support hours.

Error messages are the last untested UI in the product, and free AI infrastructure is the right tool for the job. Extract the messages, audit them against four quality dimensions, and turn the worst offenders into regression tests. Token cost is negligible, server cost is zero, and the tickets you avoid are the real return.

If you want to run this workflow today, start with user-facing validation strings—those are the ones that turn into tickets first. Copy the two scripts above, set MONKEYCODE_ENDPOINT and MONKEYCODE_API_KEY, and point them at MonkeyCode's free server and 10-million-token allowance; the open-source repository documents the current limits. Run extract, pipe into audit, rewrite anything that scores below 4, and add a keyword assertion so "Invalid input" cannot land again.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.