Catch Cheap-Model Truncation Before It Wastes Your CI Minutes

You don't need a bigger model to catch one of the most expensive failure modes of discounted coding assistants: a patch that stops mid-expression. The symptom looks like a logic bug — a NameError, an indentation error, or a failing test — but the real cause is that the model hit its output ceiling and stopped before closing a bracket, string, or block.

A cheap way to avoid this is to fail fast on syntactic completeness before running the full test suite. The idea is narrow: parse the generated file with the language's own parser, and if the parser says the file ends before the syntax does, treat it as truncation instead of submitting the patch to CI.

Here is a Python smoke test that does exactly that. It uses only the standard library, so you can run it on a free server or in a pre-commit hook.

#!/usr/bin/env python3
"""Fail fast when generated Python is syntactically incomplete."""
import ast
import pathlib
import sys

def main() -> int:
    if len(sys.argv) != 2:
        print('usage: smoke.py <file.py>')
        return 2

    source = pathlib.Path(sys.argv[1]).read_text()
    try:
        ast.parse(source)
    except SyntaxError as exc:
        print(f'SYNTAX_FAIL line={exc.lineno} col={exc.offset}: {exc.msg}')
        lines = source.splitlines()
        lo = max(0, (exc.lineno or 1) - 4)
        hi = min(len(lines), (exc.lineno or 1) + 1)
        for idx in range(lo, hi):
            cursor = '>>>' if idx + 1 == exc.lineno else ''
            print(f'{cursor}{idx + 1:4}{lines[idx]}')
        return 1

    print('SYNTAX_OK')
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

ast.parse() catches the common truncation shapes: an unclosed (), [], or {}, an unterminated triple-quoted string, a try: without its body, or a def that never reaches a complete suite. The script prints only four lines before the failure, which is enough to see whether the break is at the end of the model's output.

That check is useful by itself, but the more valuable part is the regeneration loop. When the smoke test fails, instead of asking a human to finish the file, shrink the prompt and ask the model to regenerate from a smaller context. The loop is deliberately unglamorous:

#!/usr/bin/env bash
set -u

# Replace MODEL_CMD with a wrapper that reads the prompt on stdin and writes
# generated Python to stdout. For CLIs that do not read stdin, define:
#   model() { your-cli --prompt "$(cat)"; }
MODEL_CMD="${MODEL_CMD:-cat}"
PROMPT_FILE='prompt.txt'
OUTPUT_FILE='generated.py'
MAX_ATTEMPTS="${MAX_ATTEMPTS:-5}"

attempt=1
while [ "$attempt" -le "$MAX_ATTEMPTS" ]; do
  "$MODEL_CMD" < "$PROMPT_FILE" > "$OUTPUT_FILE"

  if python3 smoke.py "$OUTPUT_FILE"; then
    echo "attempt ${attempt}: syntactically complete"
    exit 0
  fi

  echo "attempt ${attempt}: incomplete; shrinking prompt"
  python3 - "$PROMPT_FILE" <<'PY'
import pathlib, sys

path = pathlib.Path(sys.argv[1])
text = path.read_text()
marker = '## Task'
if marker in text:
    text = text[text.rfind(marker):]
else:
    lines = text.splitlines()
    text = '\n'.join(lines[-(max(1, len(lines) // 3)):])
path.write_text(text)
PY

  attempt=$((attempt + 1))
done

echo "gave up after ${MAX_ATTEMPTS} attempts; read the last smoke-test output"
exit 1

The shrink step intentionally removes whatever came before the final ## Task marker. Many prompts accumulate file listings, style rules, and prior conversation turns that are not needed to complete one function. If you don't use that marker, the fallback keeps only the final third of the prompt. This does not make the model smarter; it reduces the number of tokens between the model and the unfinished construct, which is often enough for the output to close cleanly.

A sample run looks like this:

SYNTAX_FAIL line=14 col=0: '(' was never closed
      10 def build_query(filters):
      11     clauses = []
      12     for f in filters:
      13         clauses.append(
>>>   14
attempt 1: incomplete; shrinking prompt
SYNTAX_OK
attempt 2: syntactically complete

The loop stops after five attempts by default. Capping the retries matters because a low ceiling may still refuse a large file no matter how much you trim; at that point the correct fix is to split the file or use a model with more output room, not to retry forever.

Where cheap model tiers fit

This workflow is cheap enough to run on a shared or free server because each attempt only parses one file and calls the model once. MonkeyCode's free model access and free server option are relevant here: the loop is disposable by design — regenerate, check syntax, shrink, repeat — so it can run several times before a human reviews anything. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm not assuming a specific quota, model name, or server duration here; the loop is bounded and should respect whatever limit you actually have.

Keep in mind that "free server" is not a license to process private code. If the file is proprietary, make sure the service's data policy and your own security rules allow the request before putting it into any external model.

When the smoke test is enough

Situation	Does this help?	Why
Missing `)`, `]`, `}`, or unclosed triple quote	Yes	`ast.parse()` fails at or near EOF
Incomplete `try`/`if`/`for`/`def` block	Yes	Python requires an indented suite
A name used before definition	No	The file parses; the error is runtime
A wrong algorithm that compiles cleanly	No	You still need tests or assertions
SQL, JSON, YAML, or config files	Not with this script	Use `json.loads`, `yaml.safe_load`, or a parser for that format

The smoke test is intentionally not a linter. It answers one question — "is this file syntactically whole?" — before the more expensive questions about behavior.

Limitations and who should skip this

The biggest limitation is that syntactic completeness is not correctness. A regenerated file can close all of its brackets and still call a nonexistent function, miss a required argument, or silently change behavior after the prompt was shrunk. That is why the smoke test belongs before the test suite, not instead of it.

Second, this example is Python-specific. The same principle works in other languages — node --check for JavaScript, ruby -c for Ruby, json.loads for JSON — but each parser has different edge cases, so don't treat Python's ast as universal.

Third, prompt shrinking is a trade-off. Removing context can make the model complete the syntax while losing constraints from the deleted portion. If the generated file must respect a repo-wide rule, keep that rule in the retained section or verify it with a separate check.

Skip this approach if:

your generated code is private and cannot be sent to an external free tier;
the task genuinely needs the whole repository in context to be safe;
you are generating non-Python files and don't have an equivalent parser in the loop;
your pipeline cannot tolerate the extra regeneration time; or
the output is already far beyond a reasonable output ceiling, in which case splitting the file is the real fix.

The cheapest improvement is not a bigger model or a longer prompt. It's catching the cheap failure before it spends expensive CI minutes. Try the smoke test on one generated file that has failed recently; if the parser points to the end of the file, you've found truncation that a smarter prompt would have hidden.