My infra tool could show drift, grade it, and Slack it — but it still couldn't fail a CI build

A while back I wrote about teaching my self-hosted drift detector to stop being
pull — instead of waiting for someone to open a tab, it now pushes a Monday
briefing into Slack. That fixed the human case.

It didn't fix the case where there is no human. A pull request opens, CI runs,
something merges — and the one moment where drift should be able to stop a
change cold is a pipeline that had no way to ask my tool anything. A dashboard
assumes eyes. A Slack digest assumes someone reading Slack. CI reads neither. It
reads exit codes.

So this time the feature is small and unglamorous: a command line.

python manage.py syncvey scan --system e-commerce   # live scan + record a snapshot
python manage.py syncvey drift --env prod           # what drifted, right now
python manage.py syncvey drift --exit-code          # exit 1 if anything drifted
python manage.py syncvey drift --format json        # same, for a pipeline to parse
python manage.py syncvey status                     # systems / envs / counts / last scan

The whole point is that fourth-from-top line. --exit-code turns drift into a
build gate:

- name: Fail on infrastructure drift
  run: python manage.py syncvey drift --exit-code

Now a console change that never went through Terraform can turn a pipeline red.

"But terraform plan already runs in CI"

This is the first objection, and it's fair — plenty of teams run terraform plan
in CI and call the non-empty diff a drift gate. The reason I still needed my own
command is the same reason this whole project exists: plan compares your
state to your config. It never compares either one to reality.

If someone opens a security group in the console, there's no Terraform resource
for it, so plan has nothing to diff. Exit 0. Pipeline green. The most dangerous
kind of drift — the click nobody wrote down — is exactly the kind plan is
structurally blind to.

syncvey drift diffs the live AWS state (scanned with boto3) against the last
snapshot. So --exit-code is the gate plan's exit code can't be. Same form
factor, different question.

Lesson 1: the plugin seam I already had didn't fit — and Django had a better one

I keep the advanced pieces of this tool as detachable apps: risk scoring,
CloudTrail attribution, blast-radius — each lives in its own Django app, and the
core is not allowed to import it. Install the app and the feature appears; remove
it and the core neither knows nor cares.

For web features, that seam is hand-built: a plugin advertises a nav entry and a
URL, and a feature flag guards the route.

class BlastRadiusConfig(AppConfig):
    syncvey_plugin = True
    def nav_items(self, request):
        return [{'label': 'Blast Radius', 'url': reverse('blast_radius:home'), ...}]

My first instinct was to build the CLI the same way — invent some registry the
core reads to decide which subcommands exist. I got about two minutes in before
noticing I was reinventing something Django already does. Every app in
INSTALLED_APPS automatically contributes its management/commands/*.py. That
is the plugin seam. I didn't have to wire anything:

syncvey_cli/
  apps.py                         # syncvey_plugin = True, feature 'cli'
  service.py                      # the actual scan/drift logic
  management/commands/syncvey.py  # the command — discovered for free

Add syncvey_cli to INSTALLED_APPS → manage.py grows a syncvey command.
Remove it → the command is gone, and nothing in the core breaks. The
"detachable" property I'd carefully engineered for web features came for free
for a CLI, because command discovery is already how Django works. The lesson I
keep relearning: before building a seam, check whether the framework already
handed you one.

The AppConfig is almost empty — it just opts into the feature registry so the
rest of the app can answer "is the CLI installed?":

class SyncveyCliConfig(AppConfig):
    name = 'syncvey_cli'
    syncvey_plugin = True
    feature_name = 'cli'

No nav_items, no URL, no route guard. A CLI isn't a web surface, so it
contributes none of that — and the seam doesn't ask it to.

Lesson 2: a CLI that computes drift differently from your UI is a liar

Here was the tempting shortcut. The command needs to know what drifted. The core
already computes that for the dashboard, but its function is _compute_raw_diff
— underscore-prefixed, "private." Re-implementing a quick diff in the CLI would
have been fewer imports and felt cleaner.

It would also have been a bug — the worst kind, where two features quietly
disagree about the same fact.

The core diff has a hard-won rule in it. A live scan emits a curated ~11 fields
per resource; a tfstate import carries 50+. If you diff by the union of keys,
every tfstate-only field reads as "deleted" and the whole thing is a wall of
false drift. So the core compares the intersection of keys and detects real
attribute changes on shared ground:

keys = (set(old.keys()) & set(new.keys())) - _DIFF_EXCLUDE

If the CLI rolled its own diff, syncvey drift and the dashboard would report
different drift for the same environment. Which one do you trust? Neither, once
you've seen them disagree. So the CLI imports the exact same function the UI
uses:

from asset_manager.views import _compute_raw_diff

Yes, I imported a "private" helper across an app boundary. The rule I actually
care about — a plugin may depend on the core; the core never depends on a
plugin — is intact. And there is now exactly one definition of "what counts
as drift" in the codebase. The CLI can't drift from the dashboard because they
run the same code. syncvey scan reuses run_scan and the snapshot recorder for
the same reason: the command mirrors the web scan flow instead of forking it.

Exit codes are the actual product

For a human, the pretty text output is the feature. For CI, the exit code is
the feature and the text is incidental. So the codes are a deliberate contract:

• 0 — ok / no drift
• 1 — drift found (only with drift --exit-code)
• 2 — a scan job failed, or your --system / --env selector matched nothing

That last one matters more than it looks. A drift gate that exits 0 because it
silently matched zero environments is worse than no gate — it's a green light
that checked nothing. 2 keeps a typo'd --system from passing as "clean."

Takeaways

• A monitoring tool that only speaks through a UI and a chat message still can't participate in the one place changes are gated: the pipeline. CI doesn't read dashboards; it reads exit codes. Give it one.
• terraform plan in CI is a state-vs-config gate, not a drift gate. It can't fail on a resource that was never in Terraform. If you care about console-made drift, you need a check that reads live cloud state.
• Before you build a plugin seam, check whether the framework already is one. Django discovers management commands from INSTALLED_APPS — a CLI plugin is detachable for free, no registry required.
• If you add a second interface (CLI, API, export) over an existing feature, reuse the exact function, don't re-implement it. Two code paths that compute "the same" answer will eventually disagree, and then both are untrustworthy.
• Make "matched nothing" a non-zero exit. A gate that passes because it checked zero things is the most dangerous kind of green.

This CLI ships with a self-hosted tool that tracks how your live AWS drifts from
Terraform — open source (MIT), one docker compose up:
syncvey.com. If you gate infra in CI today, what are you
gating on — terraform plan, a cloud-state scan, or nothing yet?