zhao-dbt-plan: a microbatch cascading time-window planner for dbt

rust dev.to

A few days ago I posted about zhao-cli, a breaking-change
gate for dbt (link at the bottom if you missed it). This is the second tool in the same family, built to fix a problem I kept hitting myself. I think this one is honestly the
bigger deal.

The problem

dbt's microbatch incremental strategy applies one flat --event-time-start/--event-time-end window across an entire --select. That's fine until a rolling-window model reads a wider span than its immediate upstream was just recomputed for, at which point dbt has no way to know it needs a wider batch too. A backfill to a daily model silently corrupts every downstream rolling aggregate that reads across the backfilled day, and dbt only ever re-triggers the model you actually selected.

zhao-dbt-plan: the microbatch cascading window planner

zhao-dbt-plan reads your compiled manifest, walks the DAG within whatever you --select, and computes the correct, minimal per-model expanded window -- as a plan you review, never a command it runs for you. It never executes dbt build or dbt run on your behalf; that boundary is permanent, not a v1 scope cut. You decide how to actually run the plan -- raw dbt, Dagster, Airflow, a Databricks Asset Bundle, whatever you already use.

$zhao-dbt-plan --select tag:microbatch_demo --event-time-start 2026-07-01 --event-time-end 2026-07-01 --pretty

[layer 0] mb_daily [2026-07-01 .. 2026-07-01]
  [layer 1] mb_rolling_7d [2026-06-28 .. 2026-07-05]
    [layer 2] mb_rolling_14d [2026-06-26 .. 2026-07-06]
      [layer 3] mb_summary [2026-06-25 .. 2026-07-07]
      [layer 3] mb_wide [2026-04-07 .. 2026-07-11]
warning: mb_wide: expanded window (96 days) exceeds max_window_expansion_days (90)
Enter fullscreen mode Exit fullscreen mode

One day's backfill at the top cascades into a correctly-sized, per-model window all the way down the DAG, and it flags the one model whose expansion is wide enough to be worth a second look, instead of silently recomputing (or silently under-computing) it. --html renders the same plan as an interactive report:

Live demo · Repo

How this compares

I'd rather be precise here than let anyone assume I haven't looked: this specific problem, correctly widening a downstream rolling model's window when only part of its upstream dependency was recomputed -- isn't solved by dbt's own native tooling. dbt's lookback config is single-model and self-contained; it doesn't propagate anything to downstream models at all.

SQLMesh does handle the underlying problem, natively. lookback there triggers automatic cascading restatement through the whole downstream chain, because SQLMesh owns persistent per-interval state across the DAG. That's a genuinely different, arguably more elegant mechanism
than what zhao-dbt-plan does. It's a reason to consider SQLMesh on its own merits, not a knock against it. But it only applies if you're on SQLMesh's engine; it's not something you get by adding a tool on top of dbt. I haven't found anything, official or community -- that fills this gap for dbt itself, so I can't claim it's the only thing that's ever tried; I can say it's solving a real, confirmed pain point in dbt's own microbatch design that I hit myself and couldn't find an existing answer for.

Making the plan actually apply: zhao_utils

There's one gap worth being upfront about: zhao-dbt-plan computes the correct window, but on its own it can't make dbt's compiled SQL actually use it, dbt's own microbatch ref() filtering can't be overridden by a project macro (confirmed against dbt-core's real behavior, not
assumed -- it's resolved outside normal macro dispatch, to build the dependency graph statically before Jinja even renders). So a plain ref() to a widened upstream still silently gets dbt's narrow, single-batch default, no error, no warning, just a rolling-window model quietly computing on too little data.

![Setup: a one-time wrapper macro in your own project, safe to blanket-replace ref with wref across your whole project since it's a no-op without meta.zhao. Effect: plain ref() reads a 1-day window and silently under-computes; wref() reads the correct 8-day window, matching the plan zhao-dbt-plan already computed]

I built a small, separately-licensed (Apache-2.0), separately-repo'd package for this:
zhao_utils -- wref() ("windowed ref"), a
drop-in ref() replacement, plus two boundary helpers, reading the exact same meta.zhao block
the planner already does. Completely optional. If you've already hand-rolled this pattern yourself (the documented dbt way to do a rolling-window read), you don't need it; zhao-dbt-plan itself works identically either way. It's for whoever's starting that pattern fresh.

Two install paths, both through GitHub, neither requiring a manually-tracked local copy:

  • As a real dbt package (packages.yml + dbt deps, versioned, upgradeable), macros called namespaced: {{ zhao_utils.wref('mb_daily') }}.
  • As a single file you copy straight into your own project's macros/ folder, no package management at all, and calls are bare by default: {{ wref('mb_daily') }}.

Both are genuinely tested, not just the primary path with a workaround bolted on: real dbt build runs against DuckDB for both install modes, plus compile-verified against dbt Fusion and a real Databricks workspace. Full details, including the exact optional/validated-argument behavior, are in the package's own README.

Built and tested honestly

Built and tested solo, as honestly as I could. I haven't been able to cover every setup alone, so if something doesn't fit yours, an issue or a PR is genuinely welcome.

Install

curl -fsSL https://raw.githubusercontent.com/allenhori/zhao-dbt-plan/master/scripts/install.sh | sh
# or: cargo install zhao-dbt-plan
Enter fullscreen mode Exit fullscreen mode

zhao-dbt-plan runs as a zhao Addon too (zhao dbt-plan ...), if you'd rather invoke it through zhao-cli directly, see the Addon contract
if you want to build your own.

Licensing -- stated plainly, not buried

zhao-cli (the first tool in this family) is Apache-2.0 -- fully permissive. zhao-dbt-plan is
AGPLv3 instead -- copyleft. If you modify it or run it as part of a network service, that has
to stay open too. Not because I'm precious about it -- I just don't want someone quietly wrapping
it into a paid product without contributing back. Both are free to use today, either way; the
license difference is about what happens if you modify and redistribute, not about whether you
can use the tool.


Repos: zhao-dbt-plan · zhao_dbt_utils · zhao-cli

If you try this and it breaks on your setup, please open an issue, that's exactly the kind of real-world coverage I can't get building this alone.

Source: dev.to

arrow_back Back to Tutorials