devLap

We picked NDJSON over SQLite, and the reason was the build pipeline

typescript dev.to

Our desktop app needs to remember which rental listings it has already replied to, across restarts. That is a database-shaped problem and the default answer in Electron is SQLite via better-sqlite3.

We used a text file with one JSON object per line, and the deciding argument was not performance.

 * NDJSON rather than sqlite on purpose: no native module, so the existing
 * electron-builder pipeline (already fiddly with the bundled Playwright
 * browsers) stays untouched. Updates are appended as a new row with the same
 * id and readers fold last-wins, which keeps writes atomic by construction.
Enter fullscreen mode Exit fullscreen mode

The native module tax is real

A native module in Electron means rebuilding against Electron's own Node ABI, for every platform and architecture you ship, forever. Our matrix is macOS arm64, macOS x64 and Windows x64. Add electron-rebuild to the build, an ABI bump on every Electron major, and a class of bug that only reproduces in a packaged app on an architecture you do not own.

Our packaging was already carrying a bundled Chromium for Playwright, with per-architecture paths and asar unpacking rules. Adding a second thing that can break differently per platform, for a workload measured in thousands of rows, was not a good trade.

If you genuinely need indexes, joins or concurrent writers, pay the tax. We needed a set and a counter.

Append-only makes writes atomic for free

The core move is that nothing is ever updated in place. A status change appends a new row carrying the same id:

function appendRow(file: string, row: unknown): void {
  ensureDir();
  try {
    fs.appendFileSync(file, `${JSON.stringify(row)}\n`, 'utf8');
  } catch (err) {
    console.error(`[ledger] Failed to append to ${path.basename(file)}:`, err);
  }
}
Enter fullscreen mode Exit fullscreen mode

Readers fold:

/** Fold append-only rows into current state, last write wins. */
function fold<T extends { id: string }>(rows: T[]): Map<string, T> {
  const byId = new Map<string, T>();
  for (const row of rows) {
    if (!row || typeof row.id !== 'string') continue;
    byId.set(row.id, { ...byId.get(row.id), ...row });
  }
  return byId;
}
Enter fullscreen mode Exit fullscreen mode

Two things fall out of this. A single small appendFileSync never leaves a previously-written record damaged, because it does not touch one — the worst case is a torn last line. And the merge is a spread, so a writer that only knows about { id, status } can update a record without having read the whole thing first.

The torn last line is handled where it appears:

for (const line of raw.split('\n')) {
  const trimmed = line.trim();
  if (!trimmed) continue;
  try {
    rows.push(JSON.parse(trimmed) as T);
  } catch {
    // A torn final line (power loss mid-append) must not poison the read.
  }
}
Enter fullscreen mode Exit fullscreen mode

Per-line parsing is the whole reason NDJSON beats "a JSON array in a file". One corrupt line costs you one record. One corrupt byte in a JSON array costs you the entire file — and for us, losing the ledger means the app forgets what it has already replied to and messages a set of landlords a second time.

Compaction is just a rewrite

Append-only grows forever, so:

const REPLIES_MAX_ROWS = 8000;
const REPLIES_KEEP = 2000;
Enter fullscreen mode Exit fullscreen mode

Past 8000 rows, fold, keep the most recent 2000, write to a temp file, rename over the original. Same temp-plus-rename trick as anywhere else — rename is atomic within a filesystem, so a crash mid-compaction leaves the old complete file, not a half-written new one.

The gap between the threshold and the keep count matters. Compact at 8000 down to 7999 and you re-compact almost every write. A 4x gap means compaction is rare enough to be invisible.

Deciding what not to store

The header is blunt about scope:

 * This is not a history feature. It exists so the reply engine can answer two
 * questions across restarts: has this listing already been messaged, and how
 * many replies have gone out recently. Found listings are not recorded at all —
 * the alert email is the record of those.
Enter fullscreen mode Exit fullscreen mode

Listings we merely found are not written down. The email we sent is already a durable, user-visible, searchable record of that event, sitting in their inbox. Duplicating it into local storage would add a growing file, a privacy surface, and a second thing that can disagree with the first.

"What is the minimum this component must remember to do its job correctly after a restart?" is a better starting question than "what might we want to query later". The first gives you a 200-line file. The second gives you a schema.

Where it runs

The ledger ships inside the desktop app — notifio.app/download for macOS and Windows builds. If you install it and enable auto-reply, the NDJSON file appears in the app's data directory and you can tail -f it while a reply runs. Every status transition is one line, which is most of why the format is pleasant to debug.

Source: dev.to

arrow_back Back to Tutorials