Last week I published apitap - an open-source engine that moves whole tables between databases. One function, no config files, no pipeline DAGs:
import apitap
report = apitap.transfer(
"postgres://user:pass@src-host/db",
"clickhouse://default:pass@ch-host:8123/db",
table="public.events",
)
print(f"{report.rows:,} rows in {report.elapsed_ms} ms")
On a 16-core box, that call moves 10 million rows from Postgres to ClickHouse in 9.9 seconds - measured from the exact wheel you get with pip install apitap, not from some tuned lab build. The same table takes ingestr 111 seconds and dlt 1,893 seconds.
This post is about how, and - more usefully - about what I learned when I put those numbers next to the vendors' own published benchmarks. Spoiler: everyone's numbers are "true." That's exactly the problem.
Why I built it
I've run the whole ELT tool ladder in production as a data engineer. Airbyte first. It works - and it's expensive in the way that doesn't show up in the pricing page: a beefy VM or a Kubernetes cluster just to host it, sync workers that want gigabytes of RAM each, and full refreshes slow enough that the infrastructure to move the data cost more than the infrastructure to store it.
Then dlt and ingestr - genuinely lighter, genuinely faster than Airbyte, and I respect both projects. But at the 10-million-row tables I deal with daily, they still burn serious CPU-hours and memory (the numbers below make that concrete). The pipeline machine is a 24/7 line item, and these tools size it.
What I actually needed had three requirements that no tool met at once:
Fast - wire speed, not ORM speed;
Big - tens of millions of rows per table, routinely;
Small - running on the cheapest box on the pricing page.
Because if you can move a 10M-row table on a 0.5 vCPU / 256 MB container in under a minute - instead of an 8 GB worker grinding for ten - the savings aren't a benchmark bragging right. They're your monthly cloud bill. That's the entire thesis of apitap: the engine should be so cheap to run that the databases become your only cost.
What apitap is
apitap follows the Polars model: a Rust core with thin Python bindings, MIT-licensed, pip install and go. It currently speaks four routes:
Postgres → Postgres - raw binary COPY passthrough. No row decode at all; bytes relay from socket to socket, like psql | psql without the shell.
Postgres → ClickHouse - the binary COPY stream is transcoded in flight to ClickHouse RowBinary: byte swaps, epoch rebasing, exact NUMERIC → Decimal scaling. Neither database ever touches text.
MySQL → ClickHouse / Postgres - MySQL has no COPY, so apitap decodes the binary wire protocol directly (no ORM, no driver row objects) and re-encodes the destination's binary format.
Every transfer stages into a shadow table and swaps in atomically - readers never see a partial load, an empty source never wipes a good table, and a mid-run crash leaves the previous data untouched. Memory is bounded by design: a 256 MB container moves tables of any size, because bytes stream with TCP backpressure instead of materializing.
The benchmark
I didn't invent a benchmark that flatters my tool. I took ingestr's own benchmark schema - their 15-column table with strings, decimals, timestamps, JSON - seeded 10 million rows, and ran every tool under identical conditions:
same box, same stock Docker databases (no server tuning, at all)
every tool capped at 16 vCPU / 4 GB, auto settings for everyone
every result checksum-validated across engines before the time counts - counts, sums, exact decimal sums, ordered MD5 samples
re-run across days: dlt reproduced within 0.1–2%, so nothing here is a lucky run
The results, with apitap installed from PyPI like any user would (the wheel's sha256 matches the release build byte for byte):
¹ dlt's pyarrow backend infers MySQL DOUBLE as decimal128(38,9) and refuses the conversion without hand-written schema hints.
Everything is reproducible from benchmarks/README.md - including my own operational mistakes, which are documented there too, because a benchmark that hides its mistakes isn't worth trusting.
Lesson 1: rows are not a unit
Here's where it gets interesting. dltHub's own benchmark blog reports ~360 million rows per hour with the pyarrow backend. My measurement says dlt+pyarrow does ~100M rows/hour. Is someone lying?
No - and the reconciliation is the most useful part of this whole exercise.
Their benchmark runs TPC-H: 43.3M rows totaling 8 GB, so ~185 bytes per row. My benchmark table measures 462 bytes per row (pg_table_size ÷ rows). Same engine, same effort per byte - but "rows per hour" makes one number look 2.5× better before anything real happened.
Normalize to bytes and the mystery evaporates:
dltHub's tuned headline: 68 GB/hour
dlt+pyarrow measured on my single wide table: 44 GB/hour
Same class. Their number is real. It's just not comparable to a single-table copy.
apitap, same table, untuned: 4.41 GB in 9.9 s. And because a 10-second burst isn't a rate, I re-ran it at 10× the size: 100M rows / 46 GB in 139 s - a measured, checksum-validated 1.2 TB/hour, sustained past the page cache.
When you read any ELT benchmark, find the bytes. If the post only gives you rows, it's telling you about the schema, not the engine.
Lesson 2: "parallel" across tables is not "parallel" inside one
The remaining gap between their 68 and my measured 44 GB/hour has a structural cause. TPC-H is eight tables, and dlt's workers settings parallelize across tables - their extract ran eight concurrent streams.
But the job most of us actually have is "move THIS one big table." So I re-ran dlt+pyarrow with dltHub's exact tuned recipe from their blog - workers 8/8/8, chunk_size=150000, typed file formats:
pg→ch: 360 s untuned → 370 s tuned
pg→pg: 694 s untuned → 701 s tuned
Nothing. On a single table, dlt's extract is one sequential reader no matter what you set, because its parallelism unit is the table. apitap splits inside the table - contiguous primary-key ranges (or physical TID ranges when there's no PK at all) feeding a work-stealing queue of parallel pipes. That one design difference is most of the 35×.
dlt parallelizes across tables. apitap parallelizes inside them.
Lesson 3: bounded memory is a design property, not a knob
dlt's fastest documented backend is connectorx - a Rust/Arrow extractor. In my runs it was OOM-killed on all four routes, because it materializes the full result set in memory, and 10M × 462 bytes doesn't fit in a 4 GB cap.
The same 4 GB cap apitap ran every benchmark in. In fact, apitap completes every route in a 0.5 vCPU / 256 MB container - the cheapest box you can rent - because its memory use is pipes × chunk_size, never a function of table size. The pipe count even auto-derives from the cgroup's CPU and memory limits, so it sizes itself down instead of dying.
You can't retrofit that with a setting. It has to be how the engine moves bytes.
What the engine doesn't spend
The deeper story of the numbers is what isn't there. At 16 cores, every apitap route is bottlenecked by the databases themselves - we sampled Postgres wait events during loads to prove it (the destination backend spends 53% CPU parsing tuples, 23% on WAL; the source spends 62% waiting on client backpressure). The engine's own overhead is approximately zero: no per-row Python objects, no text serialization round-trip, no double-write (dlt's full refresh writes every row twice - temp table, then rewrite; apitap COPYs once and swaps names).
Getting there took a zero-copy audit (buffer recycling, stack-allocated encoders, transcoding straight from input slices), PGO-built release wheels, and one 512 KiB socket-buffer patch - each change A/B-tested at 10M rows, kept only if it survived an interleaved same-minute comparison. The losers (jemalloc! io_uring! client-side compression on localhost!) are documented in the repo alongside the winners.
Honest limitations
Full-table replace today - incremental sync (cursor-based append & merge) is next on the roadmap.
Four routes today; Snowflake/BigQuery and Arrow export are coming (the connector architecture makes a new destination one file).
Linux x86–64 wheels today; aarch64/macOS coming.
Come build it with me
If "fast, handles big data, runs on tiny infra" is a problem you care about, apitap is open for contributors - and the architecture was refactored specifically to make joining easy: one generic driver runs every route's lifecycle, and a new database is one connector file implementing a Source and/or Sink trait. Snowflake, BigQuery, DuckDB, MySQL-as-destination, incremental sync, aarch64/macOS wheels - the roadmap is open, and so are issues and PRs. The bar for changes is the same one the engine was built with: one lever at a time, measured at 10 million rows, checksum-validated before it counts.
Full disclosure, because this audience deserves it: the open-source version was built in pair-programming sessions with Claude Code - the Rust engine, the zero-copy audit, the benchmark harness, and the adversarial review of the refactor. Every optimization you read about above was A/B-tested on real hardware before it was kept (and the AI's rejected ideas - jemalloc, io_uring - are documented in the repo right next to the wins). The methodology is the point: it doesn't matter who typed the code; it matters that every claim survives a checksum.
And one more honest note: AI helped me write this article too. I'm a data engineer, not a writer. But I had something real to share, and I'd rather share it with help than not share it at all. The numbers, the years of paying for slow pipelines, and the stubborn idea that data movement shouldn't need a big box - those are all mine.
Try it
pip install apitap
Code: github.com/apitap/apitap-lib
Full usage guide: docs/usage.md
Reproduce every number in this post: benchmarks/README.md
If your table is big, your box is small, and your patience is finite - that's the exact corner of the design space apitap was built for.