Over at I-Dacs Labs, we run high-throughput telemetry pipelines on edge devices (Raspberry Pis, Advantech gateways, and embedded x86/ARM boxes). We've been using LF Edge eKuiper for local stream processing (SQL filtering, sliding windows, and MQTT/Kafka sinks), but kept hitting the classic edge computing wall:
- JVM engines (Apache Flink): Incredible throughput, but require >1 GB RAM and take 20+ seconds to boot. Unusable on small industrial hardware.
- Go engines (Upstream eKuiper, Benthos, Telegraf): Much lighter, but continuous Stop-The-World GC sweeps introduced tail latency jitter. Worse, under burst sensor loads (10k–100k events/sec), Go channel buffer saturation led to silent packet loss. We decided to rewrite the entire engine in pure Rust: rekuiper (v0.421-beta, dual licensed under MIT / Apache-2.0). --- ### What We Built
-
Core Architecture: Lock-free stream bus (
StreamBus), Tokio async actors for rule execution, and bounded actor queues for sinks with zero runtime GC pauses. - 100% Drop-In Parity: Fully compatible with the existing eKuiper Manager Web UI, OpenAPI 3.0 schemas, and standard streaming SQL. Zero scaffolded stubs across all 98 REST endpoints.
- Footprint: 9.60 MB stripped static binary, ~6 – 8.2 MB idle RAM consumption.
- Sub-15ms Cold Boot: 12.5 – 14.5 ms internal daemon bootstrap; 123 ms end-to-end process-spawn-to-ready.
Empirical Head-to-Head Benchmarks (500,000 Records)
Rather than hand-waving estimates, we ran all engines head-to-head on the exact same Linux machine (WSL2 / Ubuntu x86_64) using an identical 500,000-record telemetry workload:
Pipeline: Parse 500k JSON events → Compute formula (temp * 1.8 + 32) → Filter (temp > 20.0) → Project (id, temp_f) → Sink
| Engine | Runtime | 500k Elapsed | Throughput | Data Drops | Memory |
|---|---|---|---|---|---|
| rekuiper (0.421) | Pure Rust | 1.176 s | 425,308 eps | 0 (0.0%) | ~8 MB |
| Apache Flink | Java / JVM | 2.144 s | 233,209 eps | 0 (0.0%) | ~1,022 MB |
| Telegraf | Go | 8.194 s | 61,019 eps | 0 (0.0%) | ~50 MB |
| Upstream Go eKuiper | Go | 11.290 s | 44,287 eps | 72,921 (14.6%) | ~45 MB |
| Redpanda Connect | Go | 19.236 s | 25,993 eps | 0 (0.0%) | ~38 MB |
Key Observations:
-
Channel Saturation in Go: Under sustained 500k burst ingestion, upstream Go eKuiper dropped 72,921 records (14.6% data loss) due to channel saturation (
buffer full, drop message).rekuiperprocessed all 500,000 events with 0 drops in 1.176s (9.6x faster). -
vs Apache Flink: Flink’s execution graph is fast (233k eps), but the JobManager + TaskManager JVM consumed over 1 GB of RAM.
rekuiperbeats it in single-core throughput while consuming 125x less RAM (< 8.2 MB).
3. Cold Boot Time: rekuiper boots internally in ~13 ms (123 ms OS spawn to socket ready), compared to 1.2s for Go eKuiper and 20s for Apache Flink.
Reproduce It in 2 Minutes
All reproduction scripts and Docker configs are in the repository. Anyone can run the whole suite:
git clone https://github.com/ankur-paan/rekuiper.git
cd rekuiper
./test/benchmark/run_all.sh