Can you build observability ingestion on S3 alone — no Kafka, no disks, no coordination layer?
rust
dev.to
TL;DR — A Kafka + Flink + OTel ingestion pipeline cost us ~$700–800/month at 10 MB/s. We rebuilt it as a single binary where the data, the write-ahead log, and the Iceberg catalog all live in S3 alone — no Kafka, no local disks, no coordination service — for ~$100/month. Here's the design. Self-hosted observability sooner or later runs into the problem of storing state. Query load, CPU, and data volume can all be handled by scaling out, but the stateful layer is something you have to operat