Why Tokens Per Joule Matters More Than Tokens Per Second
python
dev.to
Most GPU benchmarks report tokens/sec, but that metric ignores the dominant driver of real-world inference cost: energy. I built a cross-platform telemetry suite to measure Tokens Per Joule (T/J) — tokens/sec ÷ watts — alongside throughput. Think of it as miles per gallon for inference. The reference data across Apple Silicon and NVIDIA challenges some common assumptions about hardware selection. TL;DR Metric: Tokens Per Joule (T/J) = tokens/sec ÷ watts. The inference equivalent of