devLap

Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984

python dev.to

If you are building streaming text-to-speech, you have to pick a decoder — the thing that turns a representation into a waveform. I tried to pick one from the literature and could not, so I measured twenty of them under one frozen protocol on identical audio.

Three things came out of it. Two are about the decoders. The most useful one is about my own measuring stick, which failed a test I had written in advance specifically to catch it.

The problem with the published record

Real-time factors quoted without naming the hardware. "Streaming" claimed for architectures that need the whole utterance before they emit anything. Quality figures from different corpora, at different sample rates, through different measurement paths. Each number is probably fine on its own. None of them compose into a decision.

So: one GPU, one corpus, one protocol, twenty arms. Representation → waveform only. Generator and autoregressive timing are a different question, and a decode-only number reported as end-to-end TTS performance is a failed report.

One framing decision changed the candidate set before any measurement happened. I nearly excluded several codecs for not being causal — then realised only the decoder needs to be causal for a TTS build. The encoder runs at training time on complete utterances; at inference the generator emits tokens directly and no encoder sits in the serving path. NVIDIA's NanoCodec makes it concrete: non-causal encoder, causal HiFi-GAN decoder. "Is this codec causal?" is the wrong question.

Five gates, each frozen before it ran

gate question
0 — candidate which decoders are runnable, licensed and comparable at all?
1 — functionality does it produce structurally valid audio?
2 — streaming does it truly stream, or is it stateless chunking?
3 — performance latency, real-time factor, first-audio, jitter, memory
4 — quality what does the round trip cost the signal?
Q5 — identity does the speaker survive, and do they stay one person?

The protocol for each gate was written and hashed before the data it governs existed. Two later freezes were not prospective — Gate 4's analysis layer and three revisions of the Q5 spec were post-measurement repairs, and the records say so rather than pretending otherwise. That distinction matters more than it sounds like it does, and I got it wrong in the first published version of this work.

Gate 2: "streaming" is a property almost nothing has

An arm counts as truly incremental only if both hold:

  1. stateful chunked decode reproduces full-context output to max|err| ≤ 1e-3, and
  2. state is load-bearing — stateless error is at least 10× the stateful error.

The second condition exists because the first can be satisfied by an arm that ignores state completely. If you only check condition 1, a stateless chunker passes and you ship it believing it streams.

Of twenty arms: three are truly incremental, all three explicitly causal FocalCodec configurations. Fifteen are stateless chunking. One streams its network but never emits audio from partial input. One was blocked on platform and reports no number at all rather than a fabricated one.

The three that pass do so with state ratios of 3,530 to 36,740 against a threshold of 10. State is not marginally load-bearing there; it is decisive.

The negative controls are what make that mean anything. The non-causal FocalCodec configurations run through the identical stateful code path. Their ratio is exactly 1 — carrying state changes nothing. If a negative control had passed, the experiment would have been void, not the arm promoted.

Several of the fifteen never claimed to stream. The classification measures what they do; it is not an accusation about what they advertised.

Gate 3: speed is not validity

First-audio latency (TTFA) across the arms that have one spans 1.37 ms to 27.64 ms at the ~80 ms chunk anchor. MelFlow does not have one: it emits spectrogram frames, and producing playable PCM from a partial stream needs an overlap-add stage this study records as not established. Its 375.91 ms is a steady-state median, a different quantity, reported separately and never mixed into the TTFA range. (I mixed them in v1.0.0. More on that below.)

Real-time margin spans 64× down to 0.2×.

But two arms produce streamed output that fails the study's own validity gate at every chunk size tested. A third fails at the anchor and passes at larger sizes — its smallest viable configuration is 640.8 ms, not 80 ms.

The sharpest case: vocos_mel24 has the fastest first-audio in the whole study and no achievable streaming configuration at all. Its timing numbers are real. What they timed is not a valid streaming configuration. If your benchmark reports latency without a validity gate, this is the arm that wins it.

One more result governs how any of these numbers may be read. A controlled probe — byte-identical tokens, same GPU, one arm run in every environment — found a 50.2% difference attributable purely to the transformers version, device-side, at identical torch. So one arm in this study may not be compared numerically against the others at all. That is not a statement about whether it is fast. It is a statement that the comparison does not exist.

Gate 4: the metric failed the test I wrote to catch it

This is the part worth your time even if you never touch audio.

Gate 4 ranks on one metric, mel-cepstral distance, chosen after nine variants were implemented and attacked. Alongside it, the pre-registration installed a validity instrument: put Griffin-Lim in the panel. Griffin-Lim is a phase-retrieval algorithm from 1984 with zero trained parameters. The pre-registration stated in advance what it would mean if Griffin-Lim scored close to the trained neural codecs — a finding about the metric, not about the decoders.

It did not score close. It scored best, in six states of six.

The obvious explanation was tested and refuted. The hypothesis that the metric's low-level frame floor favours a magnitude-matching algorithm failed: splitting each cell at its own median energy, Griffin-Lim leads in both halves, and by more in the loud half.

What remains is an explanation consistent with the result rather than a demonstrated mechanism, and the difference is worth being careful about. Mel-cepstral distance here is derived from the magnitude spectrum and is blind to phase by construction. Griffin-Lim iterates toward consistency of the linear STFT magnitude and optimises nothing else, accepting whatever phase error that leaves. Aligned in domain — but not the same quantity, and nothing in this package demonstrates that Griffin-Lim minimises the released cepstral distance exactly.

What is established does not depend on the mechanism: a metric whose ranking puts an untrained phase-retrieval algorithm first is not, on its own, a sufficient authority for cross-decoder quality.

That is the whole lesson, and it generalises well past audio. The pre-registration naming this outcome in advance is the only reason it reads as a finding rather than an embarrassment — and the reason no metric was swapped in afterwards to produce a tidier table. If you are choosing anything on a published quality number, the question worth asking first is whether that metric has ever been shown a floor it should beat.

Q5: does the speaker survive?

A single similarity score conflates two different failures, so this measures them separately. Retention: how close a reconstruction sits to its own source. Dispersion: how tightly different reconstructions of the same speaker cluster. A decoder can move a voice consistently — a stable new identity — or make the speaker wander. Those are not the same problem and they do not have the same fix.

Three separately calibrated speaker encoders, chosen mechanically from a previously frozen fourteen-encoder study before any of these numbers existed. Cosine values are encoder-specific and never averaged. The three encoders show broadly similar ordering with some pairwise reversals, which is exactly why a bare cosine without its encoder name is not a result.

The streaming detector compares each arm against its own offline decode on the same recording, so no ceiling or floor enters and no cross-arm ranking is implied. Two distinct quantities, never quoted as one range: S1, the offline↔streamed cosine, and S2, the paired retention delta.

For the three causal arms, across all three encoders: S1 0.999866–0.999946, S2 −3.9×10⁻⁵ to −5.3×10⁻⁶. A near-zero median additional change, under the tested encoders, supported recordings and imposed chunking regime. Small, and consistently negative — this study establishes no minimum detectable change and no equivalence threshold, so that is not a claim that the difference is below what the instrument can see.

The other thirteen lose −0.078 to −0.742 on S2, which is nowhere near that scale. For one arm the offline-versus-streamed cosine itself reaches −0.013: the streamed output is essentially orthogonal to that arm's own offline decode in this embedding space. That says nothing about waveform or linguistic relationships, which were not measured.

This agrees with Gate 4's separate streaming detector — an aligned maximum-error detector on the waveform — which found the same three arms at 0.69–0.74% error where every other arm sat at 105–225%. Two measurements sharing no implementation, the same three survivors.

They do share their input: the same reconstructions of the same corpus. So the agreement is corroborating evidence, not proof. Two implementations agreeing makes an independent error in both less likely; it cannot rule out an error in either, and it says nothing about an artefact of the material both were computed on.

What this does not establish

  • Perceptual quality was never measured. No listening test was run. Which decoder sounds best is not established and no sentence here implies it. Two metrics ranked a phase-blind algorithm first; both are blind to the same thing.
  • The identity result is about speaker-encoder representations, not about how audio sounds or about cloning quality.
  • A large majority of cells fall outside the quality metric's validated support domain. Support is carried as a covariate, never as a silent exclusion.
  • For one decoder family, dispersion and band-limiting cannot be separated in this design, so no causal claim is made about it.
  • Seven speakers, one studio, one recording chain, English. Absolute values are not comparable to VoxCeleb-scale benchmarks.
  • Thirteen arms were driven with chunked context they were never designed for. The result is the cost of streaming a decoder that was not built to stream — not a defect report.

The part I got wrong, and why it is in the article

Six rounds of independent adversarial review ran against this work. Four before publication found three defects that changed conclusions — a reference that admitted pairs the measurement forbade, a ceiling measured on the wrong duration, and a pair-grouping convention that was order-dependent.

Then a fifth round ran after publication, against the published package, and found what four earlier rounds structurally could not: they had all audited the private research tree, not what shipped from it.

All three of my advertised public reproduction commands failed from a clean copy of the release. The scripts resolved their data relative to where they sat in my private tree — one directory above the artifacts — and the release ships them flat. The claim audit opened a deliberately withheld file unconditionally. And the verifier I pointed readers at was the manifest of the private tree, which cannot verify a redacted package and duly reported 26 of 32 artifacts as drift. A first-audio range had borrowed a steady-state number for its upper end. Three sentences claimed more than the measurements carry.

A sixth round then attacked the correction itself and found five more, two of them claims in the repair that outran their evidence in the same way as the originals.

The pattern has now recurred six times in different disguises: verification that checks what is present rather than what is required. The estimator without its reference. The instrument without its evidence. The manifest without its inventory. A package that verified its own private origin rather than what it shipped. A privacy scanner that listed the paths it knew about instead of declaring what counts as private. A verifier that printed its own integrity fingerprint without ever comparing it to anything.

That last one is my favourite, because it looks like working code. It computed the hash. It printed the hash. It printed the recorded hash right underneath. It never compared them, so corrupting the fingerprint exited 0. Printing a number beside another number is not a check. Only a comparison that can fail is one.

All of it is corrected in v1.0.1, itemised with the original wording beside the corrected wording. No measurement changed — 51,495 non-string scalar leaves were compared across every JSON artifact common to both versions, and none differ. v1.0.0 stays permanently available, because deleting the version that was wrong would be the same mistake in a different disguise.

Reproducing it

Three commands run from the public package with no private input: regenerate every table, re-trace every headline claim to the artifact it came from, and re-hash the bytes that shipped.

That is a consistency check, and the README now says so rather than overclaiming. It is not independent reconstruction — rebuilding the aggregates and their confidence intervals needs the withheld per-recording rows and speaker embeddings, which are not published because an embedding of a named consenting human is biometric data. One published claim is marked WITHHELD rather than PASS in the public audit for exactly that reason. Source audio, embeddings and the 35.65 GB of reconstructions stay private; the reconstructions are retained because human listening and adversarial detectability are declared later phases.

CC BY 4.0.

Source: dev.to

arrow_back Back to Tutorials