I tried to stop paying $2.99 per backing track. The transcription worked; the accompaniment never did

If you post cover songs, you know the drill. You want an instrumental of a track, you go to a site like Karaoke Version, and you pay about $2.99. They have more than 106,000 of them, and they market the licensing clarity as hard as the audio quality, because that is what you are actually buying.

You are not paying for someone to mute a vocal track. You are paying because the recording you would otherwise use belongs to a label.

A friend of mine skips all of that. He transcribes by ear and builds his own backing tracks. I had that stuck in my head for a while, so I spent a day finding out how far a machine gets doing the same job.

It did not get there.

What makes the story worth writing is that I was wrong about where it broke three separate times, and each time I was confident enough to start building on top of the wrong diagnosis before anything told me otherwise.

Why re-recording is the only distributable path

Two rights sit on top of any commercial track. There is the musical work, and there is the sound recording itself. Strip the vocal out of a released master and the second one still belongs to the label. The processed file is a derivative of their recording.

✕  remove vocals from the master  → sound recording still the label's  → cannot distribute
◯  transcribe and perform again   → a recording I generated            → room to distribute

The composition copyright does not evaporate because you played it again. Mechanical licensing still applies, and I am not a lawyer, so I am not going to draw the line for you. What I could measure is whether "perform it again, by machine" holds up technically.

That is the same logic the paid services run on. They re-record. I wanted to know what it costs to do it yourself with open source.

One track: "Koi Darou" by the Japanese band wacci, 4 minutes 50 seconds. Every number below comes from that one file. It is a full band arrangement with synths and backing vocals layered on top, which is to say it is not an easy transcription target.

The stack

Three steps. Separate, transcribe, render.

I filled in the license column before anything else, because distribution was the whole point. One non-redistributable component anywhere in the chain and the output is stuck no matter how good it sounds.

Compute turned out to be a non-issue. Demucs runs at 2.2x realtime on CPU, so the 4:50 track split into 4 stems in 2 minutes 12 seconds on hardware I already owned.

There goes one of my reasons to buy a GPU.

MT3 and Omnizart are the other multi-instrument transcription options. I did not benchmark either of them. Basic Pitch produced usable output first and the comparison stopped being necessary. MT3 shows up below, but as a README quote, not as a measurement of mine.

Wrong guess 1: "transcription cannot handle pop"

Transcription was my first suspect. A monophonic line, sure. But pop with synths, guitars and stacked backing vocals turning into discrete notes? I did not believe it.

Separation is what made it work. The MT3 README warns that the model was not trained on singing voice, so feeding it audio with vocals produces strange output. Separate first and that constraint never gets a chance to apply.

To check that separation was actually doing something, I ran three inputs through a pitch detector and compared the confidence distributions.

The mix and the instrumental are both locked onto bass frequencies and never track the melody. The vocal stem has none of that low end left, and the histogram overlap between vocals and no-vocals dropped to 0.09. Separation works.

Then I ran Basic Pitch on three stems. The in-range ratio is defined like this:

#: valid pitch range per instrument (MIDI)
EXPECTED = {
    "bass": (28, 55),      # E1 - G3
    "other": (48, 84),     # C3 - C6
    "vocals": (45, 79),    # A2 - G5
}

low, high = EXPECTED[stem]
in_range = float(np.mean((pitches >= low) & (pitches <= high)))

Results:

The rightmost column is what settled it. Three stems were transcribed independently and every pitch class that came out fits the E major scale (E F# G# A B C# D#). Three chains that never see each other's output landed on the same key. That does not happen by accident. The vocal result also matches a score I had already generated through a different path (SwiftF0).

Transcription was working.

My first suspect had been innocent the entire time.

Wrong guess 2: "the synthesis quality is the wall"

With transcription standing up, I decided synthesis was the problem. MIDI through a sine wave sounds terrible, obviously. Render it with a real instrument and it should become listenable.

I raised the timbre in stages and had someone listen at each one.

Four steps up in timbre and the verdict barely moved. If swapping a sine wave for a SoundFont piano changes nothing, the timbre is not what is broken.

I suspected clutter and wrote a note-thinning filter. The other stem carries 11.2 notes per second. I assumed dense chords; it was actually short notes in sequence, with synth pad and reverb tails being picked up as notes.

#: drop notes shorter than this (seconds)
MIN_DURATION = 0.2
#: drop notes quieter than this, killing weak false positives
MIN_VELOCITY = 50
#: polyphony cap. it is an accompaniment, it needs a chordal floor
MAX_SIMULTANEOUS = 5

notes = [
    n
    for n in instrument.notes
    if n.end - n.start >= MIN_DURATION and n.velocity >= MIN_VELOCITY
]

3228 notes came down to 1770 and it got slightly better. Slightly better and still unlistenable.

The fact that the music box scored highest says something too. It plays melody only. When the version with the fewest notes wins, the missing ingredient is not audio quality.

Wrong guess 3: the arrangement step was missing

So I went and looked at how karaoke backing tracks and music box arrangements actually get made. In both cases, a human arranges and sequences.

Shigeshi Miki, president of C-Music, a company that produces karaoke audio, describes the workflow:

All data entry is done by ear. We do not receive MIDI data from the record labels, though we often get the track before release so we can start early.
-- Shigeshi Miki (C-Music) / DTM Station (Ken Fujimoto)

Nobody hands them data. Someone sits down with the record and works it out note by note, and in Japan there is a MIDI certification that maps directly onto this job, which should tell you how specialized the work actually is.

Music box arrangements are the same. Most of what YouTube calls a "music box arrangement" is electronic audio with a music box timbre, freely rearranged. Actual recordings of an actual music box turning are hard to find.

Side by side:

The middle column was completely empty.

Tracing the original and rebuilding it into something that works on the target instrument are different jobs, and a music box arrangement only holds together because someone already did the second one. Swapping the timbre on a raw transcription does not get you there.

That middle column is exactly what my friend was doing. The machine can take over pulling the notes off the recording. Rebuilding those notes into a playable shape, he was doing by hand. When he described it to me, I did not count that as a step. "Transcribing by ear" is one phrase covering two different jobs.

Reproducing it

# 1. split into 4 stems
python -m demucs -n htdemucs -d cpu -o <output-dir> <audio>

# 2. build an isolated venv for transcription
uv venv --python 3.10 amt-venv
VIRTUAL_ENV=$PWD/amt-venv uv pip install basic-pitch 'numpy<2' 'setuptools<81' scipy

# 3. hand it the paths and run
export SONGFIT_STEMS=<demucs output>/htdemucs/<track>
export SONGFIT_WORK=<working dir>
./amt-venv/bin/python amt_check.py bass other vocals   # transcribe and evaluate
./amt-venv/bin/python render_fluid.py all              # synthesize

Step 2 is where I lost time. basic-pitch requires numpy<2, so dropping it into the venv of a project on numpy 2 breaks that project. Keep the transcription venv separate.

Paths go through environment variables rather than arguments so that generated audio never lands inside the repository by accident. Derivative material stays out of range of a git add ..

Where it broke, in order

The first two rows, my first two suspects, were both innocent. I suspected things in the order of how easy they are to adjust, which felt like debugging and was very nearly the opposite of it. Transcription accuracy and synthesis timbre both have parameters: you turn a knob and a number moves. It is comfortable to suspect a place where numbers move.

What was actually empty had no parameters at all. Where the step itself does not exist, there is no tuning surface to find.

An expiry date on this, since it describes open source as of August 2026:

Next stop is chord estimation on the other stem. Getting a chord per bar out of those 3228 notes would give me something to rebuild an accompaniment from.

If you have hit the same wall, I want to know which end you started attacking it from.