Every pixel in a photograph of a person against a wall is a blend of two things. The compositing equation is the whole problem in one line:
C = αF + (1 - α)B
C is the pixel you can see. F is the foreground colour, B is the background colour, and α is how much of that pixel belongs to the foreground. Solid shirt, α is 1. Empty wall, α is 0. A single strand of hair crossing that pixel, α is 0.3 and the pixel is genuinely part person, part wall.
Now count the unknowns. F is 3 numbers, B is 3 numbers, α is 1. That's 7. And you know 3, the RGB of C.
Seven unknowns, three equations. The problem is underdetermined and no amount of cleverness makes it not underdetermined. Everything that follows is about finding constraints from somewhere else.
What a green screen actually buys you
It gives you B.
If you know the background is (0, 177, 64) everywhere, you've eliminated 3 unknowns and the system becomes solvable per pixel. That's chroma keying, and it's why it's been in use since the 1930s. It turns an ill-posed inverse problem into a colour distance test:
distance = np.linalg.norm(pixel - key_colour, axis=-1)
alpha = np.clip((distance - tolerance) / softness, 0, 1)
That's the core of it. Everything else in a keyer is refinement around the edges.
The reason nobody shoots headshots on green is that the green bounces. Light hits the screen, reflects, and lands on the side of your subject's face and hair. Now your foreground colour F is contaminated by the thing you're trying to remove, and you need spill suppression, which is its own pile of heuristics.
Without a green screen, you get two families of approach
Segmentation. Train a network to output a mask directly. U²-Net and IS-Net are the ones most background removal tools are built on, and they're doing binary-ish classification: person or not person, per pixel.
This is fast, runs in a browser via ONNX, and handles the 95% of the image that's obviously shirt or obviously wall. It falls apart on exactly the pixels where α is between 0 and 1, because the model is being asked for a yes or no on a pixel where the honest answer is 0.3.
That's why cheap background removal gives you either a jagged cut-out edge or a soft halo of the old background. The model picked, and picking was the wrong operation.
Alpha matting. Split the image into three regions first: definitely foreground, definitely background, and an unknown band around the boundary. That's a trimap. Then solve for α only inside the unknown band, using the known regions as colour samples to constrain the equation.
Closed-form matting does this by assuming F and B are locally smooth, which turns the whole thing into a sparse linear system you can actually solve. Deep matting approaches learn the same thing end to end.
Matting is where the quality lives, and it's slower by an order of magnitude, which is why not everything uses it.
Hair is the whole difficulty
I want to be precise about why, because "hair is hard" gets said a lot without the mechanism.
A human hair is about 70 micrometres across. Photographed at portrait distance on a phone, that's well under one pixel wide. So a pixel containing hair contains hair and whatever is behind it, permanently mixed at capture time. The information about where one ends and the other begins was destroyed by the sensor.
You cannot recover it. You can only make a plausible guess, and the guess is constrained by the colours around it.
Which leads to a rule that's genuinely useful when you're taking the photo: the recoverability of your hair edge depends on the contrast between your hair and the wall. Dark hair against a dark bookshelf is unrecoverable, because the mixed pixel and the pure background pixel are the same colour and no algorithm can tell them apart. Dark hair against a light grey wall separates cleanly.
That's not a limitation of the tool. It's information theory. The tool cannot invent a distinction that isn't in the file.
The second failure: colour spill survives a correct alpha
Say you nailed α. You composite onto a new background:
C_new = αF + (1 - α)B_new
And it still looks wrong. There's a faint fringe of the old wall around the shoulders and through the hair.
That's because F itself was contaminated. Light from the old background bounced onto your subject. The semi-transparent pixels carry a colour cast that has nothing to do with the new scene, and mathematically your α was fine.
Spill suppression is a separate pass, and most automatic tools either skip it or apply something crude like desaturating the boundary band. Which is why swapped backgrounds so often have a grey rim that reads as "cut out" even when you can't name what's wrong.
What this means if you're taking the photo
Everything above collapses into four shooting decisions.
-
Stand a metre or more off the wall. Less bounce onto you, less spill in
F, and the background goes slightly out of focus which helps segmentation. - Pick a wall that contrasts with your hair, not your shirt. Hair is where the algorithm needs the signal.
- Avoid backlight. A window behind you puts a bright rim through your hair, which is beautiful and which destroys the hair edge for matting, because now the hair pixels are the same brightness as the background.
- Plain beats interesting. A busy background gives the segmentation model more edges to confuse with your outline.
Do those and even a mediocre segmentation model gives you a clean result. Skip them and the best matting implementation in the world is working with a file that doesn't contain the answer.
Doing it in practice
If you just want to swap the backdrop on a photo you already have, the background options that come out of a matting pipeline look very different from the ones that come out of a plain segmentation mask, and it's usually obvious at the hair line which you're looking at. Zoom in on the edge before you commit to anything.
For the crop and the export you can do it in a browser without installing anything, and if the source photo is the problem rather than the background, BetterPic generates from selfies instead, which sidesteps the matting question entirely because the background was never real to begin with.
The takeaway that isn't obvious
Background removal quality is decided at capture, not in post. The algorithm can only redistribute information that's already in the file, and the mixed pixels at the hair boundary either contain a recoverable distinction or they don't.
One metre off the wall and a background that contrasts with your hair. That's most of it.
Disclosure: I work on BetterPic, one of the tools linked above.