Putting a processed face back into the frame without a visible rectangle

Most face pipelines do not work on the whole picture. They cut out the face, process the small crop, and then have to put it back. That last step sounds trivial. It is not. Paste the crop back with a plain assignment and you get a rectangle on the frame, a patch of skin a shade off from the cheeks around it, an edge you can trace with your finger.

This is a walk through paste_pic, the function that does the putting-back in Wunjo Make. It is short. The interesting part is two things: the coordinate bookkeeping that gets the crop to the exact pixel it came from, and one call, cv2.seamlessClone, that makes the paste read as part of the frame instead of a sticker on top of it.

To be clear up front, this is a different job from the lip-sync seam blend I wrote about earlier. That one blends only the mouth region with a Laplacian pyramid. This one pastes the whole processed face crop back into the full frame, and it leans on a single Poisson clone to do it.

The crop the model returns was lit by the model, not by the room. Pasting it back means matching the room. Photo: Unsplash.

The setup: a crop, and the note that says where it came from

Earlier in the pipeline, the face was detected and cropped out of the original image. paste_pic gets two things back: the processed frames (the crop, one per video frame) and crop_info, the receipt that records exactly where that crop was taken from.

if len(crop_info) != 3:
    print("you didn't crop the image")
    return
else:
    clx, cly, crx, cry = crop_info[1]
    oy1, oy2, ox1, ox2 = cly, cry, clx, crx

crop_info[1] is the box: left x, top y, right x, bottom y. The function renames it into the same up-down, left-right order it uses everywhere else (oy1, oy2 for the vertical span, ox1, ox2 for the horizontal). This is not busywork. Half of getting a paste right is keeping x and y straight, because OpenCV indexes arrays as [row, column], which is [y, x], the opposite of how you say coordinates out loud. Naming the four numbers once, clearly, is cheaper than debugging a transposed paste later.

The full original image is loaded separately, either as a still or as the first frame of a video:

if pic_path_type == "static":
    full_img = cv2.imread(pic_path)
else:
    # read the first frame of the source video
    full_img = frame

full_img is the canvas. The processed crops are what we paint onto it.

Resize the crop back to its real size

The model probably worked at a fixed resolution. So before the crop goes anywhere, it gets resized to the exact width and height of the box it was cut from:

p = cv2.resize(crop_frame.astype(np.uint8), (crx - clx, cry - cly))

crx - clx is the box width, cry - cly the box height. If you skip this and the processed crop is even a few pixels off the original box size, every later coordinate is wrong and the paste lands in the wrong place. Resize first, then place.

The mask is the whole crop

Here is the part that surprises people who expect a clever mask. The mask is a solid white rectangle the exact shape of the crop:

mask = 255 * np.ones(p.shape, p.dtype)

Every pixel is 255. That means "clone all of it." There is no feathering, no face-shaped cutout, no soft edge. The whole crop is fair game. The softness does not come from the mask. It comes from the clone method, which is the next line.

seamlessClone: paste the gradients, not the pixels

location = ((ox1 + ox2) // 2, (oy1 + oy2) // 2)
gen_img = cv2.seamlessClone(p, full_img, mask, location, cv2.NORMAL_CLONE)

location is the center of the box, because seamlessClone wants the center point where the patch should land, not the top-left corner. (ox1 + ox2) // 2 is the horizontal middle, (oy1 + oy2) // 2 the vertical middle. Get this wrong by treating it as a corner and the face lands half a face-width off.

Now the part that earns the function its name. A plain paste copies pixel values:

full_img[oy1:oy2, ox1:ox2] = p   # what we are NOT doing

That copies the crop's absolute colors straight in. If the crop is a touch brighter or cooler than the surrounding skin, you see the step at the border, hard and obvious.

cv2.seamlessClone with cv2.NORMAL_CLONE does something different. It is OpenCV's implementation of Poisson image editing (Pérez, Gangnet, Blake, SIGGRAPH 2003). Instead of copying the crop's pixel values, it copies the crop's gradients, the change from each pixel to its neighbor, and then solves for the actual pixel values so that the colors at the border match the surrounding frame exactly.

The everyday version: imagine you have a photo print and you want to drop a smaller print into the middle of it. A plain paste is taping the small print on top, edges and all. Poisson cloning is more like repainting the patch by hand, keeping every brushstroke and detail from the small print, but mixing the paint to match the colors already on the big print right at the seam. You keep the content of the patch. You inherit the lighting and color of the background. The border has nowhere to show up, because the two sides agree at the boundary by construction.

That is why this beats the plain overwrite. The face the model produced was lit by whatever the model imagined. The frame it goes back into was lit by the real scene. The Poisson clone lets the processed face keep its identity and detail while taking on the brightness and color of the actual frame around it. No rectangle, because there is no color step at the edge to draw one.

Two output modes, one clone

The function supports two preprocess modes, and they only differ in what they save, not in how they clone:

if preprocess == "resize":
    frame_w = ox2 - ox1
    frame_h = oy2 - oy1
else:
    frame_h = full_img.shape[0]
    frame_w = full_img.shape[1]

In crop mode (the else), the output frame is the full original size, so you get the whole scene with the new face cloned in. In resize mode, the output is just the box size, and after cloning the function trims back down to the box:

gen_img = cv2.seamlessClone(p, full_img, mask, location, cv2.NORMAL_CLONE)
if preprocess == "resize":
    gen_img = gen_img[oy1:oy2, ox1:ox2]

Note the order. Even in resize mode it clones onto the full image first, then crops the result. That matters, because Poisson cloning reads the colors of the surrounding region to set the boundary. If you cloned onto a bare box with no margin around it, there would be nothing to match against and the whole point would be lost. Clone wide, crop after.

Each finished frame is written out, and at the end the audio is muxed back on:

out_tmp.write(gen_img)
# ...
new_video_name = VideoManipulation.save_video_with_audio(tmp_path, new_audio_path, video_save_dir)

The progress bar is even labelled seamlessClone:, which tells you where the per-frame cost actually goes.

Gotchas worth knowing

A few things I would warn anyone about before they reach for seamlessClone:

• It needs margin. The center location plus the mask must sit fully inside the destination, with room around it. Push a crop to the very edge of the frame and OpenCV will throw, because the clone needs surrounding pixels to read. The center-point math here keeps the box where it was detected, which is normally well inside the frame.
• The center, not the corner. location is the center of the destination box. This is the single most common mistake with this call.
• A full-white mask clones everything. That is the right choice here because the crop is already just the face region from the detector. If your crop contains background you do not want, the mask is where you would carve it out.
• It is not free. Poisson cloning solves a system over the masked region every frame. On a long video that adds up, which is why the loop has its own progress bar.

Wrap-up

paste_pic is maybe forty lines and most of them are file plumbing. The two ideas that matter are small. Keep the crop coordinates honest so the face lands exactly where it was taken from, and use cv2.seamlessClone so it lands without a color step at the edge. The first is bookkeeping. The second is a 2003 paper doing the heavy lifting in one function call.

The full function is in visual_processing/utils/paste_pic.py in the Wunjo Make repo. If you are pasting processed faces back into frames and fighting a visible box, swapping a plain assignment for seamlessClone is the first thing I would try.

References

• Pérez, Gangnet, Blake. "Poisson Image Editing." ACM SIGGRAPH 2003. (The method cv2.seamlessClone implements.) PDF