devLap

Filling a hole in video with the next frame, not a guess

python dev.to

You paint a mask over a person walking across a shot and ask the app to remove them. Behind that person, in this exact frame, there is nothing. The pixels do not exist. But two frames later the person has moved, and the patch of wall they were standing in front of is right there, fully visible.

So why would you ask a network to imagine that wall when the camera already filmed it?

That is the whole idea behind the object remover in Wunjo Make. Most of the hole does not get hallucinated. It gets carried in from frames where the same spot was not covered. The code does this with optical flow, and the interesting part is the ordering: flow first, pixels second, and a transformer only for the leftovers. This is a walkthrough of that pipeline as it runs in portable/src/visual_processing/remover/.


Across a strip of frames, a covered spot in one frame is usually open in another. Flow is the map that connects them. Photo: Unsplash.

The plan in plain words

Think of flow as a set of tiny arrows, one per pixel, that say "this pixel moved here in the next frame." If you have those arrows, you can follow a pixel backward and forward through time. A spot hidden by the object in frame 10 might be visible in frame 7. Follow the arrows from 7 to 10 and you can drop frame 7's real pixel into frame 10's hole.

There is a catch. The arrows themselves are missing inside the mask. Optical flow is computed from image content, and inside the hole the content is the object you want gone. So the motion of the background under the object is unknown. The pipeline solves this in a specific order:

  1. Estimate flow between every pair of adjacent frames (RAFT).
  2. Fill in the flow inside the masked region, so the arrows exist even where the background was hidden (recurrent flow completion).
  3. Use that completed flow to warp real pixels from neighbor frames into the hole, and shrink the mask wherever this works.
  4. Whatever is still masked after all that, because it was never visible in any frame, gets painted by a transformer.

Four stages, and only the last one invents anything.

Stage 1. The arrows (RAFT)

The flow estimator is RAFT (Teed and Deng, ECCV 2020). In the pipeline it runs on short clips and produces both directions at once:

flows_f, flows_b = self.fix_raft(frames[:, f:end_f], iters=raft_iter)
Enter fullscreen mode Exit fullscreen mode

flows_f is forward flow (frame t to t+1), flows_b is backward (t to t-1). raft_iter defaults to 20. RAFT is iterative: it starts from a zero flow field and refines it a fixed number of times, each step looking up a correlation volume and adding a small delta. You can see that loop in raft.py:

for itr in range(iters):
    coords1 = coords1.detach()
    corr = corr_fn(coords1)              # index correlation volume
    flow = coords1 - coords0
    net, up_mask, delta_flow = self.update_block(net, inp, corr, flow)
    coords1 = coords1 + delta_flow       # F(t+1) = F(t) + delta
Enter fullscreen mode Exit fullscreen mode

The clip length is tied to resolution. The code drops it as frames get bigger:

if frames.size(-1) <= 640:
    short_clip_len = 12
elif frames.size(-1) <= 720:
    short_clip_len = 8
elif frames.size(-1) <= 1280:
    short_clip_len = 4
else:
    short_clip_len = 2
Enter fullscreen mode Exit fullscreen mode

This is a memory budget, not an accuracy choice. RAFT's correlation volume is large, so wider frames mean fewer of them in VRAM at once. The result of this stage is gt_flows_bi, a clean flow field everywhere except inside the mask, where the arrows are garbage because they were computed over the object.

Stage 2. Completing the flow inside the hole

This is the part people skip past, and it is the one that makes the whole thing work. Before you can move pixels along the arrows, you need arrows inside the hole. The background moved under that object even though you could not see it, and the model has to guess that motion.

Here is the key intuition: guessing motion is far easier than guessing pixels. A wall's appearance can be anything. A wall's motion is almost always smooth and continuous with the motion just outside the mask. If the visible background is panning left, the hidden background behind it is panning left too. Flow is a slowly-varying field, so you can extend it inward from the edges of the hole with real confidence.

The completion network starts by deleting the unreliable flow inside the mask, on purpose:

masked_flows_forward  = masked_flows_bi[0] * (1 - masks_forward)
masked_flows_backward = masked_flows_bi[1] * (1 - masks_backward)
Enter fullscreen mode Exit fullscreen mode

Multiplying by (1 - mask) zeroes the flow wherever the mask is 1. It does not try to salvage the bad arrows. It throws them out and rebuilds them. The masked flow plus the mask itself go into a 3D convolutional encoder, a dilated middle block, a bidirectional propagation module, and a decoder, all in recurrent_flow_completion.py:

inputs = torch.cat((masked_flows, masks), dim=1)
x = self.downsample(inputs)
feat_e1 = self.encoder1(x)
feat_e2 = self.encoder2(feat_e1)
feat_mid = self.mid_dilation(feat_e2)
...
feat_prop = self.feat_prop_module(feat_mid)   # bidirectional, second-order
...
flow = self.upsample(feat_d1)
Enter fullscreen mode Exit fullscreen mode

The propagation runs both directions, forward and backward, so a frame's flow is informed by its neighbors on both sides. The backward pass literally flips the clip in time, completes it, and flips it back:

masked_flows_backward = torch.flip(masked_flows_backward, dims=[1])
masks_backward        = torch.flip(masks_backward, dims=[1])
pred_flows_backward, _ = self.forward(masked_flows_backward, masks_backward)
pred_flows_backward    = torch.flip(pred_flows_backward, dims=[1])
Enter fullscreen mode Exit fullscreen mode

Then the last step, and it is a small line that matters a lot. combine_flow keeps the original measured flow everywhere outside the mask and only uses the predicted flow inside it:

pred_flows_forward  = pred_flows_bi[0] * masks_forward  + masked_flows_bi[0] * (1 - masks_forward)
pred_flows_backward = pred_flows_bi[1] * masks_backward + masked_flows_bi[1] * (1 - masks_backward)
Enter fullscreen mode Exit fullscreen mode

The network never gets to "improve" the flow it did not need to touch. Outside the hole, RAFT's measurement wins. Inside the hole, the prediction fills in. You end up with one continuous flow field with no seam at the mask boundary.

There is an edgeDetector in this network too, but check the forward pass: it only runs under self.training. At inference it is skipped. So in the shipped app it contributes nothing, it was a training-time loss helper.

Stage 3. Moving real pixels along the completed flow

Now the arrows exist everywhere. Time to follow them and carry pixels in. First the object is blanked out of the frames:

masked_frames = frames * (1 - masks_dilated)
Enter fullscreen mode Exit fullscreen mode

Then image propagation runs with the completed flow:

prop_imgs, updated_local_masks = model.img_propagation(
    masked_frames, pred_flows_bi, masks_dilated, 'nearest'
)
Enter fullscreen mode Exit fullscreen mode

Inside, this is a BidirectionalPropagation module built with learnable=False. That flag matters. The feature-level propagation later in the pipeline is a trained network. This one is not. It is plain warping and bookkeeping, no weights, because here we are moving actual RGB pixels and we do not want a network reinterpreting them. The non-learnable branch in propainter.py is where the real work happens:

mask_prop_valid = flow_warp(mask_prop, flow_prop.permute(0, 2, 3, 1))
mask_prop_valid = self.binary_mask(mask_prop_valid)

union_vaild_mask = self.binary_mask(mask_current * flow_vaild_mask * (1 - mask_prop_valid))
feat_prop = union_vaild_mask * feat_warped + (1 - union_vaild_mask) * feat_current
mask_prop = self.binary_mask(mask_current * (1 - (flow_vaild_mask * (1 - mask_prop_valid))))
Enter fullscreen mode Exit fullscreen mode

Read union_vaild_mask slowly, because it is the gate that decides "can I trust the pixel I just dragged in here." Three conditions, all multiplied:

  • mask_current is 1 where this frame still has a hole to fill.
  • flow_vaild_mask is 1 where the flow is trustworthy.
  • (1 - mask_prop_valid) is 1 where the source pixel I am pulling from was itself not a hole.

Only where all three hold does the warped pixel get written in. Everywhere else the frame keeps what it had. Then mask_prop updates: the hole shrinks wherever the fill succeeded.

That flow trust check is its own neat trick, fbConsistencyCheck. Follow a pixel forward with the forward flow, then backward with the backward flow. If you land where you started, the flow is consistent and probably correct. If you land somewhere else, something is occluded or wrong, so do not trust it:

flow_bw_warped = flow_warp(flow_bw, flow_fw.permute(0, 2, 3, 1))
flow_diff_fw   = flow_fw + flow_bw_warped
mag_sq_fw      = length_sq(flow_fw) + length_sq(flow_bw_warped)
occ_thresh_fw  = alpha1 * mag_sq_fw + alpha2
fb_valid_fw    = (length_sq(flow_diff_fw) < occ_thresh_fw).to(flow_fw)
Enter fullscreen mode Exit fullscreen mode

flow_fw + flow_bw_warped should cancel to near zero for a good pixel. The threshold scales with the flow magnitude (alpha1 = 0.01) plus a constant floor (alpha2 = 0.5), so fast motion gets more slack than slow motion. This is what stops the propagation from smearing a pixel across an occlusion boundary.

After this stage the frames are updated and the mask is smaller. The bookkeeping line that writes it back:

updated_frames = frames * (1 - masks_dilated) + prop_imgs.view(b, t, 3, height, width) * masks_dilated
updated_masks  = updated_local_masks.view(b, t, 1, height, width)
Enter fullscreen mode Exit fullscreen mode

For a lot of videos, a clean pan across a static background, this stage alone fills almost the entire hole with genuine filmed pixels. The output is real footage rearranged in time, not a generated guess.

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

Stage 4. The transformer fills only what was never seen

Some pixels were covered in every single frame. The object stood still over a spot the camera never caught. No amount of flow can carry in a pixel that does not exist anywhere. That is the only thing left for a generative model.

The inpainting generator (ProPainter, Zhou et al., ICCV 2023) runs on a sliding window of neighbor frames plus a few reference frames:

pred_img = model(selected_imgs.half(), selected_pred_flows_bi,
                 selected_masks, selected_update_masks, l_t)
Enter fullscreen mode Exit fullscreen mode

Notice it takes two masks. selected_masks is the original hole, selected_update_masks is the smaller hole left after stage 3. The encoder gets the masked frame plus both masks stacked as channels, so it knows what was filled by propagation and what still needs inventing:

enc_feat = self.encoder(torch.cat([masked_frames.view(b * t, 3, ori_h, ori_w),
                                   masks_in.view(b * t, 1, ori_h, ori_w),
                                   masks_updated.view(b * t, 1, ori_h, ori_w)], dim=1))
Enter fullscreen mode Exit fullscreen mode

Inside InpaintGenerator.forward, the flow is reused once more, this time to propagate features (the learnable feat_prop_module, downsampled flow at quarter resolution), and only then does the transformer attend across time:

_, _, local_feat, _ = self.feat_prop_module(local_feat, ds_flows_f, ds_flows_b, prop_mask_in, interpolation)
enc_feat = torch.cat((local_feat, ref_feat), dim=1)
trans_feat = self.ss(enc_feat.view(-1, c, h, w), b, fold_feat_size)
trans_feat = self.transformers(trans_feat, fold_feat_size, mask_pool_l, t_dilation=t_dilation)
Enter fullscreen mode Exit fullscreen mode

The transformer is the expensive, creative part, and by the time it runs, most of the hole is already real footage. It only has to dream up the slivers that were genuinely never filmed. That is why the output rarely looks hallucinated: there is very little hallucination in it.

Gotchas you will actually hit

A few things bite when you run this:

  • The flow array is one shorter than the frames. Flow is between pairs, so N frames give N-1 flows. The code slices accordingly, neighbor_ids[:-1] when selecting flows. Off-by-one here corrupts the warps silently, the video does not crash, it just smears.
  • Dilate the mask. The pipeline uses masks_dilated, not the raw mask. A tight mask leaves a halo of the object's edge pixels just outside the hole, and propagation happily carries that halo into other frames. Grow the mask a few pixels so the object's fringe is inside it.
  • Flow completion before propagation, always. If you propagate with raw RAFT flow, the arrows inside the hole are nonsense and you drag object pixels around instead of background. The completion step is not optional polish, it is what makes the warp point at the right source.
  • The edge detector is dead weight at inference. Do not wire it into your inference path expecting an edge map. It returns None outside training.
  • fp16 is fine downstream, not for RAFT. The pipeline runs RAFT in higher precision and casts to half later (selected_imgs.half()), because the correlation lookup is sensitive to precision in a way the conv layers are not.

Why this ordering is the point

You could build a remover that hands the whole masked video to a transformer and says "figure it out." It would invent the background frame by frame, and it would flicker, because an inventor has no reason to invent the same wall twice. The flow-guided approach inverts that. It treats inpainting as a search problem first ("where else in this video is this pixel visible?") and a generation problem only for what the search cannot find. Real pixels are temporally consistent for free, because they are the same pixels.

The code is in portable/src/visual_processing/remover/. The three model files are raft.py (the arrows), recurrent_flow_completion.py (completing the arrows inside the hole), and propainter.py (warping pixels, then the transformer). The orchestration that runs them in order is in __init__.py, in process_video_with_mask.

References

Source: dev.to

arrow_back Back to Tutorials