Replacing Video Backgrounds with Flux, SAM, and ControlNet
I was testing out some new workflows and stumbled on a way to replace video backgrounds that actually works better than I expected. It’s not perfect, but with the right setup, you can get pretty clean results without needing a green screen or manual rotoscoping. Here’s how I did it.
The Tools I Used
The workflow combines a few models:
- Flux for the final image generation — it handles the new background and blends everything together.
- SAM (Segment Anything) to isolate the subject from the original video.
- ControlNet to keep the subject’s pose and details consistent.
I didn’t expect SAM to work this well for video, but with the right settings, it does a decent job at masking frame-by-frame. The real trick was getting Flux to regenerate the background while keeping the subject intact.
How It Works (Roughly)
First, I ran the video through SAM to get a mask for the subject. This part can get a little messy if the footage isn’t clean, but for simple scenes, it’s surprisingly effective. Then, I fed that mask into ControlNet to guide Flux while generating the new background.
The key was using ControlNet’s scribble mode to loosely outline the subject, so Flux wouldn’t try to “reimagine” it too much. Without that, the model would sometimes warp the person or object in weird ways.
What Worked (And What Didn’t)
- Good: Static backgrounds or slow-moving scenes? Works great. SAM picks up the subject cleanly, and Flux fills in the gaps convincingly.
- Bad: Fast motion or complex edges (like hair)? Still hit or miss. Sometimes the mask flickers, and Flux struggles with fine details.
I’ll share the full workflow soon, but if you’re curious, you can grab the SAM checkpoint from Meta’s repo and the Flux models from Hugging Face. The ControlNet setup is the same as the one in the ComfyUI docs.
Anyway, it’s not a magic solution, but for quick edits, it’s way faster than manual masking. Let me know if you’ve tried something similar — I’m still tweaking the settings.
🔧 Key Steps Covered:
- Selecting and masking objects in videos with Florence and SAM.
- Generating AI-based backgrounds using the Flux model.
- Ensuring accuracy and mask consistency with ControlNet.
- Bonus tips for refining videos with Adobe After Effects.
