AI video generation has taken a huge leap with WAN 2.1, especially when combined with Control LoRA and depth maps. If you’ve struggled with flickering or inconsistent motions in tools like AnimateDiff, this method offers smoother, more stable results.
In this guide, I’ll walk you through my tested workflow—using depth maps for motion control and text prompts for styling—so you can create high-quality AI videos without expensive hardware.
What You’ll Need
✅ Control LoRA for Depth Maps (Downloadable safetensors )
✅ Control LoRA for Tile (Downloadable safetensors )
Step 1: Setting Up Depth Maps for Motion Control
First, we extract a depth map from the reference video using Depth Anything V2. This black-and-white image acts as a motion guide.
How It Works:
- The Control LoRA reads the depth map and replicates movements.
- Your text prompts define the style (e.g., “cyberpunk dancer in neon lights”).
Pro Tip: If you want precise styling, use Florence 2 to auto-generate prompts from a reference image.
Step 2: Running the Workflow in ComfyUI
Here’s the breakdown:
- Load the Model – Use the 1.3B One 2.1 text-to-video model for fast results.
- Connect Depth Map – Feed it into the Control LoRA node.
- Dual Samplers –
- First Sampler (10 steps): Rough outline of motion.
- Second Sampler (10 steps): Refines details.
- Generate Video – Adjust steps (e.g., 15+15 for higher quality).
My Test Results:
- 3-second clip → Rendered in under 2 minutes (RTX 4090).
- Output: Same dance moves, but with a new outfit and background.
Step 3: Enhancing Quality (Upscaling + Tile LoRA)
For sharper results:
- Upscale the initial output (1.5x).
- Apply Tile Control LoRA to reduce artifacts.
- Optional: Use Segment Anything for selective styling (e.g., only modify the dancer’s clothes).
