VACE: All-in-One Video Creation and Editing

Today's paper introduces VACE, an all-in-one model for video creation and editing. VACE unifies multiple video tasks including reference-to-video generation, video-to-video editing, and masked video-to-video editing within a single framework. This approach not only reduces deployment costs but also enables creative combinations of different video manipulation capabilities.

Method Overview

VACE (Video All-in-one Creation and Editing) is built on the Diffusion Transformer (DiT) architecture, which provides a strong foundation for handling long video sequences.It introduces a unified interface called Video Condition Unit (VCU) that integrates multiple input modalities, including images, videos, references, and masks.

The system organizes different types of inputs—whether they're videos to be edited, reference images/videos, or masks indicating areas to modify—into this standardized VCU format. This allows the model to process various tasks through a consistent pipeline. To help the model understand which aspects of the input should be preserved versus modified, VACE implements a concept decoupling strategy that differentiates between editing and reference information.

The Context Adapter structure injects task-specific concepts into the model. This adapter translates the requirements of different tasks (such as which areas to edit or which reference to follow) into formalized representations across temporal and spatial dimensions. By doing so, it enables the model to flexibly handle arbitrary video synthesis tasks without requiring task-specific training.

The unified approach allows VACE to not only perform individual tasks effectively but also to combine capabilities in novel ways. For example, users can perform subject swapping by combining reference-based generation with inpainting, or create animations by combining frame references with pose control. This compositional nature significantly expands the creative possibilities available to users.

Results

The paper evaluates VACE on a custom dataset of 480 evaluation samples spanning 12 different tasks, comparing its performance against specialized models designed for specific tasks. The experimental results demonstrate that despite being a unified model, VACE achieves performance comparable to task-specific models across various video synthesis and editing scenarios.

The qualitative results showcase VACE's versatility in handling diverse tasks such as subject removal and recreation through inpainting, canvas extension through outpainting, temporal extension, structure transfer, colorization, pose transfer, and motion transfer. Furthermore, the model excels at task composition, enabling complex operations like "swap anything" (combining subject reference with inpainting) and "animate anything" (merging frame reference with pose control).

Conclusion

VACE represents a significant step toward unified video synthesis and editing. By integrating multiple capabilities into a single model through its Video Condition Unit and Context Adapter architecture, it provides a versatile and efficient solution for video content creation. The model not only performs competitively on individual tasks but also enables novel compositional applications that were previously difficult to achieve. For more information please consult the full paper.

Congrats to the authors for their work!

Jiang, Zeyinzi, et al. "VACE: All-in-One Video Creation and Editing." arXiv preprint arXiv:2503.07598 (2025).