"Exploring the Frontiers of AI in Creativity: A Deep Dive into Sora's Capabilities and Challenges"

As a former film industry professional and now a generative AI product manager, I offer a rational perspective on Sora.

Firstly, like GPT's creative capability and lack of parameter bias, it cannot replace the creation of profound creativity and compelling narratives. It acts more as a filler for a material library and a secondary processor of existing plots. Even in the face of professional jargon, a lot of illusions occur, and even simulating characters requires prompt engineering and rag engineering for hybrid reinforcement. Essentially, it's all about compressing and refining knowledge and information bodies.

Like the diffusion model and Transformers architecture, Sora faces similar challenges. Its training set mainly comes from video pixel frames and lacks specific physical training and engine parameters.

Sora's soft physical simulation emerged as a "spontaneous attribute" with the large-scale expansion of text-to-video training.

Against some direct objections, such as the view that "Sora doesn't learn physics; it just manipulates pixels in 2D space," might oversimplify Sora's operation. In reality, reducing Sora's work to 2D pixel manipulation is like saying GPT-4 doesn't learn programming but only samples strings, ignoring the underlying complexity and capability of the model.

The essence of the Transformers architecture, whether for text or video, lies in manipulating a series of numbers - whether representing token IDs for text or pixel values for images. This numerical manipulation, backed by algorithms and architectural frameworks, captures a wealth of patterns and relationships, working hard to implicitly compensate in time series and spatial patches.

Sora's soft physical simulation, or its ability to model physical phenomena without explicit physics engine support, is a "spontaneous attribute" that emerged as the model's training scale expanded. This means that although Sora wasn't directly trained to understand or follow physical laws, it can learn how objects move and interact in the real world by analyzing a vast amount of video and sound data, reproducing these physical behaviors visually in generated videos.

For example, to generate executable Python code in a video segment, GPT4 must internally learn grammar, semantics, and data structures. Similarly, for Sora to accurately simulate video pixels, it must learn some "implicit" knowledge of text-to-3D conversion, 3D transformations, ray tracing rendering, and physical rules. This is all about compression and refinement.

Just as GPT4 must internally learn some form of grammar, semantics, and data structures to generate executable Python code, Sora needs to learn some "implicit" knowledge of text-to-3D, 3D transformations, ray tracing rendering, and physical rules to simulate video pixels as accurately as possible. It needs to grasp the concept of game engines to meet its objectives. These conditions form the basis of soft physics, including referring to sound elements for further reasoning.

It's noteworthy that although Sora can generate content that visually conforms to physical intuition, this capability is far from perfect at present. Content generated by Sora may contain illusions and errors that do not conform to real-world physical rules. This indicates that although Sora has made progress in simulating soft physics, it still cannot replace specialized physics engines, especially in scenarios requiring highly accurate and complex physical simulations, such as high-end movie special effects tools (fluid simulation, digital physical effects simulation) and sophisticated game physics engines (UE).

If we disregard interaction issues, UE5 is a (very complex) process for generating video pixels. Sora is also a process for generating video pixels but is based on end-to-end transformers. They are on the same abstract level. The difference is that UE5 is manually crafted and precise, while Sora is purely learned from data and "intuitive."

Sora's synthetic audio capability, obtained from ElevenLabs, indeed provides significant support for its soft physical modeling. This integration not only allows Sora to create coherence in visual effects but also adds a layer of realism on the audio level, making the generated video content more rich and multidimensional. To control audio output through text prompts, Sora needs to understand the dynamics and context of the scene and how these factors affect the production and propagation of sound.

It is controlled by text prompts, but the correct conditions should apply to both text and video pixels. Learning an accurate video-to-audio mapping also requires modeling some "implicit" physical rules in latent space.

End-to-end transformers need to figure out to correctly simulate sound waves:

1. Determine the category, material, and spatial location of each object.

2. Determine the high-order interactions between objects: Is a stick hitting wood, metal, or a drumhead? At what speed?

3. Determine model needs to overlay multiple sound tracks based on their spatial positions.

Learning an accurate video-to-audio mapping requires Sora to model some implicit physical rules in latent space. These rules include identifying the properties of objects, understanding the interactions between objects, recognizing environmental features, retrieving typical sound patterns from internal memory, running soft physical rules to synthesize or adjust sounds, and blending multiple sound tracks in complex scenes. This process demands that Sora not only handle visual information but also engage in complex reasoning and generation of sound information, heavily relying on its capability gained from extensive multimodal data training.

The enhancement of multimodal learning and generation capabilities significantly strengthens Sora's soft physical simulation ability, enabling it not only to simulate physical interactions visually but also to provide matching physical feedback audibly.

However, despite Sora's potential in video content generation and soft physical simulation, it currently cannot reach the level of physical engines used in professional movie effects and game development. It is best suited for material synthesis and as a limited concept generator in the creative preview stage. For low-quality video creators and web novels, it might be a boon.

It's worth mentioning that Sora employs advanced technological methods, compressing video content into latent space and further processing it in patch form, effectively addressing the significant computational resource requirements for processing and generating high-resolution video content. This approach's core lies in the video-to-latent space transformation, where OpenAI's custom tokenizer plays a crucial role, differing from traditional frame-by-frame compression methods by compressing a series of frames into latent space, possibly a key innovation in the Sora model.

Regarding video-to-latent space compression, Sora uses a unique encoder to compress video content into latent space, a crucial step in video processing that allows the model to handle video data more efficiently and reduce computational resource needs.

As for the conversion from latent space to patches, the data in latent space are further converted into patches, representing a highly abstracted version of video content. This "highly scalable" and "efficient" representation method might imply a very high compression ratio, making subsequent processing more efficient.

The best application of Transformers comes into play after the compression and conversion, where the patches are fed into a transformer-based model for further processing and generation, fully utilizing the transformer's capability in handling sequential data.

In terms of valuable prospects, the development of Sora technology not only opens new avenues for AI-generated video content but also offers new possibilities for processing real-time visual data. Its potential applications extend far beyond video content generation, potentially affecting how live broadcasts, surveillance, and other real-time video data are efficiently processed and analyzed.

Sora's capability for efficient information compression and extraction could become a powerful tool for converting complex visual data from the real world into formats manageable by AI models, providing rich training data. Despite Sora's reduced computational power demand through efficient compression, its training and inference processes still require significant computational resources. Future technological developments need to balance between increasing computational power and optimizing algorithm efficiency.

Sora technology might pave new paths for AI model training, especially in handling multimodal data and understanding complex real-world scenarios in a compressed form. Ultimately, soft physics can potentially be integrated into 3D engine models, bridging from pixels to the skies and rapidly building automated virtual worlds.

Sora will not replace game engine developers or visual effects artists; its understanding of physics is fragile and far from perfect. It still produces a significant number of illusions not aligned with our physical common sense and lacks a robust understanding of object interactions, leading to many surreal results. It also cannot replace human high-level intellect or the brilliance of creativity. For the average person, it might assist in democratizing creative production, but ultimately, it's a matter of compression and fitting, merely supplementing elements to accelerate the imagination process. Understanding the technological logic and extrapolating developments, I am neither overly optimistic nor pessimistic; let's wait and see.

Director of the Sino-French AI Innovation Lab, Doone Song