Multimodal Large Language Models (MLLM): The Future of Tackling Real-World Complexities

Multimodal understanding truly represents human perception

Human experiences are like a puzzle with many pieces. We see things, hear sounds, feel textures, taste flavors, and smell scents. All of these pieces fit together to help us understand the world around us.

For example, when we eat an apple pie, we taste it, feel its texture, smell its aroma, and maybe even enjoy how it looks. All of these things together make the experience even better.

Similarly, when we watch a sci-fi thriller, we see the pictures, hear the sounds, and might even feel excited or scared. This combination of senses makes the movie more fun and helps us understand the story better.

In short, our experiences are made up of many different parts that work together to create a complete picture of the world. These parts are all connected and influence each other. To truly understand human experiences, we need to consider all of these parts together, just like a computer program that combines information from different sources to create a meaningful output. This is why multimodal embedding spaces are so important – they allow us to represent and understand human experiences in a way that captures the complexity and interconnectedness of our senses and perceptions.

They are not perfect though

While multimodal semantic space is a promising step towards understanding human perception, it's important to remember that it's not a perfect representation. Just as artificial neural networks are a simplified approximation of the human brain, multimodal semantic space is a limited model of how humans perceive the world.

MLLM extend LLM capabilities

MLLMs are like upgraded LLMs. They can do more than just understand words. They can also understand pictures, sounds, and even videos. This makes them better at doing things like describing images, answering questions about videos, and creating new content that combines words and pictures.

What is LVLM (Large Vision Language Models)?

LVLM are close cousins of MLLM. Both understand multiple modalities however LVLMs are specifically tailored for tasks involving vision and language, making them particularly well-suited for applications like image captioning and visual question answering. On the other hand, MLLM are still focused on language tasks.

Some examples:

LVLM and MLLM are often used interchangeably. However, for better clarity, here are some specific examples:

LVLMs (Large Vision Language Models)

1. LLaVA (Language-and-Vision-Aware): Derived from the LLaMA language model, integrating a vision encoder.
2. CLIP (Contrastive Language-Image Pre-training): A foundational model for multimodal tasks.
3. FLAVA (Flow-based Language and Vision Attention Model): Known for its strong performance in various multimodal tasks.
4. Qwen-VL: A versatile model for tasks like image captioning, visual question answering, and text-to-image generation.
5. GPT-4V: OpenAI's multimodal version of GPT-4, capable of processing images and text simultaneously.

MLLMs (Multimodal Large Language Models)

1. DALL-E 2: Known for generating high-quality images from text descriptions.
2. Stable Diffusion: A popular text-to-image generation model.
3. PaLM 2: Google's powerful multimodal language model.
4. Imagen: Another text-to-image generation model known for its high-quality outputs.
5. Midjourney: A popular AI art generator that can create images from text descriptions.

Multimodal RAG (Retrieval Augmented Generation)

Image 1 above shows how Multimodal RAG works. Multimodal Embedding models (like CLIP or ViT) help capture the "complexity and connections" between different types of information, such as video, images, text, and audio.

During inference, the user query is converted into an embedding, followed by a semantic search in the multimodal semantic space to retrieve relevant data from the vector database. Finally, the relevant data and search query are sent to the LVLM (Large Vision-Language Model) for inference, resulting in a high-quality response.

Example:

We will take help of LVLM to analyze a video titled as :SpaceX launched to space to bring back stranded astronauts ( https://www.youtube.com/watch?v=-SIyey_Yo1k)

>>> ' can you describe this image "..\space1.jpg"
llava: Added image '..\space1.jpg'
 This image shows a screenshot of a television screen displaying an aerial view of spacecraft launchpad,
presumably at NASA's Kennedy Space Center. The video is titled "NASA to begin sending people to the moon," with
text underneath indicating that "Spacecraft Launches."

There are overlaid news-related elements: in the bottom left corner, there is a news logo and the caption
"Breaking News" with additional text reading "Spacecraft Launches to Retrieve Stranded Astronauts," suggesting an
event of concern or importance. 
The background image appears to be a satellite view of Earth with a cloud-covered sky, and there are blurred texts
visible on the bottom left corner of the screen, likely referring to additional context or information about the
event being covered by the news program.        

Now look at another frame from mid of the video.

>>>  can you describe this image "..\space3.jpg"
llava: Added image '..\space3.jpg'
 The image shows a split-screen view of two different scenes. On the left side of the screen is an aerial view of a spacecraft
launchpad, presumably at NASA's Kennedy Space Center. There are blurred texts visible on the bottom left corner of the screen,
likely referring to additional context or information about the event being covered by the news program.

On the right side, there is a news segment featuring two individuals in what appears to be a spacecraft environment. One person
is standing and holding a microphone, while the other individual is seated with their hands resting on the control panel of the
cockpit. The setting suggests they are in a simulated spacecraft, possibly part of a news report or documentary, discussing the
upcoming launch to retrieve stranded astronauts.

The top right corner has a news logo and the caption "Breaking News" with additional text reading "Spacecraft Launches to
Retrieve Stranded Astronauts." 

The overall style of the image suggests it is a still frame from a television news program or video, meant to convey
information about a space exploration event.        

We used LVLM to analyze the video titled SpaceX Launched to Bring Back Stranded Astronauts. The model described an image from the video in great detail, even without additional context, showing its capability to understand visual content as a human might.

LVLMs analyze videos frame by frame. If a frame has captions or encoding, it uses them. If not, the model generates its own interpretation based on its training, much like how a person would describe what they see. This ability brings LVLMs closer to human-level understanding by connecting visuals, text, and audio to create a well-rounded comprehension

Conclusion:

Multimodal understanding, as shown by MLLMs and LVLMs, mimics how humans process information by combining data from different senses. Just as we use sight, sound, and touch to form a full experience, these models merge information from images, text, video, and audio to create a deeper understanding.

LVLMs, in particular, excel at analyzing and interpreting visual and linguistic data, approaching human-like comprehension. Their ability to generate insights from multimodal data marks a major step forward in AI's ability to "see" and "understand" the world.

This technology is transforming how we handle complex, interconnected information, and its use will continue to grow, bringing AI closer to human perception in many applications.

I’m eager to hear your insights and experiences; your perspective is invaluable!

#MultimodalAI, #AIInnovation, #MachineLearning, #ArtificialIntelligence, #ComputerVision, #NaturalLanguageProcessing, #DeepLearning, #FutureOfAI, #MLLM, #DataScience, #AIApplications, #TechInnovation, #VisionLanguageModels, #GenerativeAI, #AIResearch #GenAIi #LLM #MLLM #LVLM