The Era of Generative AI: How Large Multimodal Models are Reshaping Industries in 2024

GenAI: The year of Large Multimodal Model (LMM)

Welcome to 2024: The Era of Generative AI! As we step into this transformative year, we are at the heart of a revolution in artificial intelligence, powered by the incredible rise of Large Multimodal Models (LMMs). These formidable AI engines are pushing the frontiers of technology, enabling an unprecedented integration of text, image, video, audio, and data.

This year signifies a major evolution in AI, where LMMs are not merely a fleeting trend, but the foundational drivers of innovation across diverse industries. Join us as we delve into our article to uncover how LMMs are revolutionizing our global landscape, enhancing interactions and sharpening decision-making in ways previously unimaginable. Prepare for an insightful journey into the future of AI, where the distinction between human and machine intelligence is becoming increasingly blurred, opening up new and thrilling possibilities.

Facts

The world of AI has witnessed during 2023 significant milestones in the development of Large Multimodal Models (LMMs). These models have revolutionized how machines understand and interact with multiple forms of data, including text, images, audio, and more.

Here's a snapshot of some groundbreaking LMMs:

Why It Matters

The relevance of LMMs lies in their ability to process and synthesize information from various data sources. This multimodal approach mirrors human cognitive processes more closely, allowing for more intuitive and efficient interactions between humans and AI systems. The advancements in LMMs are crucial for developing AI that can understand and operate in our complex, multifaceted world.

What is LMM?

LMM stands for Large Multimodal Model, a type of advanced artificial intelligence system that has the capability to process and understand information from multiple data types or modalities simultaneously. Unlike traditional AI models that typically specialize in one type of data, such as text or images, LMMs can handle a diverse range of inputs including, but not limited to, text, images, audio, and even video.

The "multimodal" aspect refers to the model's ability to integrate and interpret these different forms of data in a cohesive manner. This integration allows LMMs to have a more comprehensive understanding of complex scenarios and tasks, similar to how humans perceive and process multifaceted information from the world around us.

LMMs are particularly significant in the field of AI because they represent a step towards more advanced, human-like artificial intelligence. They are used in various applications such as enhancing natural language processing, image and speech recognition, and even in generating human-like responses in chatbots or virtual assistants. The development of LMMs marks a significant advancement in AI's capability to interact with and understand the world in a more holistic and nuanced way.

Example of Use Cases

LMMs open up a plethora of use cases across various industries:

1. Telecom:LMM-powered Smart Agents for customer inquiries in any written or voice channel.Virtual assistants to guide users in resolving connectivity issues using images and video.
2. Banking: Automation of processes that require analyzing large quantities of documents.Financial AI Advisor, LMM-driven financial advice tailored to individual customer profiles, with better user experience.
3. Insurance: Automate claim processing with image recognition to assess damages.Precise risk assessment.LMM-generated personalized policy recommendations based on user data.
4. Utilities: Predictive models for infrastructure maintenance.Forecast energy demands accurately.Improve customer interaction with AI-based virtual support for queries and problem-solving.
5. Airlines: Deploy virtual assistants for efficient customer service and offer personalized travel solutions.Image recognition for regular maintenance and safety checks.
6. Retail: Enhance the shopping experience with personalized recommendations.Manage inventory through advanced image recognition.Analyze market trends using AI to forecast demand.
7. Mining:Implement LMM for predictive equipment maintenance, monitor safety conditions using image analysis.Optimize resource extraction and processing through data-driven insights.
8. Healthcare: Assist doctors in diagnosing diseases with medical imaging analysis and helping doctors to interpret medical images and reports for quicker diagnoses.Assit doctors to develop personalized treatment plans using patient data.Provide virtual health assistance for routine inquiries and patient monitoring.
9. Public Sector: Streamline public service delivery with automated systems.Process governmental documents efficiently, inform policy decisions with data analysis.Engage citizens through LMM-powered platforms.
10. Entertainment:LMMs can generate and recommend content based on a mix of user preferences and behaviors.
11. Contact Centers:Provide more intuitive and human-like interactions, understanding both verbal and non-verbal cues.
12. Education:LMMs can create immersive learning experiences by combining text, images, and audio.
13. Geography:LMM shows strong performance in geo-localization in urban areas and text extraction from historical maps but struggles with natural landscapes and lacks precision in areas without distinct artificial objects.
14. Environmental Science:In air quality evaluation, LMM can estimate Air Quality Index (AQI) categories but lacks the ability to predict precise AQI values.
15. Agriculture:LMM are effective in identifying cropland types and nutrient deficiencies in crops but faces challenges in complex surroundings.
16. Urban Planning:LMM demonstrates an excellent understanding of urban planning theories and practices and shows proficiency in urban street design but struggles with balancing constraints and proposed solutions.

Takeaways

The emergence of LMMs marks a significant leap in AI's journey towards more human-like understanding and interaction capabilities.

These models are not just academic feats; they are practical tools that can transform how we live and work, making technology more intuitive and aligned with our natural modes of communication.

ANNEX: Delving deeper into the models

IMAGEBIND

This model is designed by Meta to learn a joint embedding across six different modalities: images, text, audio, depth, thermal, and IMU data. It uses image-paired data to bind these modalities together, leveraging large-scale vision-language models and extending their zero-shot capabilities to new modalities.

The model demonstrates strong emergent capabilities in various tasks, including cross-modal retrieval, composing modalities with arithmetic, and more, setting a new standard in emergent zero-shot recognition tasks across modalities.

Strengths:

• Zero-Shot Capabilities: Extends large-scale vision-language models' zero-shot capabilities to new modalities using their natural pairing with images.
• Emergent Applications: Enables emergent applications like cross-modal retrieval, composing modalities with arithmetic, cross-modal detection, and generation.
• Performance: Sets a new state-of-the-art in emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models.
• Few-Shot Recognition: Demonstrates strong performance, surpassing previous works.
• Flexibility and Simplicity: Conceptually straightforward and can be implemented in various ways, making it adaptable and easy to adopt.

Limitations:

• Lack of Specific Downstream Task Training: Embeddings are trained without a focus on specific tasks, which can result in lagging behind specialist models in performance.
• Prototype Stage: Currently a research prototype, not immediately applicable for real-world applications.
• Potential for Improvement: Could benefit from incorporating other alignment data and further research into adapting embeddings for specific tasks.

Use Cases

IMAGEBIND's joint embedding approach is suitable for a variety of multimodal tasks, including:

• Cross-Modal Retrieval: Retrieving relevant data across different modalities.
• Composing Modalities with Arithmetic: Creating new modalities by combining existing ones.
• Cross-Modal Detection and Generation: Identifying elements in one modality based on data from another, and generating new content.
• Evaluating Vision Models: Assessing pretrained vision models for non-vision tasks.
• Upgrading Existing Models: Enhancing models like Detic and DALLE-2 to use audio data.

Achievements

IMAGEBIND has achieved several significant milestones:

• State-of-the-Art Performance: Set new benchmarks in emergent zero-shot recognition tasks across various modalities.
• Outperforming Supervised Models: Demonstrated superior performance compared to supervised models in emergent zero-shot classification and retrieval tasks.
• Versatility in Modalities: Effective across diverse modalities, including non-visual ones like audio and IMU.

Additional Notes

• The model uses a Transformer architecture for encoding modalities and optimizes embeddings using an InfoNCE loss.
• It leverages large-scale web datasets for image-text pairings and utilizes natural pairings of other modalities with images.
• IMAGEBIND's approach is distinguished by its emergent zero-shot classification, where it can classify or retrieve data in a modality without direct training on that modality, using image-paired data instead.

The effectiveness of IMAGEBIND in handling multiple modalities and extending zero-shot capabilities to new domains marks a significant advancement in the field of multimodal machine learning.

GPT-4V(ision)

GPT-4 with Vision, often referred to as GPT-4V, represents a significant advancement in the field of artificial intelligence, developed by OpenAI. This enhanced version of the GPT-4 model integrates the traditional text-based input system with the ability to process and understand images. This dual-modality capability marks a substantial leap from previous AI models that were solely text-based, broadening the potential applications and functionalities of GPT-4.

GPT-4 with Vision allows for a more comprehensive understanding and interaction with content, combining visual and textual information.

GPT-4V extends the capabilities of large language models (LLMs) by integrating multi-sensory skills, particularly in visual understanding, to achieve stronger generic intelligence. It is designed to process a mix of different input modalities, including images, texts, and visual pointers.

Strengths:

• Multimodal Input Processing: Demonstrates an unprecedented ability to understand and process an arbitrary mix of input images, sub-images, texts, scene texts, and visual pointers.
• Wide-Ranging Capabilities: Exhibits impressive capabilities across a variety of domains and tasks, including open-world visual understanding, multimodal knowledge, document reasoning, coding, temporal reasoning, abstract reasoning, and emotion understanding.
• Innovative Prompting Methods: Strong in understanding pixel space edits and capable of visual referring prompting, which allows for nuanced instruction and example demonstrations.
• Zero-shot Recommendation Abilities: GPT-4V has remarkable abilities to provide recommendations across diverse domains without prior specific training, thanks to its robust visual-text comprehension capabilities and extensive general knowledge.
• Coherent Recommendations: It can accurately identify contents within images and offer relevant recommendations, including identifying specific periods and styles for artworks and recognizing movie titles and genres from movie posters.
• Image Recognition and Understanding: show good performance in basic image recognition tasks.
• Text Recognition in Images: excel in extracting and recognizing text from images.
• Image Inference Abilities: demonstrate common-sense understanding in image reasoning.
• Multilingual Capabilities: effectively complete multilingual tasks, showcasing good recognition, understanding, and output capabilities in multiple languages.

Limitations:

• Tendency for Similar Responses: GPT-4V has a tendency to provide similar responses when given similar inputs.
• Manual Evaluation Process: Due to the unavailability of GPT-4V's API, the evaluation process for the case study was manual, which may introduce limitations in scalability and diversity of test samples.
• Performance Below Human Level: GPT-4V don′t achieved robust abstraction abilities at humanlike levels.
• Inconsistent Translation of Visual Grids to Text: GPT-4V struggled with consistently translating visual grids into text representations.
• Substantially Worse Multimodal Performance: GPT-4V performed substantially worse than the text-only version of GPT-4 on minimal tasks.
• Issues with Abstract Rule Identification: In some cases, GPT-4V accurately described the output grid but identified an incorrect abstract rule, or vice versa.

Use Cases

GPT-4V's use cases cover a broad range of vision and vision-language scenarios, including:

• Image description and recognition in various domains.
• Dense visual understanding and multimodal knowledge.
• Scene text and document reasoning.
• Temporal motion and video understanding.
• Abstract visual understanding and reasoning.
• Emotion and sentiment understanding.

Achievements

• Generalist Multimodal System: GPT-4V is recognized as a powerful multimodal generalist system due to its ability to handle diverse multimodal inputs and tasks.
• Human-Computer Interaction Innovations: Its capability to understand visual markers drawn on input images has led to the development of new human-computer interaction methods like visual referring prompting.
• Inspiring Future Research: The model's capabilities are expected to inspire future research in multimodal task formulation and the development of advanced LMM-based intelligent systems.
• Demonstrated Diversity in Recommendations: GPT-4V is shown to be capable of providing diverse recommendations, from similar artists to different art forms, enhancing the user experience in various recommendation scenarios.
• Increased Performance with Detailed Prompts: Using a more informative one-shot prompting method resulted in higher accuracy for GPT-4 compared to previous simpler methods.

Gemini

Gemini is an MLLM developed by Google, designed for multimodal integration, combining language and visual understanding capabilities.

Gemini 1.0 comes in three sizes: Ultra for highly-complex tasks, Pro for enhanced performance and scalability, and Nano for on-device applications.

The Gemini models are natively multimodal, trained jointly across text, image, audio, and video, setting new benchmarks in state-of-the-art performance across a broad range of domains.

Gemini is designed to excel in a wide range of tasks encompassing language, coding, reasoning, and multimodal tasks. Its focus is on leveraging its cross-modal reasoning and language understanding capabilities for diverse applications.

Strengths

• Multimodal Capabilities: In multimodal tasks, Gemini Pro Vision shows proficiency, particularly in temporal-related questions, despite lagging behind GPT-4V in overall performance.
• Image Recognition and Understanding: show good performance in basic image recognition tasks.
• Text Recognition in Images: excel in extracting and recognizing text from images.
• Image Inference Abilities: demonstrate common-sense understanding in image reasoning.
• Multilingual Capabilities: effectively complete multilingual tasks, showcasing good recognition, understanding, and output capabilities in multiple languages.
• Detailed and Expansive Answers: Excels in providing detailed and expansive answers, accompanied by relevant imagery and links.
• High Generalization Capabilities: Boasts high generalization capabilities and a potential reshaping of the multimodal AI landscape.

• State-of-the-Art Performance: Gemini Ultra, the most capable model in the family, advances the state of the art in 30 of 32 benchmarks, notably achieving human-expert performance on the MMLU exam benchmark and improving performance in all 20 examined multimodal benchmarks.
• Wide Range of Capabilities: The Gemini family demonstrates proficiency in "Factuality," "Long-Context," "Math/Science," "Reasoning," and "Multilingual" tasks, covering a holistic range of more than 50 benchmarks across these capabilities.
• Performance: Gemini Pro’s performance is comparable to GPT-3.5 Turbo and marginally better across several language datasets, though it lags behind GPT-4 Turbo. About 65.8% of Gemini Pro's reasoning processes are evaluated as logically sound and contextually relevant, showcasing its potential for application across various domains.

Limitations

• Challenges in Specific Domains: While the paper doesn't explicitly mention limitations, such advanced models may face challenges in maintaining high performance across all tasks and modalities, especially when compared to models specifically tailored for single-domain tasks.

• Challenges in Specific Areas: Gemini Pro faces significant challenges in temporal and social commonsense reasoning, as well as in emotion recognition in images. It often misunderstands contextual information, indicating areas for further development.
• Error Analysis: Common error types include context misinterpretation, logical errors, ambiguity, overgeneralization, and knowledge errors. These findings suggest areas where current LLMs and MLLMs can improve, especially in complex or nuanced scenarios.
• Gemini's Single-Image Input Mode: Falls short in tasks requiring understanding of temporal sequence or multiple images.
• Need for Prompt Adjustments: Gemini may require prompt adjustments to align with its architecture.

Use Cases and Achievements

• Evaluation Across Diverse Domains: Gemini Pro was tested on 12 commonsense reasoning datasets covering general, physical, social, and temporal reasoning, both in language-based and multimodal contexts. The model demonstrated robust capabilities across these diverse domains.
• Comparative Analysis: Gemini Pro was evaluated alongside other LLMs like Llama2-70b, GPT-3.5 Turbo, and GPT-4 Turbo in language-based tasks, and alongside GPT-4V in multimodal tasks. This comparative analysis provided insights into Gemini's performance relative to other leading models in commonsense reasoning.
• Diverse Applications: Gemini models are suitable for complex reasoning tasks, enhanced performance applications at scale, and memory-constrained on-device applications.
• Benchmark Achievements: Gemini Ultra notably excels in the MMMU benchmark, covering multiple disciplines that require college-level knowledge and complex reasoning. It outperforms previous state-of-the-art models in various disciplines within this benchmark.

In summary, the Gemini model family represents a significant advancement in multimodal AI, offering a range of models suitable for a variety of applications, from complex reasoning tasks to on-device deployment. The models' ability to handle tasks across different modalities and their state-of-the-art performance in various benchmarks highlight their potential for wide-ranging real-world applications.

Overall, the Gemini model exhibits strong potential in commonsense reasoning across various domains, with specific areas identified for improvement, particularly in understanding the interplay between visual cues and commonsense reasoning.

Comparing Gemini vs GPT-4V(ision)

Here we compare both models across key dimensions such as Vision-Language Capability, Interaction with Humans, Temporal Understanding, and assessments in both Intelligence and Emotional Quotients. It delves into the distinct visual comprehension abilities of each model, evaluating their performance in various industrial application scenarios.

Comparative Performance Analysis

• Image Recognition and Understanding: Comparable performance by both models.
• Text Recognition and Understanding in Images: Both excel, but Gemini better in reading table information.
• Image Inference Abilities: Both excel in common-sense understanding, Gemini slightly lags behind GPT-4V in IQ tests.
• Textual Inference in Images: Gemini shows lower performance in complex reasoning tasks.
• Integrated Image and Text Understanding: Gemini falls behind due to its single-image input limitation.
• Object Localization: Comparable performance, with Gemini slightly less adept at abstract image localization.

• Temporal Video Understanding: Gemini’s single-image input mode falls short in sequence comprehension.
• Multilingual Capabilities: Both models exhibit good multilingual recognition and understanding.
• Integrated Image and Text Understanding: Gemini's inability to input multiple images at once limits its performance in complex text and image tasks.

• Temporal Video Understanding: Gemini’s single-image input mode falls short compared to GPT-4V in comprehending sequences and video content.Precision and Succinctness: GPT-4V distinguishes itself with precision and succinctness in responses.

• Detailed and Expansive Answers: Gemini excels in providing detailed and expansive answers, accompanied by relevant imagery and links.
• Comprehensive Multimodal Understanding: Both models demonstrate a high level of multimodal understanding and reasoning abilities across multiple aspects.
• Challenges in Fine-Grained Recognition and Counting: GPT-4V faces limitations in tasks requiring fine-grained recognition and precise counting, especially in complex scenarios and with occlusions. The model's performance varies across problem domains and image complexities, indicating a need for more specialized domain training.

DocLLM

DocLLM is distinguished from existing multimodal LLMs by its focus on bounding box information, which is used to incorporate the spatial layout structure of documents. This model avoids the use of expensive image encoders and is designed to handle the complexities of enterprise documents like forms, invoices, and receipts, which feature rich semantics at the intersection of textual and spatial modalities.

Strengths

• Efficiency and Modality Integration: DocLLM excels in various document intelligence tasks by integrating bounding box coordinates of text tokens (obtained via OCR) without relying on vision encoder components. This approach results in a smaller increase in model size and reduced processing times.
• Innovative Training Approach: The model employs a disentangled spatial attention mechanism and an infilling pre-training objective tailored to handle irregular layouts and heterogeneous content of visual documents.
• Performance: DocLLM outperforms state-of-the-art LLMs in 14 out of 16 datasets across multiple tasks and shows good generalization to previously unseen datasets.

Limitations

• Specialization in Layout-Intensive Tasks: While DocLLM shows superior performance in layout-intensive tasks and outperforms most multimodal language models, it underperforms compared to GPT-4 in tasks involving complex reasoning and abstraction, such as visual question answering (VQA).

Use Cases and Achievements

• Document Intelligence Tasks: The model is fine-tuned on a large-scale instruction dataset covering four core tasks: visual question answering, natural language inference, key information extraction, and document classification.
• Comparative Analysis and Results: DocLLM demonstrates superior performance in comparison to other models like Llama2 and mPLUG-DocOwl in specific settings. It excels particularly in key information extraction and document classification tasks, indicating its effectiveness in processing and understanding visually rich documents.

In summary, DocLLM represents a significant advancement in the field of multimodal document understanding, offering an efficient, layout-aware approach to processing and interpreting complex visual documents.

LLaVA-Plus

LLaVA-Plus is designed as a multimodal assistant that can systematically expand the capabilities of large multimodal models (LMMs). It integrates a range of pre-trained vision and vision-language models to respond to multimodal inputs from users, executing these tools in real-time to accomplish a variety of tasks.

Strengths:

• Outperforms its predecessor, LLaVA, in existing capabilities.
• Exhibits new capabilities by actively engaging images in human-AI interaction sessions.
• Versatile, with a skill repository for a broad range of tasks.

Limitations:

• Experiences limitations due to hallucinations and conflicts in tool use in practice.

Use Cases

• External Knowledge Retrieval: Utilizes CLIP search API to access knowledge beyond pre-trained model weights.
• Image Generation and Editing: Employs Stable Diffusion and Instruct-Pix2Pix for these tasks.
• Visual Prompts: Supports interaction with user-drawn points, sketches, and boxes.
• Skill Composition: Handles complex tasks requiring combinations of different skills like segmentation, tagging, and captioning.

Achievements

• Achieved new state-of-the-art (SoTA) results on VisiT-Bench with a diverse set of real-life tasks.
• Significantly extends the capabilities of LMMs.
• Developed a new pipeline for creating vision-language instruction-following data for human-AI interaction sessions.

Training:

• LLaVA-Plus combines curated tool use instruction data with the LLaVA-158K dataset.
• It employs a unified prediction format and is built in two settings: one using all tools as external knowledge and another using tools 'on the fly'.

Serving:

• It is served using the FastChat system, which comprises web servers, model workers hosting the LMM and multiple tools, and a controller coordinating these components.
• The 7B LLaVA-Plus and all tools are operable on an 80G GPU.

UNIFIED-IO 2

UNIFIED-IO 2 is a large multimodal model (LMM) with 7 billion parameters. It encodes and produces various modalities like text, image, audio, video, and interleaved sequences. The model is trained from scratch on a diverse multimodal pre-training corpus using a multimodal mixture of denoisers objective, and further fine-tuned on an ensemble of 120 datasets with prompts and augmentations.

Focus

• Multimodal Integration: The focus is on integrating various modalities into a shared semantic space, processing them with a single encoder-decoder transformer model. This approach enables the model to handle a wide range of tasks across different modalities.

Strengths

• Versatility and Performance: UNIFIED-IO 2 demonstrates state-of-the-art performance on the GRIT benchmark and excels in over 35 different datasets. It performs well in vision and language tasks, matching or outperforming other VLMs. Its capabilities extend to image generation, where it surpasses models leveraging pre-trained diffusion models, especially in terms of faithfulness. It also shows effectiveness in video, natural language, audio, and embodied AI tasks, indicating a broad range of capabilities.

Limitations

• Challenges in Multimodal Learning: While the paper does not explicitly detail specific limitations of UNIFIED-IO 2, the general challenges in multimodal learning include managing the complexity of integrating different modalities, ensuring effective learning across all modalities, and avoiding biases or errors due to the diverse nature of the data sources.

Use Cases and Achievements

• Broad Scope of Tasks: UNIFIED-IO 2 covers a wide range of tasks and outputs, such as keypoint estimation, surface normal estimation, vision and language tasks, image generation, video and audio understanding, and embodied AI tasks. Its ability to learn from scratch aligns more naturally with how humans learn modalities simultaneously, enhancing its potential for diverse applications.

In summary, UNIFIED-IO 2 represents a significant advancement in multimodal learning, providing a versatile and effective solution for integrating and generating multiple modalities, such as image, text, audio, and action, in a unified framework.

Flamingo

Flamingo is designed to rapidly adapt to a variety of image and video tasks using few-shot learning. It bridges powerful pretrained vision-only and language-only models and can handle sequences of arbitrarily interleaved visual and textual data.

Strengths:

• Capable of performing various multimodal tasks from only a few input/output examples.
• Efficiently accepts and processes arbitrarily interleaved visual data and text, generating text in an open-ended manner.
• Sets a new state of the art in few-shot learning on a wide array of multimodal language and image/video understanding tasks.

Limitations:

• Inherits weaknesses from the pretrained language models it's built on, such as occasional hallucinations, ungrounded guesses, and poor generalization to longer sequences than those in training.
• Classification performance lags behind state-of-the-art contrastive models that optimize directly for text-image retrieval.

Use Cases

• Flamingo is suitable for a range of tasks including visual question-answering, captioning tasks (describing scenes or events), and multiple-choice visual question-answering.

Achievements

• Outperforms models fine-tuned on significantly more task-specific data across numerous benchmarks.
• Demonstrates effective adaptation to various tasks using only a few examples.
• Achieves state-of-the-art results on six out of sixteen tasks studied, using only 32 task-specific examples.

Training:

• Flamingo's training leverages the MultiModal MassiveWeb (M3W) dataset, which contains interleaved text and image data extracted from approximately 43 million webpages. This training approach is critical for developing its few-shot capabilities.

Serving:

• While the document doesn't specify the exact serving mechanisms, it emphasizes the importance of training data mixture, visual conditioning, and the freezing of language model components to prevent catastrophic forgetting, which are crucial for its deployment and performance

References

• IMAGEBIND: One Embedding Space To Bind Them Allhttps://arxiv.org/pdf/2305.05665.pdf
• GPT-4V: Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studyhttps://arxiv.org/pdf/2311.04199.pdf
• Gemini: A Family of Highly Capable Multimodal Modelshttps://arxiv.org/pdf/2312.11805.pdf
• LLaVA-Plus: LEARNING TO USE TOOLS FOR CREATING MULTIMODAL AGENTShttps://arxiv.org/pdf/2311.05437.pdf
• DOCLLM: A LAYOUT-AWARE GENERATIVE LANGUAGE MODEL FOR MULTIMODAL DOCUMENT UNDERSTANDINGhttps://arxiv.org/pdf/2401.00908.pdf
• UNIFIED-IO 2:Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actionhttps://arxiv.org/pdf/2312.17172.pdf
• Flamingo:A Visual Language Model for Few-Shot Learninghttps://arxiv.org/pdf/2204.14198.pdf

• Gemini vs GPT-4VA Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Caseshttps://arxiv.org/pdf/2312.15011.pdf
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuninghttps://arxiv.org/pdf/2309.05444.pdf
• Towards Auto-Modeling of Formal Verification for NextG Protocols: A Multimodal cross- and self-attention Large Language Model Approachhttps://arxiv.org/pdf/2312.17353.pdf