Exploring Google Gemini 1.5 Pro

Contributed by : Adil Shahzad , 拉扎阿萨德

Gemini Labs: https://github.com/GDGCloudLahore/Build-With-AI-Labs-Google-Gemini

Google Gemini 1.5 Pro

Google has unveiled Gemini 1.5 Pro, a highly efficient, multimodal mixture of experts model. This advanced AI system is designed to handle tasks that involve recalling and reasoning through extensive content. Capable of processing long documents that may contain millions of tokens, including extensive video and audio files, Gemini 1.5 Pro elevates the performance standards in applications such as QA for lengthy documents, videos, and context-dependent automatic speech recognition (ASR). It not only meets but also exceeds the capabilities of Gemini 1.0 Ultra across established benchmarks, achieving almost flawless retrieval rates (>99%) for up to 10 million tokens, marking notable progress over other extensive-context language models.

Additionally, Google is introducing a trailblazing experimental model with a 1 million token context window, soon to be accessible for testing in Google AI Studio. For perspective, the largest context window previously available in any language model was 200k tokens. This expansion to a 1 million token window with Gemini 1.5 Pro is set to facilitate a variety of new applications, including question and answer sessions over large PDFs, code repositories, and comprehensive videos within Google AI Studio.

Architecture

Gemini 1.5 Pro is a sparse mixture of experts (MoE) transformer-based model, built upon the multimodal capabilities of Gemini 1.0. The advantage of an MoE architecture is that it allows the total number of model parameters to increase while maintaining a constant number of activated parameters. You can find more architectural details in this PDF file, with the entire article's information also sourced from the document: https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf.

It has been reported that Gemini 1.5 Pro requires significantly less compute for training, is more efficient to deploy, and features architectural modifications that enhance its understanding of long contexts (up to 10 million tokens). The model is pretrained on data encompassing various modalities and fine-tuned with multimodal data, with additional adjustments based on human preference data.

Results:

Long-context?performance:

• Near-perfect recall (>99%) on needle-in-a-haystack tasks across modalities, up to millions of tokens.
• Outperforms previous models on long-document QA and long-video QA benchmarks.
• Achieves state-of-the-art performance on long-context ASR.

Core?capabilities:

• Matches or surpasses Gemini 1.0 Ultra on various tasks despite being more efficient.
• Shows improvement in math, science, reasoning, coding, multilinguality, and instruction following.
• Competitive performance on image and video understanding benchmarks.
• Excels at speech recognition and translation tasks compared to specialized models.

Gemini 1.5 Pro surpasses Gemini 1.0 Pro on the majority of benchmarks with significant performance in math, science, reasoning, Multilinguality, video understanding, and code. Below is the table summarizing the results of different Gemini models. Gemini 1.5 Pro also outperforms Gemini 1.0 Ultra on half of the benchmarks despite using significantly less training compute.

Capabilities

To check the capabilities of Gemini 1.5 Pro, we will try long document analysis, video analysis, video understanding, and code reasoning.

Video Understanding

Gemini 1.5 Pro is developed with multimodal capabilities from inception and showcases proficiency in video understanding. We conducted tests using several prompts based on a Devfest 2023 event by GDG Cloud Lahore

The initial question posed was, 'What is this video about?'. While straightforward, the response was satisfactory as it accurately summarized the video content.

The second prompt we gave to the model was to transcribe this video, but the model was unable to transcribe it due to its limitations. Then we provided the prompt to summarize this video, and the results were accurate.

Long Document Analysis

To demonstrate the abilities of Gemini 1.5 Pro to process and analyze documents, we start with a very basic question-answering task. The Gemini 1.5 Pro model in Google AI Studio supports up to 1 million tokens, allowing us to upload entire PDFs. The example below shows that a single PDF has been uploaded along with a simple prompt: Can you summarize this document?

Code Reasoning

With its capability for long-context reasoning, Gemini 1.5 Pro can answer questions about codebases. Utilizing Google AI Studio, which allows up to 1 million tokens, users can upload an entire codebase and engage the model with various coding questions or tasks. The technical report includes an example in which the model is provided with the entire JAX codebase (~746K tokens) and tasked with identifying the location of a crucial automatic differentiation method.

English to Kalamang Translation

Gemini 1.5 Pro can be supplied with a grammar manual (consisting of 500 pages of linguistic documentation, a dictionary, and about 400 parallel sentences) for Kalamang, a language spoken by fewer than 200 people worldwide. It translates English to Kalamang with the proficiency of someone studying the same materials. This demonstrates the in-context learning capabilities of Gemini 1.5 Pro, facilitated by its ability to handle long contexts.

Kalamang is a low-resource language spoken by fewer than 200 people in western New Guinea. This means there's very little data available for training traditional machine translation models.

Subscribe to GDG Cloud Lahore to get the latest updates on Google Cloud, Gemini technologies, workshops, and events.