Multimodal Prompting with Llama 3.2

Introduction to Multimodal Prompting

In the world of advanced AI, multimodal prompting is gaining prominence. This concept involves simultaneous processing and generating responses based on different types of inputs—such as text, images, or both. The ability to handle multimodal data expands the functionality of models beyond simple text generation, enabling them to respond intelligently to visual inputs or a combination of text and visuals.

In this blog, we’ll dive into implementing multimodal prompts using Llama 3.2 and explore a step-by-step approach with Python code to demonstrate how text and image inputs can be processed together.

Getting Started with Environment Setup

Before starting, we need to configure our environment. The load_env function initializes our environment variables.

from dotenv import load_dotenv, find_dotenv

def load_env():
    _ = load_dotenv(find_dotenv())
        

Implementing Llama 3.2 Multimodal Prompting

We define a function, llama3, which handles requests to the Llama model API. This function sends messages to the model, specifying options like the number of tokens, temperature, and stopping conditions.

import os
import json
import requests

def llama3(messages, model_size=11):
    model = f"meta-llama/Llama-3.2-{model_size}B-Vision-Instruct-Turbo"
    url = f"{os.getenv('DLAI_TOGETHER_API_BASE', 'https://api.together.xyz')}/v1/chat/completions"
    
    payload = {
        "model": model,
        "max_tokens": 4096,
        "temperature": 0.0,
        "stop": ["<|eot_id|>", "<|eom_id|>"],
        "messages": messages
    }
    
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.getenv('Api_Key')}"
    }
    
    res = json.loads(requests.request("POST", url, headers=headers, data=json.dumps(payload)).content)

    if 'error' in res:
        raise Exception(res['error'])
    
    return res['choices'][0]['message']['content']
        

This method accepts a list messages containing text or image-based prompts, sends a POST request to the model API, and returns the AI's response.

Displaying Images with the disp_image Function

Next, let’s add functionality for displaying images directly in our code. The disp_image function takes a URL or a local file path and displays the image using matplotlib.

from PIL import Image
from io import BytesIO
import matplotlib.pyplot as plt

def disp_image(address):
    if address.startswith("https://") or address.startswith("https://"):
        response = requests.get(address)
        img = Image.open(BytesIO(response.content))
    else:
        img = Image.open(address)
        
    plt.imshow(img)
    plt.axis('off')
    plt.show()
        

Text Input-Only Prompting

Let’s begin with a simple text-only prompt where the model is asked a question:

# Example prompt
messages = [
  {"role": "user", "content": "How far is the moon?"}
]

# Invoking the model
response = llama3(messages, 90)
print(response)        

Image-Based Prompting

To take it a step further, let’s introduce an image as part of the input. Below is a simple example of describing an image using a URL.

# Image URL
image_url = "https://raw.githubusercontent.com/image.jpeg"

# Create prompt with text and image
messages = [
  {"role": "user", 
   "content": [
      {"type": "text", "text": "Describe the image "}, 
      {"type": "image_url", "image_url": {"url": image_url}}
   ]
  }
]

# Display the image and get the model’s response
disp_image(image_url)
result = llama3(messages, 90)
print(result)        

This example shows how the model can analyze a given image URL and return a text description.

Follow-up Questions About an Image

Multimodal prompts can also handle follow-up queries based on an image. Let’s look at an example where the model first describes the image, and then follows up with a question.

messages = [
  {"role": "user", 
   "content": [
      {"type": "text", "text": "Describe the image"}, 
      {"type": "image_url", "image_url": {"url": image_url}}]
  },
  {"role": "assistant", "content": result},  
  {"role": "user", "content": "How many of them are having hat?"}
]

# Invoke the model with a follow-up question
result = llama3(messages, 90)
print(result)        

This setup first asks the model to describe the image, then follows up with a specific question asking about details in the image. This example shows how multimodal prompts can allow for iterative and context-based querying.

Conclusion

Multimodal prompting is an essential tool in the AI toolkit that allows models to work with both text and images, enhancing their flexibility. Whether you’re analyzing documents, and images, or handling complex queries, multimodal models like Llama 3.2 can handle these challenges efficiently.

Through this blog, we’ve covered how to set up and invoke multimodal prompts, from simple text queries to more complex image-based interactions. With a bit of practice, you can leverage these powerful tools for your projects.

https://colab.research.google.com/drive/1e263LHXUGXZcB-K9R8ppcs4xjMqlXUlG?usp=sharing