How to use Mistral 8x7B on Google Cloud

Mistral ai released Mixtral8x7B LLM, a new generation of large language models built with Mixtures of Experts and the others performance optimizations that Mistral's team baked.

https://mistral.ai/news/mixtral-of-experts/

This model, given its size, is awesomely performing!

In this article we will discover how to use Mistral's Mixtral8x7B on Google Cloud

Step 1 Connect to your Google Cloud Account

Go to https://console.cloud.google.com/

Step 2 Navigate to Vertex AI -> Model Garden

Step 3 Search for Mistral

Step 4 Select Mixtral 8x7B

Here you have multiple choices

# Click-to-Deploy to get an API endpoint where you could use Mixtral

# Open NoteBook that allows you to go through a python code that will enable you to use Mixtral and gives you an idea of how to use it in Python

Step 5 Click-to-Deploy

Step 6 select the model, model name, endpoint name, region and Deploy?

Wait for the model to deploy, this could take several minutes (in my case region us-central1 it took less than 8 minutes)

Step7 Navigate to Vertex AI -> Deploy and Use -> Online Prediction?

If the model is not deployed yet, let it some time to finish deployment

You will see a check mark ? in the status column when your model is ready to get used

Step 8 Use your Mixtral model deployed on Google Cloud

Here you have multiple choices

# Use you Mixtral8x7B Vertex AI API endpoint from Cloud Shell (step 9)

# Use you Mixtral8x7B Vertex AI API endpoint from a python notebook (step 10)

Step 9 From Cloud Shell?

9.1 Open the Cloud Shell environment?

9.2 The Cloud Shell window appears, in your cloud shell type

type

$gcloud auth application-default login        

9.3 click the auth link that Cloud Shell invite you to browse

9.4 Get you auth code by clicking the Copy button

9.5 Paste the auth code in your Cloud Shell

9.6 Create a prompt.json file

{
  "instances":
  [
  		{
			"prompt": "<s> [INST] Could you help me with questions on traveling? [/INST] Of course!</s> [INST] What is the best season to travel in Thailand? [/INST]",
			"max_tokens": 50,
			"temperature": 1.0,
			"top_p": 1.0,
			"top_k": 10
		}
	]
}        

For example using the vi text editor

$vi prompt.json        

??In vi ?? type “i” to insert text in the file?

???Insert

{
  "instances":
  [
  		{
			"prompt": "<s> [INST] Could you help me with questions on traveling? [/INST] Of course!</s> [INST] What is the best season to travel in Thailand? [/INST]",
			"max_tokens": 50,
			"temperature": 1.0,
			"top_p": 1.0,
			"top_k": 10
		}
	]
}        

??Escape?

Then

?? wq!

You now have the prompt.json file created and saved in your current working directory

9.7 Create Environment Variables

In Cloud Shell declare the following variables:

ENDPOINT_ID=" … Your endpoint ID … "
PROJECT_ID="... Tour project ID … "
INPUT_DATA_FILE="prompt.json"        

9.8 Use your Mixtral model via the API endpoint

Launch the curl call to get the answer from your model?

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/endpoints/${ENDPOINT_ID}:predict \
-d "@${INPUT_DATA_FILE}        

Lets see the model output?

Prompt:[INST] Could you help me with questions on traveling? [/INST] Of course! [INST] What is the best season to travel in Thailand? [/INST]Output: The best season to travel in Thailand depends on the region you plan to visit. Generally, the cool season from November to February is considered the best time to travel throughout the country as the weather is pleasant and cool, with relatively low humidity"

Step 10 From a Python Notebook?

Prepare your Project ID, Endpoint ID and Region

you could refer to this colab Notebook: link to come

Install the Vertex AI libraries

import sys
if "google.colab" in sys.modules:
   ! pip3 install --upgrade google-cloud-aiplatform > /dev/null
   from google.colab import auth as google_auth


   google_auth.authenticate_user()
   # Install gdown for downloading example training images.
   ! pip3 install gdown > /dev/null


   # Restart the notebook kernel after installs.
   import IPython


   app = IPython.Application.instance()
   app.kernel.do_shutdown(True)

# Then Call your API endpoint
# Load an existing endpoint instance using the endpoint name
# You can set `ENDPOINT_ID = "1234567890123456789"` to load
#   an existing endpoint with the ID 1234567890123456789.
from google.cloud import aiplatform


ENDPOINT_ID = "5946275431245152256"
PROJECT_ID = "cloud-llm-preview2"
REGION = "us-central1"


AIP_ENDPOINT_ID = (
    f"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}"
)
endpoint = aiplatform.Endpoint(AIP_ENDPOINT_ID)


instances = [
   {
       "prompt": """<s> [INST] Could you help me with questions on traveling? [/INST] Of course!</s> [INST] What is the best season to travel in Thailand? [/INST]""",
       "max_tokens": 50,
       "temperature": 1.0,
       "top_p": 1.0,
       "top_k": 10,
   },
]
response = endpoint.predict(instances=instances)


for prediction in response.predictions:
   print(prediction)        

The output of this cell :?

Prompt:

[INST] Could you help me with questions on traveling? [/INST]

Of course!

[INST] What is the best season to travel in Thailand? [/INST]

Output:

The best season to travel in Thailand is generally considered to be the cool dry season, which lasts from November to February. During this time, temperatures are more comfortable and there is less rainfall, making it ideal for sightseeing, outdoor activities

Conclusion

Mixtral8x7B is a really interesting model to use given it's performances and it's size.

We saw how you could get a Mixtral8x7B Vertex AI API endpoint in your Google Cloud account and how to use it from Cloud Shell and in Python.

Hope you'll enjoy discover the capabilities of this fantastic gem Mistral team released directly from your secure enterprise environment!

#llm #genai #mistral #mixtral #gcp #vertexai