Use Llama 3.1 as Your Private?LLM

This article will guide you through setting up Llama 3.1 as a local large language model on your machine. We’ll also build a simple application that uses Llama 3.1 and Node.js to generate jokes based on user-provided topics.

Why Use a Private?LLM?

There are many open-source LLMs that we can run on our machines, but for this article, we will be focusing on Llama 3.1 by Meta. By keeping sensitive information secure within a controlled deployment environment, private LLMs ensure that data privacy is maintained. Additionally, these models allow for extensive customization, including the ability to fine-tune the model to specifically suit the unique needs of various sectors.

Especially for sectors like banking, where data privacy is paramount, the ability to tailor functionality makes private LLMs particularly valuable for applications that require precise control over data and customized performance.

Setting Up Llama 3.1?Locally

To set up Llama 3.1 on your machine. We’ll use Ollama, a tool designed to streamline the management of local LLMs. Begin by downloading and installing Ollama from its official website .

Llama 3.1 offers a range of models tailored for different needs:

? 8B: This multilingual model supports a long context length of 128K, making it suitable for tasks like long-form text summarization and multilingual dialogue systems.

? 70B: Enhanced for more complex applications, this model offers greater multilingual capabilities and advanced reasoning, ideal for coding assistance and detailed analytical tasks.

? 405B: As the most advanced option, the 405B rivals leading AI models in general knowledge and translation capabilities, designed for the most demanding AI tasks across various fields.

In this article, we will focus on setting up and using the Llama 3.1 8B model because it is easy to run on a machine with at least 8GB of memory.

Proceed with the following commands in your command line to get Llama 3.1 up and running:

1. The first command fetches the latest Llama 3.1 model files, specifically pulling the 8B version.

ollama pull llama3.1        

2. The second command runs the Llama 3.1 8B model locally on your machine.

ollama run llama3.1        

After executing these commands, verify that the Llama 3.1 8B model is functioning by asking it to generate a response from a simple prompt.

This is how you can interact with the Llama 3.1 8B model in your local terminal.

Now, Let’s build an AI Joke Machine using our?LLM

With Llama 3.1 set up and ready to go, we can start building the AI Joke Machine.

1. Setting Up the Ollama API Connection

This ollama.js file configures the Ollama API connection for interacting with the Llama 3.1 model, specifying the local server host address for model communications.

// ollama.js
import { Ollama } from "ollama";

export const ollama = new Ollama({
  host: "https://localhost:11434",
});        

2. Importing Dependencies and Setting Up Readline Interface:

Now, create the main file ai-joke-machine.js, which imports the readline and ollama modules. The readline module enables command-line interaction with the user, while Ollama manages the connection to the AI model.

import readline from "node:readline";
import { ollama } from "./ollama.js";

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});        

3. Defining the User Prompt and Messaging Functions:

const sentenceWithTopic = (topic) => `Tell me a joke about ${topic}`;

const sendMessage = async (topic) => {
  const response = await ollama.chat({
    model: "llama3.1",
    messages: [
      {
        role: "assistant",
        content: "You are a Joker! Only respond with a joke according to the topic given.",
      },
      {
        role: "user",
        content: sentenceWithTopic(topic),
      },
    ],
  });
  return response.message.content;
};        

where, sendMessage manages the interaction with the Llama 3.1 model via the Ollama API, fetching the joke.

4. Main Function to Initiate the Joke Generation Process:

The main joke function initiates the joke generation process by prompting the user to enter a topic, then uses the sendMessage function to fetch a joke from the AI based on that topic, and displays the joke.

const joke = async () => {
  rl.question("Enter a topic for a laugh: ", async (userInput) => {
    const response = await sendMessage(userInput);

    console.log(`\n\n`);
    console.log(`AI Joke Machine: ${response}`);
    console.log(`\n\n`);

    rl.close();
  });
};        

5. Running the Application

Let’s run the application by calling that joke function,

joke();        

AI Joke Machine?Demo:

Now, you can run the application and enter any topic to see a joke generated by your very own AI Joke Machine!

Conclusion

In this article, we explored the advantages of using a private LLM for enhanced privacy and control, then walked through the steps of installing and configuring the Ollama API to run the Llama 3.1 model locally. We applied this setup in creating the AI Joke Machine, an interactive Node.js application that generates jokes based on user inputs.?

This project not only illustrates the practical use of AI in engaging applications but also opens the door to further innovation and development with private language models.

If you found the article helpful, don’t forget to share the knowledge with more people! ??