The Evolving GenAI Landscape

There are a few generative AI tools that warrant our attention. These are designed to enable working with some LLM models locally on moderate hardware with or without a GPU. These options are fully contained and operational without sharing any prompt or response information. They only need an internet connection for downloading/updating application components and for downloading the model(s) to use. In some use cases, the local, contained nature of these projects may make them an attractive alternative to online 3rd party hosted platforms that require information sharing, or at least that you trust them to care for and protect your information to some degree.

GPT4All

The most use-friendly is GPT4All - https://gpt4all.io/index.html. This has installers for Windows, OSX and Ubuntu. It provides a list of compatible models with download buttons. Some are licensed to allow commercial use and some are not, but the license terms are clearly described for each. After installing one or more models, you have a straightforward UI for interacting with the selected model. In the configuration screen, there is a plug-in for working with local documents which is intended to allow you to ask questions about the contents of your documents. GPT4All can optionally run a local REST API service for programmatic access. There is also a pip-installable GPT4All library for writing your own apps that directly interact with compatible models.

The document feature seems prone to Information from the model training dataset leaking into the responses. Therefore, answers must be verified against the documents to ensure that the answer provided is applicable to your scope. But you can include a citation request in your prompt to help with this verification.

LocalGPT

LocalGPT is an open-source project (https://github.com/PromtEngineer/localGPT) that allows you to ‘converse with your documents without compromising your privacy’. It was inspired by another open source project called PrivateGPT. Compared to GPT4All, this project is less polished, but more powerful. It is compatible with a much wider selection of models, but you don’t get them served up in a menu. A certain amount of care is required to select a model that is suitable for your use case (including licensing) and your hardware. The portal at Huggingface.co is helpful for finding suitable model candidates and LocalGPT works well with this model hub and can automatically download them using the base name. This project includes 2 UIs and an API interface.

Oobabooga

Oobabooga (https://github.com/oobabooga and https://github.com/oobabooga/text-generation-webui) is a more technical swiss army knife project for working closely with the underlying models and the training and tuning processes. It works with an even wider array of model types and allows you to tweak the model configuration options. You clone the repository then run the installer for your platform. After the initial installation, it launches a local web service and the UI is accessible using your browser.

Mistral

There is a recent development in the availability of an exciting new open licensed model named Mistral. There is a lot of excitement because this model is outperforming other models of similar size or larger. The open licensing terms, performance, and relatively small footprint make this model well suited for open source projects and for running on modest hardware. A number of derived models are emerging based on additional tuning of this model for specific purposes – such as samantha-mistral. You can find these models on the portal at Huggingface.

This landscape is changing rapidly and I expect a plethora of local products and agents to start showing up any day now that run locally on a laptop/desktop workstation or even mobile device, that are designed to bring AI capabilities to your private information and data.