#46 - Locally run AI's
Joe Houghton
Husband, dad, educator, author, facilitator, speaker, consultant, coach, photographer - wearer of many hats! "People gardener".
One of the key issues for everyone who has been experimenting with the slew of AI tools we've been presented with over the past 18 months is the worry about privacy. With all these tools being cloud-based, whatever we type in as prompts, and upload as attachments, is uploaded to a server somewhere in the AI-verse.
Some of the AI companies are open about the fact that they are using all this data to add to their training data, and the ones that claim to be more ethical? Well, we really don't know if their supposed privacy statements are true, do we? And where the billions of dollars that AI is generating meets ethics, I suspect that some decision making follows the money - especially if there's little oversight or clear visibility possible about where the data goes.
Now I'm no conspiracy nut, but wouldn't it be great if we could run these AI chatbots without all that worry? That's what we're looking at today, so read on to find out more...
Running an AI locally
Tools now exist to download AI models onto your computer, and then have them run locally, with no internet connection or uploading of data to the cloud required.
AnythingLLM
The screenshot below is of AnythingLLM (https://useanything.com/ ), and this is one app that lets you set up fully localised AI. And the local install version is free, so you can try it out and run it against your own documents on your own computer.
It comes a built-in AI (the Ollama LLM model) that lets you load data and then use this as a knowledge base against which you can run prompts - just like using ChatGPT or any of the other AI bots you are already familiar with.
The interface is a little clunky - it's clearly aimed at techies, but there are a number of getting started videos to help you get to grips with how it works.
Here's a shot of one of the setup screens from my install:
You can set up different functions that the bot can perform - most of these are capable of running fully locally, but obviously if you want to turn on the capability to browse the web for instance, then it's going to also be using an internet connection for that.
The dropdown at the top of the Agent Configuration tab also allows you to download LLM's (Large Language Models) from all of the major providers - and there are more in this list than in the screenshot below.
Some of these LLM's require you to set up what's called an API key to send commands to them. These are paid models that charge you a very small amount each time a query is sent to the model - typically the more powerful LLMs need an API key. However, many of the less powerful (but still highly capable) LLM models can be downloaded for free and then used on your local data.
This is a powerful way to experiment with LLMs and see how prompts and different ways to configure the various LLM's could affect the responses you get.
Local AI's - a USP for business
USP - Unique Selling Proposition. For many years, businesses and individuals have differentiated themselves by their knowledge and their ability to connect dots others can't. The new AI's are fast making some of this capability commoditised, as they draw on the huge data sets that the LLMs have been trained on.
But many companies and individuals have local repositories of data - documents, research, email histories - and these have largely not yet been exposed as data sets to the public LLMs.
This has been and will continue to be a source of differentiation. And it's also what's behind a lot of the worry around using cloud based AI's and sharing this data with them.
So one of the key areas of interest and development for many businesses across 2024/25 will be in this area - leveraging their intellectual property assets in a way that allows the new AI capabilities to add value while at the same time protecting the wider dissemination of these knowledge bases.
The need for better interfaces
There are other tools out there such as LibreChat that also offer similar options to AnyThingLLM - local installs, choice of models to use etc.
The problem at the moment with most of these apps is that they are built by and (for now) primarily aimed at, techies. However, most people, faced with instructions like the ones below, are going to pass - and this is the Simple Quick Start Guide!
Specialised AI's curated in friendly interfaces
This is a big opportunity for creative startups - to create user friendly applications that allow normal, non-technical users to use a range of the AI tools now available. AI's are going to be available for all kinds of specialised tasks - the big ones we are mostly using at present are great all-rounders, but there will be more highly trained models that give better results in more focused areas of knowledge.
User interfaces that integrate these into user-friendly apps that are easy to install, learn and use effectively will become very popular - lots of money to be made in this kind of an emerging market!
I asked Perplexity AI "What are 5 of the most user friendly apps for running AI LLM models locally on your own computer?" and turned the answer into one of their new Pages - click here to see how it turned the answer into a full webpage with sources in seconds...
领英推荐
Watch this space!
OK, that's enough for today - see you next time!
EPALE - The European Adult Learning Network
Do you know about EPALE - the European Adult Learning Network? I'm one of the Irish Ambassadors for EPALE, and you can join over 100,000 educators across Europe in a free online community - it's a great way to get different points of view, participate in training from across the continent and stay up to date on educational thought. Create your free account at https://epale.ec.europa.eu/en/user/login
Affiliate Links (stuff I use and recommend)
Perplexity AI - Best search and my go-to AI now... Uses the latest top AI models from ChatGPT, Claude et al, innovates constantly - give it a try!
Unriddle AI is a research site that lets you upload docs and then interrogate them.
Check out Humata - it's another AI that let's you work on your own documents and interrogate them . https://www.humata.ai/?via=joe-houghton
My tool of choice for serious AI image creation is Leonardo. The user interface is easy and very powerful, enabling you to create just what you want in any style really quickly. https://app.leonardo.ai/?via=joe
Notion
This is my tool of choice now for collecting all the bits'n'pieces of information I squirrel away for talk, articles and presentations. I can then generate webpages in a snap and share them, and they update in real-time as I add new info to them! There's so much you can do in Notion - well worth a look:
Joe Houghton is an Assistant Professor at UCD Smurfit Graduate School of Business where he directs the MSc programmes in Project Management. After a career in IT in multinationals, Joe switched into a portfolio career of University teaching, management coaching and training.
He has authored 6 books to date including "Innovative teaching with AI: Creative approaches to enhancing learning in education ", and "Project Management made easy...: the ECCSR approach ". His latest book "Applying Artificial Intelligence to Close the Accessibility Gap: A practical handbook for educators & students! " is now available on Amazon! More on this in a future edition...
Contact Joe on email at [email protected] for any requests for training, seminars, workshops or keynote speaking.