What You Run (LLM) Locally Is Your Business, But… Are You Sure Your LLM Model Isn’t Messing with Your System? ????
More and more people are running LLM models (e.g., Ollama, LM Studio, Llama, Mistral, Open-WebUI) locally to avoid the cloud and maintain full control over their data. But do you really have control? Do you know what your model is doing beyond generating text?
Could it be scanning your network or snooping around your system? ??
??? Can a Model "Do Something" on Your System?
If you run it without isolation, it can (YEEEEESSSSS):
That’s why it’s crucial to ensure security - especially if you’re working on a machine containing personal or corporate data!
??? Best Practices for LM Studio and Ollama
?? LM Studio (see https://lmstudio.ai/docs/offline)
?? Ollama
?? Selected CVEs:
??? How to Secure Your Locally Running Model?
?? Isolation in Docker
docker network create \
--driver=bridge \
--subnet=192.168.1.0/24 \
isolated_net
docker run -d \
--network=isolated_net \
--ip=192.168.1.100 \
-p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
--restart always \
--dns 192.168.1.1 \
--dns-search local \
--security-opt no-new-privileges \
--cap-drop NET_RAW \
--cap-drop NET_ADMIN \
ghcr.io/open-webui/open-webui:main
?? Option: Docker Seccomp/AppArmor
docker run --security-opt seccomp=unconfined ghcr.io/open-webui/open-webui:main
docker run --security-opt apparmor=unconfined ghcr.io/open-webui/open-webui:main
docker run --cap-drop ALL --security-opt no-new-privileges ghcr.io/open-webui/open-webui:main
docker run --read-only --tmpfs /tmp ghcr.io/open-webui/open-webui:main
?? Blocking System Calls in Python Option: Monkey Patching (Disabling os.system)
Before launching the model, you can disable key functions:
领英推荐
import os
import subprocess
os.system = None
subprocess.call = None
subprocess.run = None
subprocess.Popen = None
This prevents executing system commands from Python.
?? Running in a Sandbox
?? Using SELinux (Linux)
If you are on a system that supports SELinux, you can set a policy that blocks access to /bin/sh, /usr/bin/python, etc.
sudo semanage fcontext -a -t sandbox_model_t "/path/to/model(/.*)?"
sudo restorecon -Rv /path/to/model
?? Limiting User Privileges
The simplest method: Run the model on a dedicated user account without sudo privileges.
sudo useradd -m -d /home/llm_user -s /usr/sbin/nologin llm_user
sudo chown -R llm_user:llm_user /path/to/model
sudo -u llm_user python3 run_model.py
This ensures the model cannot execute commands requiring admin privileges.
?? Monitoring Model Activity
?? Monitoring Network Traffic and System Logs
If you suspect that the model might be trying to execute commands, check:
strace -p $(pgrep -f model) # Tracing system calls
sudo auditctl -w /bin/sh -p x # Monitoring attempts to launch the shell
Conclusion
?? "Trust, but verify."
What you run locally is your business (thing) - but if you don’t implement isolation, you might unknowingly give your model access to your network and files. LM Studio and Ollama are great tools, but you should still monitor their behavior and limit their access to your system and the internet.
How do you secure your local LLM instances? Do you have your own best practices? Let us know in the comments! ????
Source:
https://genai.owasp.org/resource/genai-red-teaming-guide/ OWASP Top 10 For Large Language Model Applications & Generative AI OWASP? Foundation
iOS Engineer SDE III @ Expedia Group
1 个月Nice, is it useful to also include podman, and not only docker? You know, daemon-less arch, rootless containers, etc...
Senior Security Engineer @ Application Security, DevSecOps
1 个月Really cool article Sebastian Obara. As you have the deployment's best practices covered, it deserves a continuation in terms of preventing LLM attacks in such setup.