Secure your private AI setup

Running AI on your own hardware keeps client data off third-party clouds, but only if the box itself is locked down. Here's a practical checklist for a small business.

1. Lock down admin accounts with a password manager

Your AI box usually has several admin logins: the OS account, the router, Open WebUI or AnythingLLM admin, Docker/Portainer, NAS and backup targets. Give each one a long, unique, randomly generated password and store them in a shared business password manager, not a spreadsheet. Turn on two-factor authentication anywhere it's offered, especially on the web UI admin account.

Keeper Security is a business password manager with shared vaults and 2FA. Bitwarden (open source, can be self-hosted) is a good alternative.

2. Never expose the AI box directly to the internet

Don't port-forward Ollama (port 11434) or your web UI on your router. Ollama's API has no authentication by default, and exposed instances are routinely found by scanners. Keep services bound to your LAN, and for remote access use a private network instead:

• Tailscale (or WireGuard) creates an encrypted private network between your devices. It's usually the simplest way to reach your AI box from home or on the road, and it has a free tier for small teams.
• A commercial VPN such as NordVPN or PureVPN protects staff traffic on public Wi-Fi. Note that it does not give remote access to your office box. For that, use Tailscale/WireGuard.

3. Back up models, configs and documents (3-2-1)

Keep 3 copies of important data on 2 different types of media, with 1 copy offsite. For an AI box that means your document library, vector database, chat history (if kept), Docker compose files and config. Models can be re-downloaded, but note which ones and versions you use.

An external drive or NAS such as Seagate storage works well for the local copy; encrypt offsite/cloud copies. Test a restore at least once.

4. Keep everything updated

• Enable automatic security updates for the OS (e.g. unattended-upgrades on Ubuntu, or Windows Update).
• Update Ollama, Open WebUI and other containers regularly (e.g. docker compose pull && docker compose up -d), and read release notes for security fixes.
• Update router and NAS firmware.

5. Protect the endpoints too

Staff laptops that connect to the AI box need up-to-date antivirus or endpoint protection, such as Windows Defender or Webroot, plus full-disk encryption (BitLocker / FileVault).

6. Limit who sees what

• Create separate user accounts in your web UI; don't share one login.
• Disable open sign-up after creating accounts.
• For very sensitive data, encrypt the disk and keep the box in a locked room.

Some links on this page may be affiliate links; we may earn a commission at no extra cost to you. Some links may be affiliate links; we may earn a commission at no extra cost to you. This never changes what we list.