~50-user team AI stack on one RTX A6000 (vLLM + LiteLLM + Open WebUI)
An administrator guide for a self-hosted team platform: vLLM serves Gemma 4 31B on a single 48GB RTX A6000, LiteLLM handles keys and rate limits, Open WebUI is the chat/RAG front end, with PII redaction and monitoring.
11-50 peopleDocument search / chat with filesWriting assistantCoding$3k+Sensitivity: High (client/confidential)
Hardware
| Item | Approx. price (as of 2026-10-16) |
|---|---|
| On sale −6% MSI GeForce RTX 5060 Ti 16GB Ventus 3X OC (Newegg, related current offer) | $579.99 |
| Server with 1x NVIDIA RTX A6000 (48GB), 8+ cores (16+ recommended), 64GB RAM (128GB recommended), 200GB+ SSD | Not stated in source |
Source prices are as stated by the source. Retailer offers: prices checked 2026-10-16, may change. Some links may be affiliate links; we may earn a commission at no extra cost to you. This never changes what we list.
Software stack
- vLLM
- LiteLLM
- Open WebUI
- Ollama (CPU, embeddings)
- Presidio (PII)
- Prometheus/Grafana/Loki
- Tailscale or SSH tunnel
Models used
- google/gemma-4-31B-it-qat-w4a16-ct
- nomic-embed-text (embeddings)
Good at
- Guide states ~15-25 concurrent chat sessions and ~50 total users with queuing and rate limits
- vLLM is never exposed to the network; only Open WebUI and LiteLLM bind to localhost
- Per-key rate limits and usage tracking via LiteLLM
Weak at
- Many moving parts (gateway, monitoring, logging) to maintain
- IDE/agent traffic shares the same GPU and limits; users may see HTTP 429 errors under load
Rough total cost
Not stated in source.
Getting started
- Install Ubuntu 22.04/24.04, NVIDIA driver and Docker.
- Copy .env.example, set HF_TOKEN and LiteLLM master key.
- docker compose up the stack (vLLM, LiteLLM, Open WebUI, Ollama embeddings, monitoring).
- Create user accounts in Open WebUI and issue LiteLLM keys with rate limits.
- Give staff access over Tailscale or SSH tunnel.
Source & credit
SFotopoulou/self-hosted-ai - Team AI Stack administrator guide — GitHub
Summarized from the source; quotes are the source author's words. Check the original before buying.