Re: Deploying an LLM Model on my Own Server with Docker and Tailscale
A lot has changed in my Homelab since that last post about my AI/Ollama configuration. I figured it was time to update what it looks like today.
Technology Stack
The biggest change is that I have switched my main server from Windows 11 to Proxmox. This allows me way more flexibility when it comes to configuration and VM management. I deployed a new Debian VM running docker with PCIE Passthrough for my GPU. This was automatically deployed via a Proxmox template file I made by using a Bash script I developed.
The second main change I made was how the service was served. Originally I used Tailscale Serve which was great at the time but once I started to learn the Tailscale platform more, I learned that Serve should be used more for short term ephemeral private demos not long term deployments. That’s where Tailscale Docker sidecars come in. These sidecar containers run their own instance of the Tailscale service with its own machine name and IP. This way the service does not use the VM hosts IP but its own. This is helpful when you overlap ports, want a custom URL for your service or want more granular ACL rules on a per container level then just the host and all its containers put together.
| Technology | Product/Platform | Reason |
|---|---|---|
| Server/Virtual Machine Platform | Proxmox | Modern, Opensource OS and flexible for other projects down the road. |
| Operating System | debian-12-genericcloud-amd64.qcow2 | These images are the cloud native industry standard for any type of public cloud platform. Learn more here. |
| Containerization Platform | Docker Engine on Debian | Open Source, very popular and I wanted to expand my skills with the platform. |
| Large Language Model (LLM) | llama2 | Made by Meta and is lightweight (7b) for quick testing and deployments. |
| LLM Engine | Ollama | Free, open source and is a new tool I wanted to explore! |
| Web Server and Graphical User Interface (GUI) | Open Web UI | Open Source and has conveniently made packages for Ollama. |
| Virtual Private Network (VPN) | Tailscale | Free, mostly open source and helps streamline networking. |
| Networking Serving | Tailscale Docker Sidecar Proxy | Deploys a Tailscale sidecar container which allows you to have a separate Tailscale IP and URL for Open WebUI and Ollama. |
Configuration
More detailed notes will be listed in my Creating a Proxmox Automation Script page.
- Deploy Proxmox VM via my Bash script
- SSH into the VM and install Ollama and Open WebUI via my docker configs notes.
- Once Deployed Open the Tailscale side car URL on the Open WebUI
- Login and install your LLM.
- Done!
Overall I am really happy with this new current setup, its much more maintainable and repeatable with my automation scripts. I will be sure to update if there are any more infrastructure changes.
Resources