Building a Local AI Server

Introduction

• Developed a local AI server named Terry.
• Includes a GUI with a chat interface and multiple models.
• Integrated with Obsidian notes.
• Built for personal use and for his daughters.

Hardware and Setup

Hardware Specifications

• Case: Leon Lee Zero 11 Dynamic EVO XL
• Motherboard: ASUS X670E Creator Pro Art
• CPU: AMD Ryzen 9 7950X (4.2GHz, 16 cores)
• Memory: 128GB G.Ki Trit D5 Neo DDR5 6000
• GPU: 2 MSI 4090s (liquid-cooled, 24GB each)
• Storage: 2 Samsung 990 Pros (2TB each)
• Power Supply: Corsair AX 1600I (1600W)

Installation & Initial Setup

• Troubleshooting: Ubuntu installation failed; switched to Pop!_OS by System76.
• Required components to build your own local AI: Only need a computer (Windows, Mac, or Linux)._

Installing and Running AI Models

Step 1: Installing Llama

1. Download Llama from alama.ai
2. Use the Linux version even on Windows via WSL:
   • Open Terminal
   • Install WSL: wsl --install
   • Set up username and password
   • Update packages: sudo apt update && sudo apt upgrade -y
   • Install Llama: a single curl command
   • Ensure GPU is recognized for better performance (install Nvidia Cuda drivers if needed)

Step 2: Adding AI Model to Llama

• Pull down the desired model (e.g., Llama 2): alama pull llama 2
• Run and test the model: alama run llama 2
• Performance monitoring using Nvidia: watch -n 0.5 nvidia-smi

Step 3: Web UI Setup (Open Web UI)

1. Install Docker:
   • Update and get Docker’s GPG key.
   • Install Docker with a single command.
2. Run Open Web UI with Docker:
   • Run Docker container for Open Web UI: sudo docker run ... command.
   • Access through localhost:8080 in the web browser.

Exploring Web UI

1. Sign up and log in (first account is the admin).
2. Chat with AI models, add new models, and manage responses.
3. Admin Settings: User management, whitelist models, and AI model files.

Custom Model Files

• Example: Create model file for daughter Chloe to restrict cheating (named model Deborah).
• Input custom prompts to guide and restrict AI responses.

Advanced Features

Stable Diffusion Setup (Image Generation)

1. Prerequisites: Install Python and necessary packages.
2. Install Stable Diffusion using Automatic 1111 UI.
   • Download and make a directory.
   • Run installation script.
3. Integrate Stable Diffusion with Open Web UI.
   • Use localhost:7860 for automatic 1111.
4. Generate images from text prompts.

Obsidian Integration

1. Use Obsidian for notes and integrate with local AI:
   • Install BMO Chatbot plugin.
   • Connect to Llama models.
   • Use AI to enhance note-taking and referencing.

Conclusion

• Running local AI provides privacy and control.
• Explore more applications and integrations.
• Engage with community for feedback and more projects.