How to Run Local AI Models on Linux with Ollama: A Complete Step-by-Step Guide
Objective
This guide shows you how to install Ollama on Linux, download your first AI model, and run it entirely on your own machine with no cloud connection, no API fees, and no data leaving your computer. By the end, you will have a working local AI you can use for coding, writing, document analysis, and automation, completely private and completely free to run.
The Problem with Cloud AI Services
Every time you use ChatGPT, Claude, or any cloud-based AI tool, your input goes to a remote server. Your code, your documents, your questions, your private business information, all of it leaves your machine. You pay either with money (API fees that add up fast) or with your data (which trains their models). For anyone working with sensitive material or simply valuing privacy, that is a real problem.
Ollama solves this. It lets you run powerful AI models on your own Linux machine, locally, offline after the initial download, and at zero ongoing cost.
What Is Ollama
Ollama is a lightweight open-source tool that makes downloading, managing, and running large language models on your local machine as simple as a few terminal commands. Instead of configuring complex machine learning environments, setting up Python dependencies, or wrestling with model formats, Ollama handles all of that underneath. You just tell it which model you want and start talking to it.
All processing happens on your hardware. Your prompts never leave the machine. The model runs in memory and responds directly in your terminal or through a local API that your own applications can call.
How the Process Works
- Step 1: Install Ollama on your Linux system with a single command
- Step 2: Pull a model from Ollama's library (downloaded once, stored locally)
- Step 3: Run the model in your terminal or call it via the local REST API
- Step 4: Use it offline for anything: coding help, writing, analysis, automation
Supported AI Models
Ollama supports a wide range of open-weight models. Here are the most popular ones and what each is best at:
Start with llama3 or mistral for general use. If your machine has only 4 to 6 GB RAM, use gemma2 or phi3 instead, they are smaller but still capable for most everyday tasks.
Prerequisites
Before installing, confirm your system meets these requirements:
No GPU is required. Ollama runs fine on CPU-only machines. A GPU speeds up inference significantly if you have one, but it is not a barrier to getting started.
Step 1: Install Ollama
Update your system first, then install Ollama using the official script:
# Update system packages first
sudo apt update && sudo apt upgrade -y
# Install Ollama with the official one-line installer
curl -fsSL https://ollama.com/install.sh | sh
# Verify the installation
ollama --versionIf the version number prints successfully, for example ollama version 0.x.x, the installation worked. If the Ollama service doesn't start automatically, start it manually:
# Start the Ollama server manually if needed
ollama serveLeave this running in one terminal and open a second terminal to run your models.
Step 2: Download and Run Your First Model
This command downloads llama3 (if not already installed) and immediately starts an interactive chat session:
# Download and run llama3
ollama run llama3The first run downloads the model, which is around 4.7 GB. Depending on your connection speed this takes a few minutes. After that, you'll see a prompt where you can type directly:
>>> Explain what a Docker container is in simple terms.
>>> Write a Python function to read a CSV file and return the column names.
>>> Summarize the key points of this text: [paste your text here]
>>> /byeType /bye or press Ctrl+D to exit the session. The model stays installed on your machine and loads instantly on future runs.
Step 3: Learn the Essential Commands
Step 4: Use the Local REST API
Ollama runs a local REST API on port 11434. This lets you call the model from your own scripts, applications, and automation tools without going through the interactive terminal session:
# Send a prompt via curl (non-streaming response)
curl http://localhost:11434/api/generate \
-d '{
"model": "llama3",
"prompt": "What is the difference between a process and a thread?",
"stream": false
}'# Python example using the requests library
import requests
response = requests.post(
"http://localhost:11434/api/generate",
json={
"model": "llama3",
"prompt": "Summarize this in three bullet points: " + your_text,
"stream": False
}
)
print(response.json()["response"])This is where Ollama becomes genuinely powerful for developers. You can build document summarizers, code review tools, automated report generators, internal chatbots, and any other AI-powered application, all calling your own local model over localhost with no API keys, no rate limits, and no charges per call.
What You Can Actually Build With This
Troubleshooting Common Issues
GPU Acceleration: Optional but Worth Knowing
If you have a compatible GPU, Ollama detects it automatically and uses it for inference. This makes responses significantly faster, especially for larger models.
- NVIDIA GPUs: Ollama uses CUDA automatically if the NVIDIA drivers and CUDA toolkit are installed
- Verify with:
nvidia-smi(should show your GPU) - Install drivers:
sudo apt install nvidia-driver-535(version may vary)
- Verify with:
- AMD GPUs: Ollama uses ROCm for AMD GPU acceleration
- ROCm support is improving but more limited than NVIDIA at this time
- Check compatibility at
rocm.docs.amd.com
- No GPU: Ollama falls back to CPU inference automatically. It works, just slower. For most tasks on a modern CPU, the speed is entirely usable.
Quick Reference Card
Conclusion
Ollama makes running a capable AI on your own Linux machine genuinely straightforward. One install command, one model download, and you have a private AI that costs nothing to run, works offline, and keeps every prompt on your own hardware.
- No API fees, no subscriptions, no data leaving your machine
- Works without a GPU, though a compatible GPU makes it faster
- The local REST API lets you build real applications on top of it
- Start with llama3 or mistral for general use, codellama for coding tasks, gemma2 if your machine has limited RAM
- Most troubleshooting issues are straightforward: not enough RAM, service not running, or a PATH issue after install
The technology that was behind expensive API paywalls two years ago is now running on ordinary Linux machines for free. If you work with sensitive data, build tools that need AI capabilities, or simply want to stop paying for something your own hardware can do, Ollama is where to start.