Running Local LLMs with Ollama: Complete Guide
Running LLMs locally has moved from a niche hobbyist pursuit to a practical infrastructure choice for many organizations. Ollama, an open-source project launched in early 2024, has been the primary driver of this shift. It packages the complexity of model download, quantization, GPU acceleration, and API serving into a single command-line tool that “just works.”
The appeal is straightforward. Local models offer complete privacy — your data never leaves your machine, making them suitable for sensitive applications in healthcare, legal, finance, and government. They provide zero-latency inference after the first token, since there’s no network round trip. They work offline, in air-gapped environments, and on devices with intermittent connectivity. And they eliminate ongoing API costs — after the initial hardware investment, inference is free. Companies like Apple (with MLX), Mozilla (with llamafile), and Meta (with Llama) have all invested in making local inference accessible.
Why Run LLMs Locally?
The privacy argument is the strongest for most users. When you send a prompt to OpenAI or Anthropic, that data is processed on their servers, stored for monitoring and improvement, and potentially subject to data breaches. For applications involving proprietary business information, personal medical data, or confidential legal documents, this creates unacceptable risk.
Latency is another compelling factor. API-based models have a best-case round-trip time of 300-500ms. Local models, after the first token, can generate at 50-100 tokens per second on a modern GPU — essentially real-time. For interactive applications like coding assistants or chatbots, this responsiveness transforms the user experience.
Cost is the third factor. At scale, API costs become significant — $2.50 per million input tokens for GPT-4o, $15 per million output tokens. An organization processing 100 million tokens per day would spend $500-$1,500 daily on GPT-4o. A local setup requires upfront hardware investment ($5,000-$30,000 for a production-grade GPU server) but zero per-token cost after that. For high-volume applications, the break-even point is typically 2-6 months.
Installation and Setup
Ollama supports macOS, Linux, and Windows. Installation on Linux is a single command: curl -fsSL https://ollama.com/install.sh | sh. The installer configures Ollama as a systemd service, creates a dedicated user, and sets up logging. On macOS, download the DMG from ollama.com. On Windows, use the official installer.
After installation, verify with ollama --version. The default installation listens on localhost:11434 and stores models in ~/.ollama/models/. You can change the storage location with the OLLAMA_MODELS environment variable and the listen address with OLLAMA_HOST. For production deployments, set OLLAMA_HOST=0.0.0.0 to accept remote connections (secure with a reverse proxy and authentication).
Ollama’s server can be configured for specific hardware. Set OLLAMA_NUM_PARALLEL to control concurrent requests (default: 1 for CPU, 4 for GPU). Set OLLAMA_MAX_LOADED_MODELS to limit simultaneously loaded models (default: 1). For GPU acceleration, ensure NVIDIA drivers and CUDA toolkit are installed — Ollama automatically detects and uses compatible GPUs. For AMD GPUs, use the ROCm-enabled build. For Intel GPUs, use the Vulkan backend.
Downloading and Managing Models
Ollama provides access to hundreds of models through its library. The ollama pull <model> command downloads the model and its default quantization. ollama run <model> pulls (if not already downloaded) and runs the model in interactive mode with a chat interface. ollama list shows downloaded models with their sizes. ollama rm <model> deletes a model to free disk space.
Popular model choices include: Llama 3.2 (3B parameters, fastest, runs on CPU/8GB RAM, good for simple Q&A and classification), Mistral (7B, excellent for coding and reasoning, needs 8GB+ RAM), Llama 3.1 (8B, strong general-purpose, similar requirements to Mistral), Qwen 2.5 (7B-72B, excellent multilingual support), Phi-3 (3.8B-14B, Microsoft’s efficient models, surprising capability for their size), Gemma 2 (2B-27B, Google’s open models, strong instruction following), DeepSeek Coder (6.7B-33B, specialized for code generation), and Llama 3 (70B-405B, frontier-quality but requires 48GB+ VRAM for the 70B).
Model selection is a trade-off between quality and resource requirements. The 7B-8B class models offer the best quality-to-hardware ratio — they run on consumer GPUs (RTX 3060/4060 with 8-12GB VRAM) and provide 90%+ of the quality of larger models for most tasks. The 70B+ models approach frontier quality but require professional GPUs (A6000, A100, H100) or multi-GPU configurations.
Hardware Requirements and Optimization
The hardware math is straightforward but important to understand. A 7B model in 16-bit precision requires 14 GB of memory (7 billion × 2 bytes). In 8-bit (Q8_0), it needs 7 GB. In 4-bit (Q4_K_M), it needs 4 GB. The quantization format directly determines both memory requirements and output quality.
CPU inference works for models up to 13B on modern hardware. Expect 2-8 tokens per second on a modern desktop CPU — usable for chat but too slow for real-time applications. The bottleneck is memory bandwidth, not compute. Systems with DDR5 memory (50-80 GB/s) perform significantly better than DDR4 (25-40 GB/s).
GPU inference is 10-50x faster than CPU. A mid-range GPU (RTX 4070, 12GB VRAM) runs 7B models at 40-80 tokens/second in 4-bit. A high-end GPU (RTX 4090, 24GB VRAM) runs 13B models in 4-bit at 30-50 tokens/second or 70B models in 4-bit with CPU offloading.
Apple Silicon (M-series chips) benefits from unified memory — the M2 Ultra with 192GB unified memory can run 70B models entirely in RAM without PCIe transfers. Performance on Apple Silicon (MLX backend) is competitive with mid-range GPUs. The M4 Max provides particularly strong performance for local AI workloads.
The Ollama API
Ollama provides a REST API at http://localhost:11434 with endpoints for generation, chat, embeddings, and model management. The /api/generate endpoint handles text completion: send a JSON body with model, prompt, and optional parameters (stream, temperature, top_p, max_tokens). Streaming responses send each token as a separate JSON line (NDJSON format). The final response includes token usage statistics.
The /api/chat endpoint provides a chat-completions-style interface similar to OpenAI’s API. Send a messages array with role (system, user, assistant, tool) and content. This endpoint supports conversation history and tool calling. The /api/embeddings endpoint generates embeddings for any text using the loaded model’s embedding capability (supported by most modern models like Llama 3, Mistral, Qwen).
The Ollama API is OpenAI API-compatible for the chat completions endpoint. This means you can use any OpenAI client library with Ollama by changing the base URL to http://localhost:11434/v1. This compatibility is extremely useful — it lets you use LangChain, LlamaIndex, and other frameworks with local models by changing a single configuration value.
Python Integration
The ollama Python library provides a native interface: pip install ollama. The ollama.chat() function mirrors the API, accepting model name and messages. ollama.generate() provides text completion. ollama.embeddings() generates embeddings. ollama.list() shows available models. The library handles streaming, error handling, and connection management automatically.
For custom integration, use the requests library directly against the API. This gives you finer control over parameters and error handling. The streaming response format requires parsing NDJSON lines — each line is a JSON object with response (token), done (boolean), and at the final message, total_duration, load_duration, prompt_eval_count, eval_count, and eval_duration.
LangChain integration is seamless: ChatOllama(model="llama3.2") creates a chat model wrapper. OllamaEmbeddings(model="llama3.2") creates an embeddings wrapper. LangChain’s Ollama integration supports tool calling, structured output, and streaming. This makes LangChain an excellent orchestration layer for local-first applications.
Custom Modelfiles
Modelfiles are Ollama’s mechanism for customizing model behavior without fine-tuning. A Modelfile specifies: the base model (FROM), system prompt (SYSTEM), inference parameters (PARAMETER temperature, PARAMETER top_p, PARAMETER stop), template customization (TEMPLATE for chat format), and license information (LICENSE).
Create a Modelfile and build: ollama create my-custom-model --file ./Modelfile. The resulting model appears in ollama list and can be used like any other model. Modelfiles are portable — share the Modelfile (a few lines of text) instead of the full model checkpoint (multiple GB).
Parameter customization is powerful. Lower temperature (0.1-0.3) for deterministic tasks like code generation or data extraction. Higher temperature (0.7-0.9) for creative writing or brainstorming. Set num_ctx to adjust the context window size (default: 2048, max depends on model architecture). Set num_predict for maximum generation length. These parameters are also available at inference time via the API, overriding Modelfile defaults.
Advanced Modelfile features include: ADAPTER to load LoRA adapters (combine your fine-tuned adapter with the base model without merging), FROM <model> with quantization override (e.g., FROM llama3.2:q4_K_M), and multi-line SYSTEM prompts with conditional logic.
Production Deployment
Running Ollama in production requires additional infrastructure. Use a reverse proxy (Nginx, Caddy) for TLS termination, rate limiting, and request logging. Add authentication via API keys or JWT tokens — Ollama itself does not include authentication. Use Prometheus metrics (Ollama exposes /api/tags for model status and system metrics can be collected via standard monitoring agents).
Model serving at scale may require multiple Ollama instances behind a load balancer. Each instance should be configured with the same models. For GPU-accelerated setups, ensure CUDA/NVIDIA drivers are consistently configured across instances. Consider using Docker with Ollama’s official image for consistent deployment across environments.
Ollama’s performance characteristics under load depend on hardware and model size. A single RTX 4090 running a 7B model handles approximately 50-100 concurrent requests with batch processing. Ollama supports request batching internally — multiple requests to the same model are processed as batches on the GPU for maximum throughput. Monitor eval_duration to detect performance degradation under load.
FAQ
How do I choose a quantization level? Q4_K_M is the best default — it maintains 98%+ of full-precision quality while reducing memory by 4x. Use Q8_0 when quality is critical and you have memory headroom. Use Q2_K or Q3_K for the smallest possible model (useful for very limited hardware), accepting 5-15% quality degradation. Always benchmark your specific use case — some tasks are more sensitive to quantization than others.
Can I use GPU acceleration on a laptop? Yes, with limitations. A laptop RTX 4060 (8GB VRAM) runs 7B models in Q4_K_M comfortably at 30-40 tokens/second. The GPU will draw 80-115W under load, which will drain most laptop batteries in 1-2 hours. For extended use, reduce the power limit or run on CPU at lower speed.
How does Ollama compare to vLLM for serving? Ollama prioritizes ease of use and features (model management, Modelfiles, built-in chat interface). vLLM prioritizes raw throughput and memory efficiency (PagedAttention for 2-5x higher throughput). Use Ollama for development, prototyping, and moderate production loads (<100 concurrent requests). Use vLLM for high-throughput serving (>100 concurrent requests) with continuous batching.
Can I run multimodal models locally? Yes. Ollama supports multimodal models including Llama 3.2 Vision (11B), LLaVA (7B-34B, vision-language), and BakLLaVA. Pass images via the API as base64-encoded data or file paths in the chat interface. Multimodal inference requires more GPU memory — a 7B vision model is equivalent to a 13B text model in memory requirements.
What happens when Ollama runs out of memory? Ollama will fall back to CPU offloading for layers that don’t fit in GPU memory. This significantly slows generation — expect 5-15 tokens/second instead of 40-80. Monitor memory usage and choose models that fit entirely in your GPU’s VRAM for optimal performance. Use ollama ps to see memory usage per model.