Vector Databases: Complete Guide for AI Applications

AI & LLMs AI & LLMs 9 min read 1830 words Intermediate ExcellentWiki Editorial Team

Vector databases are the infrastructure layer that makes semantic search, RAG, and AI-powered recommendations possible at scale. They store and index high-dimensional embeddings — the numerical representations of text, images, audio, and other data — and enable fast similarity search across millions or billions of vectors.

The vector database market has exploded since 2023. Pinecone, the first dedicated managed vector database, now powers search at companies like Zapier and Gong. Qdrant and Weaviate offer self-hosted alternatives with enterprise features. Chroma provides a lightweight embedded option for prototyping. PostgreSQL’s pgvector extension brings vector search to the world’s most popular relational database. Understanding the trade-offs between these options is essential for building production AI systems.

What Is a Vector Database?

A vector database is designed specifically for vector similarity search. Unlike traditional databases that index scalar values (numbers, strings, dates) and retrieve by exact match or range queries, vector databases index dense vector representations and retrieve by distance — finding the “nearest neighbors” to a query vector.

The core operations are: INSERT (add a vector with optional metadata), SEARCH (find k nearest neighbors to a query vector, optionally filtered by metadata), UPDATE (modify a vector or its metadata), and DELETE (remove a vector). These operations must be fast, concurrent, and durable — just like any database.

Vector databases differ from vector search libraries (like FAISS, Annoy, or HNSWlib) in two critical ways. First, they manage data persistence and durability — if the server restarts, the data is still there. Second, they support CRUD operations — adding, updating, and deleting individual vectors without rebuilding the entire index. Libraries like FAISS require full index rebuilds for data changes, making them unsuitable for dynamic datasets.

How Vector Search Works: Indexing Algorithms

Exhaustive search — comparing the query vector against every stored vector — provides perfect recall but doesn’t scale. At 1 million vectors with 1536 dimensions, a single exhaustive search takes hundreds of milliseconds on modern hardware. At 1 billion vectors, it takes days.

Approximate Nearest Neighbor (ANN) algorithms trade a small amount of recall for dramatic speed improvements. Modern ANN indexes achieve 99% recall at 1,000x speedup over exhaustive search. The most common algorithms are:

HNSW (Hierarchical Navigable Small World) is the dominant algorithm — it’s the default in Pinecone, Qdrant, and Weaviate. HNSW builds a multi-layer graph structure. The top layer has few nodes (long-range connections for fast navigation to the right neighborhood), and lower layers have progressively more nodes (short-range connections for fine-grained search). Search starts at the top layer, navigates greedily to the nearest node, descends to the next layer, and repeats until reaching the bottom layer where the nearest neighbors are found.

HNSW parameters: M controls the number of connections per node (16-64, higher = better recall, more memory). ef_construction controls index build quality (100-500, higher = better recall, slower build). ef_search controls search quality (100-500, higher = better recall, slower search). Default values (M=16, ef_construction=200, ef_search=100) work well for most applications.

IVF (Inverted File Index) clusters vectors during indexing. Search identifies the nearest clusters and only searches within those clusters. IVF is memory-efficient (no graph structure) but slower for equivalent recall. Good for billion-scale indexes where memory is constrained. IVF parameters: nlist (number of clusters, 100-10,000), nprobe (number of clusters to search during query, 1-100).

DiskANN uses SSD-optimized graph indexes for billion-scale datasets. Data is stored on disk with smart caching of frequently accessed nodes. DiskANN indexes can handle 10-100x more vectors than memory-only indexes at comparable latency. Microsoft uses DiskANN for Bing search.

Major Vector Database Options

Chroma

Chroma is the simplest vector database — an embedded, Python-native library that runs in-process. Install with pip install chromadb, create a client, create a collection, add documents, and search — everything in memory by default, with optional persistence to disk.

Chroma is ideal for prototyping, small-to-medium applications (<1M vectors), and development environments. It supports OpenAI, Cohere, Google, and Sentence Transformers embeddings automatically through its embedding function abstraction. The trade-off is limited scalability — it doesn’t support distributed deployment, advanced filtering, or high concurrency out of the box.

For production beyond small datasets, Chroma offers a client-server mode with basic HTTP API. However, at scale, most teams migrate to a dedicated vector database.

Pinecone

Pinecone is the leading managed vector database. It handles infrastructure — indexing, scaling, backups, monitoring — so you focus on application code. Pinecone supports: HNSW indexing with configurable parameters, metadata filtering (pre-filtering only), hybrid search (vector + sparse/BM25), pod-based scaling (pods are units of compute and storage), and namespaces for multi-tenancy.

Pinecone’s pricing is per pod-hour plus storage. A single pod handles approximately 5M vectors with 1536 dimensions and provides sub-10ms search latency at p50. Scaling up means adding more pods or larger pod types. Pinecone is ideal for teams that want to avoid operational overhead and can accept the managed cost premium.

Pinecone’s API is clean and OpenAI-compatible. The Python client provides familiar CRUD operations. The serverless option (launched 2024) eliminates pod management — you pay per vector stored and per query, scaling automatically.

Qdrant

Qdrant is a self-hostable vector database written in Rust. It offers the best performance of the major options — benchmarks show 2-5x lower latency than Pinecone for equivalent recall. Qdrant supports: HNSW indexing, payload filtering (metadata filtering with full query language — pre-filtering and post-filtering), hybrid search, quantization for memory reduction, and multi-tenancy via collections.

Qdrant’s filtering capabilities are the most advanced. You can filter by scalar values, geo-locations, text keywords, and nested objects. Filters can include AND, OR, and NOT conditions with arbitrary nesting. This enables complex queries like: “Find products similar to this description where price < $50, category = electronics, rating > 4.0, and shipping to zip code 94105.”

Deployment options: Docker (single-node for development), Kubernetes (production with horizontal scaling), and Qdrant Cloud (managed service). Qdrant is ideal for teams that want control over infrastructure and need advanced filtering.

pgvector

pgvector is a PostgreSQL extension that adds vector storage and similarity search to the world’s most popular relational database. Install with CREATE EXTENSION vector;. Define columns as vector(1536), create an HNSW or IVFFlat index, and query with ORDER BY embedding <=> query_embedding LIMIT 10.

pgvector’s advantage is operational simplicity. If you already use PostgreSQL, you don’t need a separate database — vectors live alongside your relational data in the same database, with the same transactions, backups, and access controls. This eliminates data synchronization between a vector DB and your primary database.

The trade-off is performance. pgvector’s HNSW implementation is 2-3x slower than Qdrant’s on equivalent hardware. It doesn’t support advanced filtering (you can pre-filter with WHERE clauses but can’t combine with vector search in the same index traversal). For applications with <10M vectors and simple filtering needs, pgvector is an excellent choice.

Weaviate

Weaviate is an open-source vector database that combines vector search with semantic schema. Data is stored as objects with properties (like a document database) and vectors. Weaviate automatically generates embeddings using built-in modules (OpenAI, Cohere, Hugging Face) or custom models.

Weaviate supports hybrid search (vector + BM25), graph traversal across object references, and generative search (combine retrieval with LLM generation in a single query). The GraphQL API provides flexible querying. Weaviate Cloud offers a managed service.

Weaviate is ideal for applications that need both vector search and semantic understanding of data structure — for example, products with categories, specifications, and reviews where relationships between objects matter.

Choosing the Right Database

The decision matrix depends on your scale, operational preferences, and feature requirements. For prototyping (<100K vectors, single developer): Chroma — fastest setup, minimal configuration. For small production (<5M vectors, simple filtering): pgvector — no new database to manage if you’re already on PostgreSQL. For medium production (<50M vectors, complex filtering): Qdrant — best performance and filtering, reasonable operational overhead.

For large production (>50M vectors, zero ops overhead): Pinecone — managed infrastructure, serverless option. For PaaS convenience (want built-in embedding generation and GraphQL): Weaviate — rich features, good developer experience. For billion-scale (>1B vectors, budget for infrastructure): DiskANN or Qdrant with custom deployment.

Production Considerations

Hybrid search is essential for production. Pure vector search misses exact matches (product codes, technical terms, proper names). BM25 keyword search catches these but misses semantic matches. Hybrid search combines both, typically using RRF (Reciprocal Rank Fusion). Most vector databases support hybrid search natively.

Filtering performance matters. Pre-filtering (filter before vector search) is fast but may miss relevant results. Post-filtering (vector search then filter) is more accurate but may return fewer than k results if filtered vectors are excluded. Qdrant supports both modes. For production, benchmark both approaches on your data distribution.

Index maintenance requirements vary. HNSW indexes support incremental updates (add/delete without full rebuild). IVFFlat indexes require periodic rebuilds as data changes. Monitor index size growth and rebuild frequency. Schedule index optimization during low-traffic periods.

Cost management: Embedding storage costs roughly 4 KB per 1536-dimensional vector (float32). At 10M vectors, that’s 40 GB of storage. Quantization (converting to float16 or int8) halves or quarters storage but may reduce recall by 0.5-1%. Vector database compute costs scale with query volume and index complexity. Monitor latency-recall trade-offs — a 1% recall improvement may cost 2x more in compute.

FAQ

Do I need a vector database for RAG? For small-scale RAG (<10K documents), you can use in-memory FAISS or Chroma. For production RAG (50K+ documents), a vector database provides the durability, concurrency, and CRUD support needed for reliable operation. The transition typically happens when you have more than one developer working on the system or when you need to update documents without rebuilding the index.

How do I choose embedding dimensions? 1536 dimensions (text-embedding-3-small) is the best default. Lower dimensions (384-768) reduce storage and search latency but may miss subtle semantic distinctions. Higher dimensions (3072) improve recall but double storage costs. OpenAI’s dimensionality reduction lets you use the large model and reduce dimensions — this often outperforms the small model at equivalent dimensions.

What is the difference between vector search and semantic search? Vector search is the technical mechanism (finding nearest neighbors in embedding space). Semantic search is the application (retrieving results that match user intent, not just keywords). Vector search enables semantic search but also powers recommendation (“similar items”) and clustering (“group similar documents”).

How do I handle vector database migrations? Plan for schema evolution. Store the embedding model version as metadata on each vector. When you upgrade embedding models (which change the vector space), regenerate embeddings for all data — this is a full re-index. Test the new index in staging against a representative query set before switching. Run both indexes in parallel during transition.

Can I use multiple vector databases in the same application? Yes, and it’s common. Use lightweight, fast options (Chroma or pgvector) for real-time query serving. Use a scalable option (Qdrant or Pinecone) for the full knowledge base. The serving database is a smaller, optimized subset; the full database supports batch processing and experimentation.