Embeddings and Semantic Search: Complete Guide
Semantic search represents a fundamental shift in how we find information. Traditional keyword search, based on exact term matching (BM25, TF-IDF), is brittle — “car” won’t match “automobile,” “fast” won’t match “quick,” and “how to fix a leaky faucet” bears no keyword resemblance to “plumbing repair guide.” Semantic search, powered by text embeddings, understands meaning: it finds documents that answer the user’s intent, not just documents that share words.
The impact is measurable. Companies that have migrated from keyword-only search to hybrid keyword-semantic search report 30-50% improvements in search relevance metrics (NDCG@10, MRR). Pinterest’s recommendation system, using embeddings for pin similarity, drives 50% of user engagement. eCommerce platforms using semantic product search see 15-25% increases in conversion rates. Embeddings have become foundational infrastructure for modern AI systems, powering not just search but also RAG pipelines, recommendation engines, clustering, anomaly detection, and classification.
What Are Embeddings?
An embedding is a dense vector of floating-point numbers — typically 384 to 3072 dimensions — that represents the “meaning” of a piece of text. The critical property is that semantically similar texts produce similar vectors. “Puppy” and “dog” will be close in embedding space; “puppy” and “derivative” will be far apart. This geometric relationship enables mathematical operations on meaning: you can add vectors (king - man + woman ≈ queen), cluster them, and search by proximity.
Embeddings are generated by neural network models trained on massive text corpora using contrastive learning objectives. The training objective is simple: pull embeddings of similar texts together in vector space while pushing dissimilar texts apart. OpenAI’s text-embedding-3-small was trained on hundreds of millions of text pairs using this principle. The training data includes search query-document pairs, question-answer pairs, and paraphrases — any data where semantic equivalence is known.
The embedding space is not uniform. Different embedding models produce different vector spaces with different similarity semantics. OpenAI’s embeddings capture semantic similarity well but struggle with nuanced domain terminology. Cohere’s multilingual embeddings excel at cross-lingual search. Sentence Transformers (Reimers and Gurevych, 2019) offer specialized models for domains like biomedicine, legal, and code.
Generating Embeddings: Providers and Methods
OpenAI’s embeddings API is the most popular for production use. text-embedding-3-small (1536 dimensions) costs $0.02 per million tokens and offers the best quality-to-cost ratio. text-embedding-3-large (3072 dimensions) costs $0.13 per million tokens and provides higher accuracy at the cost of storage and retrieval latency. OpenAI also supports dimensionality reduction via a dimensions parameter — you can request 256-dimensional vectors from the large model, which often outperform the small model’s native 1536 dimensions.
Sentence Transformers (sbert.net) provide self-hosted embedding generation. Models like all-MiniLM-L6-v2 (384 dimensions) run on CPU with sub-millisecond latency and are suitable for small-to-medium deployments. bge-large-en-v1.5 from BAAI (1024 dimensions) offers state-of-the-art performance for English, competitive with commercial APIs. For multilingual applications, intfloat/multilingual-e5-large supports 100+ languages.
Cohere’s Embed API offers specialized models for classification and search use cases, with the unique ability to rerank results (a separate model re-scores retrieved candidates by relevance). Google’s Vertex AI embeddings integrate natively with BigQuery and other GCP services. Amazon Bedrock embeddings via Titan Text are tightly integrated with the AWS ecosystem.
Model selection involves a trade-off between dimensionality, quality, cost, and latency. For production RAG systems, benchmark 3-5 embedding models on your specific data using retrieval metrics (recall@k, MRR). The best model for a general-purpose web search may not be the best for medical literature or legal documents.
Similarity Metrics and Vector Math
The most common similarity metric for text embeddings is cosine similarity, which measures the angle between two vectors. Cosine similarity values range from -1 (opposite meaning) to 1 (identical meaning), with 0 indicating orthogonal (unrelated) vectors. Cosine is preferred for text because it’s invariant to vector magnitude — longer documents produce larger-magnitude embeddings, but cosine similarity normalizes this out.
Euclidean distance (L2 distance) measures straight-line distance in vector space. For normalized embeddings (unit vectors), cosine similarity and Euclidean distance are monotonic — ranking by one is equivalent to ranking by the other. Dot product similarity is computationally fastest and is equivalent to cosine for normalized vectors. Most vector databases default to dot product for performance.
Normalization is an important preprocessing step. Embedding vectors should be normalized to unit length (L2 norm = 1) before indexing. This ensures that similarity search behaves consistently and enables optimized search algorithms like maximum inner product search (MIPS) used by many vector databases.
Semantic Search vs. Keyword Search
Keyword search (BM25) and semantic search excel in different regimes. BM25 is deterministic, fast, and requires no training data or model inference — it simply counts and weights term frequencies. It’s excellent for exact matches (product SKUs, legal citations, technical identifiers) and performs well out of the box without any setup.
Semantic search requires embedding generation, vector indexing, and similarity computation — more infrastructure and latency. But it captures synonyms (“car” matches “automobile”), handles phrasing variation (“how to plant tomatoes” matches “tomato growing guide”), and understands query expansion (“best programming language for data science” returns Python, R, Julia without mentioning them).
Hybrid search combines both approaches and is the industry standard for production systems. A typical hybrid search system runs BM25 and vector search in parallel, merges the result sets using weighted scoring (Reciprocal Rank Fusion or learned weighting), and re-ranks the top candidates using a cross-encoder model. RRF, the simplest fusion method, assigns a score of 1/(rank + k) from each method and sums them, producing robust results without tuning. Weighted averaging (e.g., 0.3 × BM25 score + 0.7 × semantic score) requires tuning for each domain.
Weaviate, Qdrant, and Pinecone all support hybrid search natively. Elasticsearch 8.0+ includes native vector search alongside BM25, enabling hybrid queries through a single API. Elastic’s approach uses a kNN search with optional post-filtering or pre-filtering via Boolean queries.
Building a Semantic Search Pipeline
A production semantic search pipeline has five stages. Document ingestion processes source documents — parsing, cleaning, and extracting text from PDFs, HTML, or databases. Chunking splits documents into searchable passages, typically 256-512 tokens with 10-20% overlap. The chunk boundary strategy matters: split on paragraph boundaries for prose, on section headers for technical documentation, and on function boundaries for code.
Embedding generation runs each chunk through the embedding model. Batch processing improves throughput — most providers support batched embedding requests (up to 100 texts per batch for OpenAI). For self-hosted models, use ONNX runtime or vLLM for optimized inference. Indexing stores embeddings in a vector database with metadata (source document, position, section heading, date) for filtering and presentation.
Query serving embeds the user’s query and performs similarity search. The retrieval should return 3-10 results with relevance scores. Post-retrieval processing includes metadata filtering, result deduplication, and optional re-ranking with a cross-encoder model for improved precision. The final pipeline should instrument each stage for latency monitoring — embedding latency, search latency, and re-ranking latency should all be tracked separately.
Evaluation Metrics for Search Quality
Search quality evaluation uses standardized metrics. Recall@k measures the proportion of relevant documents found in the top k results — critical for RAG systems where missing a relevant document means the LLM can’t answer. Precision@k measures the proportion of retrieved results that are relevant — important for user-facing search where irrelevant results erode trust.
Mean Reciprocal Rank (MRR) measures where the first relevant result appears — relevant for navigational queries where the user expects one correct answer. Normalized Discounted Cumulative Gain (NDCG) accounts for graded relevance — not all relevant results are equally useful. NDCG@10 is the standard metric for web search evaluation.
Create a test set of 50-200 queries with human-judged relevance labels. Run each query through your pipeline and compute metrics. A good baseline for most domains is recall@5 > 0.8 and precision@5 > 0.6. Below these thresholds, users will notice missing results and irrelevant returns.
FAQ
What dimensions should I use for embeddings? 1536 dimensions (text-embedding-3-small) offers the best quality-to-cost ratio for most applications. Use 384 dimensions for lightweight applications where speed and storage matter more than marginal quality improvements. Use 3072 dimensions (text-embedding-3-large) for high-stakes retrieval where every percentage point of recall matters.
Can I use embeddings for classification? Yes. Embeddings combined with a simple classifier (logistic regression, SVM, or nearest neighbor) often achieve performance close to fine-tuned models, especially in few-shot settings. This approach is called “embedding-based zero-shot classification” and works well when you have limited labeled data.
How often should I update embeddings? Regenerate embeddings when your document collection changes significantly (additions, removals, updates). For static collections, generate once and index. Embedding models themselves evolve less frequently — update annually or when a new state-of-the-art model is released.
What is the difference between sparse and dense embeddings? Dense embeddings (the kind described here) are dense vectors where every dimension carries information. Sparse embeddings (like SPLADE) are high-dimensional vectors where most dimensions are zero, representing vocabulary terms with learned weights. Dense embeddings excel at semantic similarity; sparse embeddings excel at term matching and interpretability.
How do I handle multilingual search? Use a multilingual embedding model like Cohere’s embed-multilingual-v3.0 or intfloat/multilingual-e5-large. These models map texts from different languages into a shared embedding space, enabling cross-lingual search — a query in English can retrieve relevant documents in Spanish or Japanese.