LLM Application Architecture: Production Design Patterns
Building a production LLM application requires more than calling an API. Latency, cost, reliability, security, and scalability demand careful architectural decisions. A poorly architected LLM application can cost thousands of dollars per day in wasted API calls, suffer from 10-second latencies that drive users away, or fail catastrophically when a provider has an outage.
The patterns in this guide have been validated across hundreds of production deployments — from startups processing thousands of requests per month to enterprises handling millions per day. They apply regardless of the underlying LLM provider or framework. The principles are universal: cache aggressively, fail gracefully, monitor everything, and design for the weakest link in the chain.
The Basic Stack: Four-Layer Architecture
Production LLM applications decompose into four layers. The presentation layer handles user interaction — web interface, API, chat widget, or mobile app. The orchestration layer manages conversation state, routing, business logic, and composition of multiple LLM calls. The service layer provides reusable capabilities — search, knowledge retrieval, user data lookup, and third-party API integration. The LLM layer abstracts model providers, handles retries, manages token budgets, and implements safety controls.
This architecture enables independent scaling. If your application experiences a surge in traffic, you can scale the orchestration layer horizontally without duplicating LLM capacity. If you switch LLM providers, only the LLM layer changes. Each layer communicates through well-defined interfaces, typically REST APIs or message queues.
Layer isolation is critical for resilience. A failure in the service layer (database timeout) should not crash the orchestration layer. Implement timeouts (default: 5 seconds for internal calls), circuit breakers, and bulkheads (separate thread pools per layer) to contain failures. The LLM layer should have its own connection pool separate from the orchestration pool.
Caching: Response and Semantic
Caching is the single most impactful optimization for LLM applications. A well-implemented cache can reduce LLM API costs by 40-80% and latency by 90%+. There are two fundamentally different caching strategies.
Exact response caching stores the response for an identical prompt-model-configuration triple. The cache key is typically an HMAC-SHA256 hash of the full input (system prompt + messages + model + temperature + max_tokens). Use Redis with TTL for production. Choose TTL based on content freshness requirements — 5 minutes for real-time information, 24 hours for reference material, 7 days for static knowledge. Invalidation can be time-based or event-driven (triggering when source content changes).
Semantic caching returns cached responses for semantically similar (not identical) prompts. The query is embedded and compared against cached query embeddings using cosine similarity. If the nearest cached query is within a similarity threshold (typically 0.90-0.95), the cached response is returned. This catches users asking the same question with different wording — “What’s the return policy?” vs. “Can you explain your return policy?”
Semantic caching requires an embedding model and vector database, adding infrastructure complexity. Implement semantic caching only after exact caching is in place and you’ve confirmed that similar-rephrase patterns account for significant traffic. For most applications, exact caching with session-aware keys catches 30-50% of repeated queries; semantic caching adds another 10-20%.
Fallback Chains and Provider Diversity
LLM providers experience outages. OpenAI has had multi-hour outages affecting customers worldwide. Anthropic has rate-limited during traffic spikes. Google Cloud has had regional failures. A single-provider architecture is a single point of failure.
The fallback chain pattern maintains an ordered list of providers: primary, secondary, tertiary. On request, try the primary provider. If it fails (timeout, 5xx error, rate limit), try the secondary. Continue down the chain until a provider succeeds or all fail. The fallback is transparent to the user — they see a response, just potentially from a different model.
Implementation details matter. Each provider should use independent connection pools and API key configurations. The fallback should implement exponential backoff (100ms, 500ms, 2s) between attempts to allow transient failures to recover. Log which provider served each request and the reason for any fallback — this data informs provider selection and capacity planning.
Weighted routing extends the pattern for cost optimization: route 80% of traffic to GPT-4o mini ($0.15/M tokens) and 20% to GPT-4o ($2.50/M tokens). Use the smaller model for simple queries and escalate to the larger model when confidence is low (measured by log probabilities or a separate confidence classifier).
Rate Limiting and Queue Management
Rate limiting protects both your application (from abuse) and your provider account (from cost spikes and bans). Implement rate limits at multiple levels: per user (N requests per minute), per IP (across unauthenticated users), per API key (total requests per minute), and per endpoint (different limits for chat vs. embeddings).
The token bucket algorithm is the most common implementation. Each user has a bucket with a refill rate (tokens per second) and capacity (burst size). A request consumes one token. If the bucket is empty, the request is either queued or rejected with a 429 status. Redis is the standard backend for distributed rate limiting across multiple application servers.
Request queueing smooths traffic spikes. Incoming requests go to a queue (Redis, SQS, or similar). A worker pool processes requests from the queue at a controlled rate. Queue depth and processing latency become monitoring signals — growing queues indicate insufficient capacity. Priority queues let you process premium users’ requests ahead of free tier.
The queuing pattern also enables request batching. The embedding API is significantly cheaper per token with batched inputs (OpenAI batches up to 100 inputs per call). Group embedding requests from the queue into batches and send them together, reducing costs by 10-50x for high-throughput embedding workloads.
Circuit Breaker Pattern
The circuit breaker pattern prevents cascading failures when a provider is degraded. The breaker has three states: closed (normal operation, requests pass through), open (provider is failing, requests fail fast without attempting the call), and half-open (testing if provider has recovered).
Configure the breaker with: failure_threshold (number of consecutive failures before opening, typically 3-5), recovery_timeout (seconds before transitioning to half-open, typically 30-60), success_threshold_in_half_open (consecutive successes needed to close, typically 2-3). Track failures by error type — distinguish transient errors (429 rate limit, 503 timeout) from permanent errors (401 auth, 400 bad request). Permanent errors should not trip the breaker.
Implementation should be per-provider and per-endpoint within a provider. An OpenAI embeddings outage should not affect your OpenAI chat completions. Circuit breakers should be integrated with alerting — an open breaker triggers a PagerDuty notification for immediate investigation.
Structured Outputs and Validation
Structured outputs ensure model responses are parseable and valid. OpenAI’s response_format parameter with JSON Schema constrains the model’s output tokens to produce valid JSON. This eliminates parsing failures that plague applications relying on free-text responses.
Implement a two-layer validation. Schema validation checks that the response matches the expected structure — correct keys, types, and value ranges. Use Pydantic models for this. Semantic validation checks that the values are reasonable — dates are valid dates, amounts are non-negative, classifications are from the allowed set. This catches hallucinations that produce structurally valid but semantically wrong outputs.
For applications requiring 100% reliability, implement a retry-with-validation loop. If the response fails validation, resend the request with an additional instruction about the specific validation failure: “Your previous response had an invalid date format. Valid dates are in YYYY-MM-DD format. Please correct.” This pattern reduces validation failure rates from 5-10% to under 0.1%.
Observability and Monitoring
Observability is essential for production LLM systems. Log every LLM call with: timestamp, user ID (or anonymized session ID), conversation ID, model, prompt (with sensitive data redacted), response, tokens used (input and output), latency, cost, provider, retry count, and any errors. Store in a structured logging system (Elasticsearch, Datadog, CloudWatch) for querying and analysis.
Key metrics to track: latency (p50, p95, p99 per model and endpoint), error rate (broken down by error type — rate limit, timeout, invalid response), cost (per user, per feature, per model), token usage (input vs. output ratio, average per request), cache hit rate (exact and semantic), fallback rate (percentage of requests served by non-primary providers).
Set up alerts for anomaly detection: error rate > 1% for more than 5 minutes, p95 latency > 5 seconds, cost per user exceeding a threshold (indicates abuse or looping), fallback rate > 10% (provider degradation). Use dashboards to track trends — cost growth, latency degradation, cache effectiveness degradation.
FAQ
How do I handle long-running LLM requests in a web server? Use async workers with a timeout mechanism. FastAPI with async endpoints and asyncio.wait_for() gives you control over request deadlines. For requests exceeding the timeout, return a 202 Accepted with a callback URL or polling endpoint. Background workers (Celery, Redis Queue) handle the actual LLM call asynchronously.
What is the optimal batch size for LLM inference? For chat completions, batch size 1 is optimal (each request is unique). For embeddings, batch up to 100 inputs per API call for best throughput. For classification tasks where inputs share the same prefix, batch inference with vLLM can process multiple inputs simultaneously with minimal added latency — batch sizes of 4-16 are typical.
How do I estimate production costs? Estimate daily requests × average input tokens × price-per-token + daily requests × average output tokens × price-per-token. Add 20% for retries, tool calls, and evaluation. For RAG, include embedding costs: documents × chunks × embedding price + daily queries × embedding price. Monitor actual costs immediately — the first week of production often reveals estimates were 2-3x off.
Should I use synchronous or asynchronous architecture? Start synchronous for simplicity (FastAPI with async support). Move to asynchronous (message queues, background workers) when you need: request durability (messages survive server restarts), load leveling (smooth traffic spikes), or long-running agent loops (tasks that take minutes). Most applications can stay synchronous for their first 6-12 months.
How do I test for production readiness? Chaos engineering: simulate provider outages (block the API), test rate limiting (send rapid requests), saturate the queue (exceed processing capacity), corrupt responses (return invalid JSON). Each test should validate that fallbacks work, queues drain, and errors are properly reported to the user. These tests, combined with load testing at 5x expected traffic, provide confidence in production behavior.