AI Chatbots: Building Conversational Applications with LLMs
The conversational AI landscape has transformed dramatically since the early days of rule-based chatbots like ELIZA (1966) and pattern-matching systems. Modern LLM-powered chatbots understand context, maintain coherent multi-turn conversations, use tools to take actions, and personalize responses based on user history. Companies across every industry are deploying conversational AI for customer support, internal knowledge access, sales qualification, and workflow automation.
Building a production chatbot requires more than wrapping an LLM API. You need conversation management, token optimization, streaming infrastructure, memory systems, tool integration, and evaluation pipelines. This guide covers the architectural patterns and engineering practices that distinguish hobby projects from enterprise deployments.
Core Architecture: The Four-Layer Model
A production chatbot decomposes into four distinct layers. The API gateway handles authentication, rate limiting, and request routing. The chat engine manages conversation state, token budgeting, and orchestration between components. The integration layer connects to tools, databases, and retrieval systems. The LLM layer handles model selection, prompt assembly, and response generation.
This separation of concerns enables independent scaling and optimization of each layer. If your chatbot needs faster responses, you can optimize the LLM layer (switch to a smaller model or enable speculative decoding) without touching the conversation management code. If your users report irrelevant answers, you improve the retrieval layer without redeploying the full stack.
Intercom’s Fin chatbot, for example, uses a tiered architecture where a lightweight intent classifier routes queries to specialized handlers — product questions go to a documentation RAG pipeline, account questions route to CRM lookups, and complex issues escalate to human agents. This approach achieves 82% resolution rates while keeping average response latency under 1.5 seconds.
Conversation Management and State
Stateless design is the golden rule for production chatbots. Each API request should contain all context needed to generate the next response. The conversation history is stored on the client side (for web chatbots) or in a database (for API integrations), and included in every LLM request as a messages array.
This approach provides resilience — if the server restarts, no conversation state is lost. It also enables horizontal scaling since any server instance can handle any request. The trade-off is increased token usage, as history is sent with every call. For long conversations, this necessitates context window management strategies.
Sliding window is the simplest approach: keep the system prompt, the last N exchanges, and discard older messages. A typical window might be 20-30 exchanges (about 4K-6K tokens). The limitation is that the chatbot loses context of early conversation topics — if a user asks about pricing, then asks about features, then returns to pricing, the sliding window may have forgotten the initial context.
Summarization addresses this by periodically compressing older conversation turns into a summary. Every 10 exchanges, run the conversation through a summary model and replace the oldest 5 turns with a single summary message. This preserves context while keeping token usage bounded. Anthropic’s Claude uses a similar approach internally, maintaining both a running summary and recent message history.
Token-aware truncation uses tiktoken to count tokens precisely. When the message array exceeds the target budget, it removes messages from the middle (preserving system prompt and recent context) until the total fits within limits. This is more efficient than arbitrary exchange counts because different messages have different token lengths.
Streaming for Real-Time UX
Streaming is essential for production chatbots. Users expect to see responses character by character, not wait for the full response to generate. Streaming reduces perceived latency from 3-8 seconds to under 500 milliseconds for first-token delivery.
Server-Sent Events (SSE) are the standard transport for streaming. Each chunk contains a JSON object with the token delta, and the client accumulates tokens and renders them incrementally. The FastAPI StreamingResponse pattern shown earlier is the most common Python implementation. For client-side rendering, handle partial tokens carefully — token boundaries don’t align with word boundaries, so accumulate partial tokens until a space or punctuation is received before updating UI.
Edge streaming optimizes further by running the LLM inference as close to the user as possible. Cloudflare Workers, Vercel Edge Functions, and AWS Lambda@Edge can terminate the SSE connection at the edge and stream responses directly, shaving 100-300ms off first-token latency. OpenAI’s streaming API supports this pattern natively with stream_options: {"include_usage": true} for token usage metadata in the final chunk.
Tool Integration and Function Calling
Chatbots that can only talk are limited. Production chatbots need to look up information, update records, send notifications, and trigger workflows. Tool integration via function calling gives chatbots the ability to interact with external systems.
The function calling flow follows a specific protocol. The user sends a message; the model responds with either a text reply or a function call request (detected by the presence of tool_calls in the response); the application executes the function and sends the result back as a tool response message; the model generates a final text reply incorporating the tool result.
Each tool function must include: a clear description of what it does (the model uses this to decide when to call it), typed parameters with descriptions, and the execution logic. Tools should be designed to fail gracefully — if a database query times out, return a helpful error message rather than throwing an exception. The model can then decide how to communicate the failure to the user.
Zendesk’s Answer Bot uses tool integration to search its knowledge base, check ticket status, update ticket properties, and escalate to human agents. Each tool is independently tested and monitored, with dashboards tracking tool usage rates, success rates, and average execution times. Tools with high failure rates trigger automated alerts and can be temporarily disabled without affecting the chatbot’s core functionality.
Memory Systems for Personalization
Memory transforms a chatbot from a generic assistant into a personalized experience. Short-term memory (the current conversation) is handled by the message array. Long-term memory requires persistent storage of user facts, preferences, and conversation history across sessions.
The most practical implementation uses a vector database to store memory chunks with embeddings. When a user starts a new session, the chatbot retrieves the most relevant memories — their name, past projects, preferred communication style, previously discussed topics — and injects them into the system prompt. This gives the chatbot continuity across sessions without requiring the full history in the context window.
Episodic memory captures specific events: “User reset their password on March 15” or “User mentioned they prefer email over phone support.” Semantic memory stores facts: “User works at Acme Corp in the engineering department.” Procedural memory remembers user interaction patterns: “User typically asks 2-3 questions before making a purchase decision.”
Privacy considerations are paramount. Implement data retention policies, give users visibility into what the chatbot remembers about them, and provide a way to delete memories. GDPR and CCPA compliance requires explicit consent for memory storage and the ability to export or delete user data on request.
Deployment and Operations
Production chatbot deployment requires robust infrastructure. Use a dedicated API service (FastAPI, Express, or similar) with connection pooling for LLM providers. Implement the following before going live: authentication and authorization, rate limiting per user and per IP, request logging with conversation IDs for debugging, error monitoring with alerts on failure rate spikes, and gradual rollout with feature flags.
Latency budgets should be explicitly defined. A typical chatbot SLA might be: p50 latency under 2 seconds, p95 under 5 seconds, and p99 under 10 seconds. Monitor each component separately — LLM inference latency, retrieval latency, tool execution latency — to identify bottlenecks. Use APM tools like Datadog, New Relic, or LangSmith for distributed tracing.
Feedback collection is essential for continuous improvement. Simple thumbs-up/thumbs-down buttons on each response let you collect preference data at scale. Use this data to fine-tune the response model, improve retrieval rankings, and identify common failure modes. Companies like Intercom report processing over 100 million feedback signals per year for model improvement.
FAQ
How do I handle very long conversations? Implement a tiered approach: keep recent messages in context, summarize older segments, and store the complete history in a database. Use token counting to trigger summarization when the context window reaches 70% capacity. For extremely long sessions (100+ messages), consider starting a new conversation with a summary prompt.
Should I use a single model or multiple models? A multi-model architecture is more efficient. Use a small, fast model (like GPT-4o mini or Llama 3.2 3B) for intent classification and simple Q&A. Route complex queries to a frontier model. This reduces costs by 50-70% while maintaining quality for the most challenging cases.
How do I prevent prompt injection in chatbots? Separate instructions from data using delimiters, validate tool call parameters before execution, implement output filtering for sensitive content, and use models with strong instruction adherence (Claude 3.5 is particularly resistant to injection). Never include raw user input in the system prompt without sanitization.
What metrics should I track for a production chatbot? Track resolution rate (percentage of conversations that don’t require human escalation), user satisfaction score (from feedback buttons), average conversation length, average response time, token usage per conversation, and tool call success rate. Benchmark these against your pre-LLM chatbot or human support team baseline.
How often should I update the chatbot’s prompts? Treat prompts like code — version control them, review changes through pull requests, and A/B test modifications before full rollout. Re-evaluate prompts quarterly against your test suite of 100-200 representative queries to catch regression. Update prompts when you change models or add new tools.