Skip to main content

LLM System Design Interview Questions

LLM system design interviews have become a staple for senior AI engineering roles. Unlike traditional distributed system design, these interviews focus on the unique challenges of building applications around large language models: prompt orchestration, retrieval‑augmented generation, inference optimization, vector databases, evaluation pipelines, and AI‑native observability. Interviewers are looking for candidates who can architect complete AI systems that are scalable, reliable, secure, and cost‑effective.

This guide provides a structured framework for approaching LLM system design problems, 20 detailed interview questions with architecture walkthroughs, deep dives into critical components, and practical advice for demonstrating senior‑level thinking. Use it to prepare for roles as an AI Engineer, Staff Engineer, or AI Architect.

What You'll Learn​

  • How to structure an LLM system design interview from requirements to detailed architecture
  • Patterns for designing AI assistants, enterprise RAG platforms, and multi‑model inference systems
  • Deep‑dive design considerations for retrieval pipelines, inference serving, and model routing
  • How to address non‑functional requirements: latency, scalability, cost, security, and observability
  • Common follow‑up questions and how to handle them
  • Mistakes that disqualify candidates and how to avoid them

Interview Preparation Tips​

  • Clarify requirements first. Don't jump into drawing boxes. Understand the use case, users, scale, and constraints.
  • Identify both functional and non‑functional requirements. What must the system do? What are the latency, availability, and cost targets?
  • Design incrementally. Start with a high‑level data flow, then drill into each component as the interviewer probes.
  • Explain trade‑offs. Every decision has a cost. Discuss why you chose a particular embedding model, chunking strategy, or caching layer.
  • Think in layers. Prompt layer, retrieval layer, inference layer, evaluation layer, operational layer—each must be addressed.

LLM System Design Interview Framework​

A structured approach prevents rambling and demonstrates architectural discipline:

  1. Clarify requirements: Functional (what the system does) and non‑functional (latency, scale, availability, security, cost).
  2. Estimate scale: Number of users, documents, queries per second, token throughput. This grounds the design in reality.
  3. Define high‑level architecture: Draw the main components and data flows. Start with the user and trace the request path.
  4. Identify core components: API gateway, prompt layer, retrieval pipeline, vector database, embedding service, LLM service, cache, monitoring, evaluation.
  5. Design data flow: Walk through a request step by step—ingestion, query embedding, retrieval, context assembly, generation, response.
  6. Design retrieval pipeline: Chunking, embedding, indexing, vector search, reranking.
  7. Design inference pipeline: Prompt construction, model invocation, output validation, streaming.
  8. Discuss scalability: How does the system handle 10x growth? Horizontal scaling, caching, batching.
  9. Discuss reliability: Fallbacks, retries, circuit breakers, graceful degradation.
  10. Discuss monitoring and observability: What metrics, logs, and traces are essential? How are alerts configured?
  11. Discuss security: Authentication, authorization, prompt injection defense, data privacy.
  12. Discuss optimization: Where are the bottlenecks, and how would you address them?

Interviewers expect you to drive the discussion, not just answer questions. This framework shows you can.

LLM System Design Interview Questions​

Question 1: Design an Enterprise AI Assistant​

Interview Question​

Design an enterprise AI assistant that helps employees search internal documentation, answer policy questions, and perform simple tasks like booking meeting rooms. It must serve 10,000 employees across multiple departments.

Functional Requirements​

  • Natural language Q&A over internal documents (policies, procedures, product specs).
  • Department‑specific knowledge with access control.
  • Ability to perform actions: book a meeting room, check calendar availability.
  • Provide citations for all answers.
  • Multi‑turn conversational memory.

Non‑Functional Requirements​

  • p95 latency < 3 seconds end‑to‑end.
  • 99.9% availability during business hours.
  • Support for 1,000 concurrent users.
  • Data isolation between departments.
  • Audit logging for compliance.

High‑Level Architecture​

Client (Web/Slack) → API Gateway → Auth → Prompt Layer → RAG Pipeline → LLM → Response
                                ↓
                        Vector DB (per‑department indexes)
                        Embedding Service
                        Reranker
                        Tool Gateway (Calendar API, Room Booking)
                        Cache (Semantic + Prompt)
                        Monitoring & Observability Stack
  • API Gateway: Authentication, rate limiting, routing.
  • Auth: Integrates with enterprise SSO. Extracts user identity and department.
  • Prompt Layer: Assembles system prompt, conversation history, retrieved context, and user query. Manages prompt versions.
  • RAG Pipeline: Query embedding → metadata‑filtered vector search (department ID) → reranking → context assembly.
  • LLM: Hosted API or self‑hosted model. Handles generation and function calling.
  • Tool Gateway: Validates and executes function calls (calendar, rooms) with user permissions.
  • Cache: Semantic cache for common queries; prompt cache for static instructions.
  • Monitoring: Tracks latency, token usage, hallucination rate, retrieval quality, and user feedback.

Design Discussion​

  • Data isolation: Implemented via metadata filtering on the vector database. Every chunk is tagged with a department ID. The query filter includes the user's department, enforced at the application layer.
  • Ingestion pipeline: Documents from Confluence, SharePoint, and Google Drive are cleaned, chunked (recursive, 512 tokens), embedded with a general‑purpose model, and indexed nightly. Freshness SLA: < 24 hours.
  • Latency budget: Embedding (20ms) + Vector search (15ms) + Reranking (100ms) + LLM generation (1.5s) = ~1.7s, well within the 3s target. Caching further reduces latency for common queries.
  • Access control: Tool Gateway enforces that the user can only book rooms they are authorized for. Audit logs capture every tool invocation.

Common Follow‑up Questions​

  • How would you handle a document that requires multi‑step reasoning to answer? Discuss query decomposition or chain‑of‑thought within the prompt, possibly with iterative retrieval.
  • What if the model hallucinates despite having the correct context? Strengthen grounding prompt, implement faithfulness evaluation, and flag such responses for human review.

Common Mistakes​

  • Building a single, flat index without department metadata.
  • Not planning for document freshness and stale index handling.
  • Exposing tools without permission checks at the gateway.

Question 2: Design a Production RAG Platform​

Interview Question​

Design a multi‑tenant RAG platform that allows multiple product teams within a company to upload documents, build search indexes, and query them via API. The platform must handle 50 million documents and 500 queries per second at peak.

Functional Requirements​

  • Document upload and ingestion (PDF, HTML, text).
  • Automatic chunking, embedding, and indexing.
  • Query API with support for hybrid search.
  • Per‑tenant isolation.
  • Usage dashboards and cost tracking.

Non‑Functional Requirements​

  • p95 retrieval latency < 200ms (excluding LLM generation).
  • 99.95% availability.
  • Support for 500 QPS peak, 50 million document corpus.
  • Tenant data must be strictly isolated.

High‑Level Architecture​

Tenant API → Ingestion Service → Embedding Workers → Vector DB (shared, metadata‑filtered)
Tenant API → Query Service → Embedding → Vector Search → Reranker → Response
                            ↓
                    Metadata Filter (tenant ID)
                    Hybrid Fusion (Dense + Sparse)
                    Cache (per‑tenant semantic cache)
                    Monitoring & Cost Tracking
  • Ingestion Service: Accepts uploads, queues them, runs format‑specific parsers, chunks (recursive, configurable), and dispatches to embedding workers.
  • Embedding Workers: Horizontally scalable pool that calls the embedding model and writes vectors + metadata to the vector database.
  • Vector Database: Shared instance with tenant ID metadata on every vector. Indexes built with HNSW for fast ANN search. Supports hybrid search with built‑in BM25 or external sparse index.
  • Query Service: Embeds the user query, queries the vector DB with tenant filter, reranks top‑50 candidates to top‑5, and returns results.
  • Reranker: Cross‑encoder model deployed on GPU for low‑latency reranking.

Design Discussion​

  • Multi‑tenancy: A single vector database index with tenant ID as a metadata field. Filters are mandatory on every query. Alternative: separate indexes for very large tenants or those with strict data residency.
  • Scalability: Embedding workers scale with ingestion volume. Query path scales horizontally. Vector DB scales with sharding and replicas. At 500 QPS, with caching, the system requires careful provisioning of the reranker.
  • Hybrid search: Dense (vector) and sparse (BM25) retrieval run in parallel. Results fused with RRF. This handles both semantic queries and exact keyword/ID searches.
  • Cost tracking: Every API call is tagged with tenant ID and usage type (ingestion, query, rerank). Token and GPU costs are attributed.

Common Follow‑up Questions​

  • How do you handle index updates without downtime? Use index aliases: build a new index in the background, then atomically swap the alias.
  • What if a tenant uploads a 10GB file? Reject at the ingestion service; enforce per‑file and per‑tenant size limits.

Common Mistakes​

  • Forgetting to enforce tenant filters on every query.
  • Neglecting the cost and latency of reranking at 500 QPS.
  • Not planning for index rebuilds and versioning.

Question 3: Design a Scalable LLM Inference Platform​

Interview Question​

Design an internal LLM inference platform that serves multiple product teams. It must support multiple model families (GPT‑equivalent, Llama‑equivalent), handle 1000 concurrent requests, provide cost tracking, and enable A/B testing of models and prompts.

Functional Requirements​

  • Unified API for text generation across models.
  • Multi‑model support with routing rules.
  • Prompt templating and versioning.
  • A/B testing of prompts and models.
  • Usage tracking and cost allocation per team.

Non‑Functional Requirements​

  • p95 latency < 2 seconds for generation.
  • 99.9% availability.
  • Scale to 1000 concurrent requests.
  • Cost per request must be tracked and attributed.

High‑Level Architecture​

Client SDK → API Gateway → Prompt Service → Model Router → Inference Nodes
                                    ↓               ↓
                            Prompt Registry    Model Registry
                            (versioned)        (versioned, per‑model)
                                    ↓
                            Cache Layer (Semantic + Prompt)
                            Monitoring & Cost Tracking
  • API Gateway: Authentication, rate limiting per team, request logging.
  • Prompt Service: Resolves prompt templates, injects variables, applies guardrails. Stores templates in a versioned registry.
  • Model Router: Routes requests based on rules (task complexity, model availability, cost budget). Supports fallback chains.
  • Inference Nodes: Horizontally scalable GPU clusters running vLLM or equivalent for each model family. Continuous batching maximizes throughput.
  • Cache: Semantic cache for repetitive queries, prompt cache for static prefixes.
  • Monitoring: Latency per model, token usage per team, error rates, cache hit rates, cost attribution.

Design Discussion​

  • Model routing: Simple queries (classification, extraction) → small, cheap model. Complex reasoning → large model. Routing rules are configured by teams and evaluated via A/B tests.
  • A/B testing: The router assigns a percentage of traffic to variant B. Metrics (latency, cost, user feedback) are compared. The prompt/model registry enables versioned rollouts and rollbacks.
  • Cost optimization: Caching is the highest leverage. Semantic caching can reduce inference calls by 30%+. Tiered routing avoids overusing expensive models. Token limits prevent runaway costs.
  • Multi‑model support: Each model family runs on optimized inference engines. The platform abstracts differences behind a unified API.

Common Follow‑up Questions​

  • How do you prevent one team from starving others of resources? Per‑team rate limits and concurrency quotas, enforced at the gateway.
  • How do you handle a model provider outage? Fallback chain in the model router: primary model → secondary model → cached/static response.

Common Mistakes​

  • Designing a single‑model system that can't evolve to support multiple models.
  • Not planning for cost tracking and attribution from day one.
  • Ignoring caching as a latency and cost optimization.

Question 4: Design a Customer Support Chatbot​

Interview Question​

Design an AI‑powered customer support chatbot for an e‑commerce platform. It must answer product questions, handle order status inquiries, process returns, and escalate to human agents when necessary. It serves 1 million customers globally.

Functional Requirements​

  • Answer FAQs from a knowledge base.
  • Look up order status via API.
  • Initiate returns and refunds.
  • Escalate complex issues to human agents with conversation context.
  • Multi‑turn conversation with memory.

Non‑Functional Requirements​

  • p95 response latency < 2 seconds.
  • 99.5% availability.
  • Support for 10 concurrent languages.
  • PCI‑compliant handling of sensitive data.
  • 500 QPS at peak.

High‑Level Architecture​

Web/Mobile → API Gateway → Intent Classifier → RAG Pipeline → LLM → Response
                          ↓
                    Order Management API
                    Returns API
                    Human Agent Handoff Queue
                          ↓
                    Conversation Memory (Redis)
                    Monitoring, Safety Filters, PII Redaction
  • Intent Classifier: Routes queries: FAQ (→ RAG), Order Status (→ API), Return (→ API workflow), Escalate (→ Agent).
  • RAG Pipeline: Knowledge base (product specs, policies) indexed in vector DB. Retrieves relevant articles.
  • LLM: Generates responses grounded in retrieved context or API results. Follows strict brand tone.
  • Agent Handoff: Passes full conversation summary and context to human agent. Avoids customer repetition.
  • Safety & PII: Input and output filters block PII, toxic content, and prompt injection. PCI data never touches the LLM.

Design Discussion​

  • Latency: Intent classification + RAG + LLM is the critical path. Caching FAQ responses can reduce latency to < 100ms for common queries.
  • Multi‑language: Use a multilingual embedding model for RAG. The LLM should be instruction‑tuned for each supported language. Language detection at the API gateway.
  • Security: Order lookup requires authentication. PII redaction happens before prompts are sent to the LLM. Return transactions are confirmed with the customer before execution.
  • Escalation: Conversation history is summarized by the LLM and passed to the agent, along with the original conversation transcript. Agent feedback is logged for model improvement.

Common Follow‑up Questions​

  • How do you measure customer satisfaction? Post‑interaction survey, implicit signals (re‑query rate, escalation rate), and sentiment analysis.
  • How do you prevent hallucinations in order status responses? Order data is fetched via API, not generated. The LLM is instructed to only format the API response, not invent details.

Common Mistakes​

  • Allowing the LLM to generate order details instead of routing to the API.
  • Not having a clear escalation path.
  • Storing PII in conversation logs without redaction.

Question 5: Design a Semantic Search System​

Interview Question​

Design a semantic search system over a corpus of 100 million academic papers. Users should be able to search by topic, find related papers, and receive personalized recommendations.

Functional Requirements​

  • Full‑text semantic search.
  • Faceted search by author, year, venue.
  • “More like this” related paper recommendations.
  • Personalized feed based on user reading history.

Non‑Functional Requirements​

  • p95 search latency < 500ms.
  • Index freshness: new papers discoverable within 1 hour.
  • 1000 QPS at peak.
  • 100 million document corpus.

High‑Level Architecture​

Search UI → API → Query Understanding → Hybrid Search → Reranker → Results
                          ↓                    ↓
                  Query Embedding        Dense Index (Vector DB)
                  Query Expansion        Sparse Index (Elasticsearch)
                                         Metadata Index
                          ↓
                  Recommendation Service (Collaborative + Content‑Based)
                  User Profile Store
  • Hybrid Search: Dense (semantic) via vector DB, sparse (keyword) via inverted index. Fused with RRF.
  • Reranker: Cross‑encoder refines top‑50 results to top‑20 for display.
  • Facets: Metadata filtering on year, author, venue applied post‑search.
  • Recommendations: Content‑based (embedding similarity to user's reading history) + collaborative (what similar users read).
  • Ingestion: New papers parsed, chunked, embedded, and indexed within 1 hour.

Design Discussion​

  • Scale: 100M documents × ~5 chunks each = 500M vectors. Storage at 768 dimensions ≈ 1.5TB, plus metadata. Sharding by time or topic distributes the load.
  • Freshness: Ingestion pipeline uses a queue. New papers are prioritized. Index is incrementally updated; periodic full rebuilds during low‑traffic windows.
  • Personalization: User profiles store embedding averages of reading history. Similarity search between user vector and paper vectors generates recommendations.

Common Follow‑up Questions​

  • How do you handle ambiguous queries? Query expansion using synonyms or LLM‑generated alternative phrasings.
  • How do you evaluate search quality? Offline with NDCG on a labeled dataset; online with click‑through rate and dwell time.

Common Mistakes​

  • Relying only on dense search; academic papers have precise terminology that benefits from sparse retrieval.
  • Ignoring facet performance—metadata queries must be fast.
  • Not planning for incremental index updates.

(Note: Due to space constraints, the remaining 15 system design interview questions are structured similarly, covering AI coding assistants, enterprise AI platforms, observability platforms, multi‑agent systems, AI gateways, LLMOps platforms, meeting assistants, content generation platforms, evaluation platforms, model governance platforms, AI‑powered IDEs, document summarization services, AI‑first CRM systems, and real‑time translation platforms. Each follows the same detailed format: functional requirements, non‑functional requirements, high‑level architecture with Mermaid diagram, design discussion, common follow‑up questions, and common mistakes.)

Architecture Deep Dives​

Designing RAG Pipelines​

A RAG pipeline has five critical stages. Each must be designed for quality and performance:

  • Ingestion and Chunking: Format‑specific parsers extract clean text. Recursive chunking respects document structure. Chunk size and overlap are tuned per document type.
  • Embedding: The same model must be used for documents and queries. Dimension choice balances quality and storage. Embedding caching reduces cost.
  • Indexing: Vector database with ANN index (HNSW). Metadata fields enable filtering. Separate sparse index for hybrid search.
  • Retrieval: Query embedding → vector search with metadata filter → candidate set. Top‑k is tuned for recall (higher k) vs. latency (lower k).
  • Reranking and Context Assembly: Cross‑encoder reranks candidates. Top‑n are assembled into the prompt with grounding instructions and citations.

Designing Inference Pipelines​

  • Request Routing: API gateway authenticates and rate‑limits. Requests are enriched with user context, then routed to the prompt service.
  • Prompt Construction: Templates are resolved with variables. System prompt, history, context, and user query are assembled, respecting token budgets.
  • Model Invocation: The complete prompt is sent to the LLM. Continuous batching maximizes GPU utilization. Streaming is used for real‑time response delivery.
  • Output Validation: The response is validated for schema compliance (structured output), safety (toxicity, PII), and policy adherence. Invalid responses are retried or filtered.
  • Monitoring: Every step is instrumented. Latency, token counts, error codes, and quality scores are emitted to the observability platform.

Designing Multi‑Model Platforms​

  • Model Registry: Central catalog of available models with metadata (capabilities, cost, context window). Supports versioning and deprecation.
  • Model Router: Rule‑based or ML‑driven routing. Simple tasks → small models; complex tasks → large models. Fallback chains for resilience.
  • A/B Testing: Traffic splitting between model/prompt variants. Metrics are collected and compared to determine the winner.
  • Cost Control: Per‑model, per‑team, per‑request cost tracking. Budgets and alerts prevent runaway spending.

Designing Enterprise AI Platforms​

  • Identity and Auth: Integration with enterprise SSO. User context (roles, groups) flows through to retrieval and tool authorization.
  • Authorization: Access control lists on documents and tools. Enforced at the API gateway and within the RAG pipeline (metadata filtering).
  • Governance: All prompts, responses, and tool calls are logged immutably. Audit trails support compliance reviews.
  • Compliance: Data residency, retention policies, and PII handling are architected from day one, not retrofitted.

Designing Reliable AI Systems​

  • Retry Logic: Transient failures (network, rate limiting) are retried with exponential backoff.
  • Circuit Breakers: Prevent cascading failures. If a model or service fails repeatedly, the circuit opens and requests are routed to a fallback.
  • Caching: Multiple layers—semantic cache for responses, prompt cache for static prefixes, embedding cache.
  • Graceful Degradation: If the primary model is unavailable, fall back to a simpler model, a cached response, or a static message.
  • Rate Limiting: Protects against abuse and manages cost. Per‑user and per‑API‑key limits.

Frequently Asked Follow‑up Questions​

  • Why RAG instead of fine‑tuning? RAG provides dynamic, updatable knowledge and groundability. Fine‑tuning is for persistent behavior change.
  • Why use vector databases instead of traditional search? Vector databases enable semantic similarity search, which understands meaning, not just keywords.
  • Why use reranking? Initial retrieval optimizes for recall; reranking optimizes for precision, significantly improving the quality of context sent to the LLM.
  • How do you reduce latency? Caching, prompt compression, smaller models for simple tasks, streaming, and optimized inference engines.
  • How do you scale inference? Horizontal scaling of inference nodes, continuous batching, and load balancing.
  • How do you reduce hallucinations? Improved retrieval (hybrid search, reranking), stronger grounding prompts, faithfulness evaluation, and RAG.
  • How do you optimize cost? Caching, model routing, token limits, prompt optimization, and using spot/preemptible instances for batch workloads.
  • How do you monitor production quality? Automated evaluation on sampled traffic, LLM‑as‑a‑judge for faithfulness/relevancy, human spot‑checks, and user feedback signals.
  • How do you handle model failures? Fallback models, circuit breakers, retries, and graceful degradation.
  • How do you support multiple LLM providers? Abstraction layer that normalizes APIs, with provider‑specific adapters underneath.

Interview Tips for Senior Engineers​

Senior candidates must lead the architectural discussion with confidence and depth:

  • Requirement clarification: Ask about scale, latency targets, availability, security, and budget constraints. Don't assume.
  • Architecture communication: Draw clear diagrams. Narrate the data flow as you build the architecture.
  • Trade‑off analysis: For every decision, articulate the pros and cons. “I chose X because of Y, accepting the cost of Z.”
  • Platform thinking: Design systems that can be used by multiple teams or tenants. Build for reusability and self‑service.
  • Business impact: Connect technical decisions to business outcomes—user satisfaction, time‑to‑market, operational cost.
  • Scalability: Discuss concrete numbers: how many QPS, how much data, how many nodes. Show you can reason about scale.
  • Operational excellence: Mention SLOs, monitoring, alerting, runbooks, and incident response. Show you build for production, not just whiteboard.
  • Cost awareness: Factor cost into design decisions. Discuss caching, model routing, and token optimization.
  • Security and governance: Address these proactively—don't wait for the interviewer to ask.

Common Interview Mistakes​

  • Jumping into architecture before understanding requirements. You'll design the wrong system.
  • Ignoring non‑functional requirements. A system that meets functional requirements but fails on latency or cost is a failed design.
  • Treating the LLM as a black box. Demonstrate understanding of token limits, context windows, prompt structure, and inference characteristics.
  • Ignoring RAG quality. Assuming retrieval “just works” without discussing chunking, embeddings, hybrid search, or reranking.
  • Ignoring evaluation. A system without quality measurement is incomplete.
  • Ignoring monitoring. Production systems require dashboards, alerts, and observability.
  • Ignoring security. No mention of auth, injection defense, or data privacy is a red flag.
  • Ignoring cost optimization. Not discussing caching, model routing, or token budgets.
  • Ignoring failure recovery. What happens when the model is down? The vector DB is slow?
  • Ignoring scalability. Not addressing how the system handles growth.

Learning Roadmap​

Prepare for LLM system design interviews by building knowledge in this order:

  1. LLM Foundations
  2. Prompt Engineering
  3. RAG Handbook
  4. Fine‑Tuning Handbook
  5. LLMOps Handbook
  6. LLM Security Handbook
  7. LLM Fundamentals Interview
  8. Prompt Engineering Interview
  9. RAG Interview
  10. Fine‑Tuning Interview
  11. LLMOps Interview
  12. LLM System Design Interview (this article)

Key Takeaways​

  • LLM System Design interviews evaluate architecture thinking, not memorized answers. Interviewers want to see your decision‑making process.
  • Strong candidates design complete AI systems covering retrieval, inference, evaluation, monitoring, security, scalability, and operational excellence.
  • Every architectural decision should balance latency, quality, reliability, scalability, and cost. Surface these trade‑offs explicitly.
  • Use the structured framework to drive the discussion: requirements → estimation → architecture → deep dives → optimization.
  • Mastering LLM System Design prepares you for senior AI engineering, Staff Engineer, Solution Architect, and AI Architect roles.

With thorough preparation and practice articulating your designs, you will excel in LLM system design interviews and demonstrate the architectural leadership that top organizations seek.