LLM Development Environment Setup

Building LLM applications is fundamentally different from traditional web or mobile development. You are not just writing code that executes predictably on a server; you are orchestrating probabilistic systems, managing large models, handling complex data pipelines, and iterating on prompts that can behave unexpectedly. The tools you choose and the way you structure your environment have an outsized impact on your productivity, the reliability of your applications, and your ability to move from prototype to production.

A well-designed development environment is the foundation upon which everything else is built. It ensures reproducibility—so that what works on your machine works on your colleague's and in production. It provides the scaffolding for experimentation, enabling you to test prompts, evaluate models, and debug failures without fighting your tooling. And it scales with you, from a simple script that calls an API to a full-fledged RAG system with vector databases, agents, and monitoring.

This guide walks you through setting up a practical, production-ready LLM development environment. We cover the essential tools, software, workflows, and architectural considerations—from choosing an operating system and IDE to managing Python environments, working with LLM APIs, running local models, and structuring your projects for long-term maintainability. Whether you are a beginner taking your first steps or an experienced engineer transitioning into AI, this handbook will help you build a solid foundation for LLM development.

What Is an LLM Development Environment?

An LLM development environment is the complete set of tools, configurations, and workflows you use to build, test, and deploy LLM-powered applications. It encompasses everything from your code editor to your model endpoints, from your version control system to your vector database.

Developer
      │
      ▼
┌─────────────────────────────────────────────────────────────────┐
│                         IDE / Editor                           │
│              (VS Code, Cursor, Windsurf, PyCharm)             │
└─────────────────────────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Python Environment                          │
│              (uv / Poetry / Conda + virtual env)              │
└─────────────────────────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────────────────────────┐
│                    LLM Framework / Client                      │
│          (OpenAI SDK, Anthropic SDK, LangChain, etc.)         │
└─────────────────────────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────────────────────────┐
│                      LLM / Model Layer                         │
│              (Cloud API or local model runtime)               │
└─────────────────────────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Vector Database (optional)                  │
│          (Pinecone, Weaviate, Qdrant, Milvus, etc.)           │
└─────────────────────────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Application Code                          │
│              (API, services, agents, pipelines)               │
└─────────────────────────────────────────────────────────────────┘

The goal is not to use every tool available, but to select a coherent stack that supports your workflow, minimizes friction, and enables you to focus on building great LLM applications rather than fighting your environment.

Recommended Development Stack

The following table provides a high-level overview of the key categories in a modern LLM development stack, with vendor-neutral recommendations.

Category	Recommended Choices	Purpose
Operating System	macOS, Linux (Ubuntu), Windows + WSL2	Foundation for all development tools
Programming Language	Python (primary), TypeScript/JavaScript, Go, Rust	Application logic and integrations
Package / Environment Manager	uv (2026 default), Poetry, Conda, venv	Dependency management and isolation
IDE / Editor	VS Code, Cursor, Windsurf, PyCharm, JetBrains Air	Code editing, debugging, AI assistance
Version Control	Git + GitHub/GitLab/Bitbucket	Collaboration and history
Container Platform	Docker + Docker Compose	Reproducibility and deployment
LLM Client	OpenAI SDK, Anthropic SDK, LiteLLM, openai-compatible clients	API integration
Local Model Runtime	Ollama, LM Studio, llama.cpp, Docker Model Runner	Running models locally
Vector Database	Pinecone, Weaviate, Qdrant, Milvus, pgvector	Semantic search and RAG
API Testing	Postman, Insomnia, curl, HTTPie	Testing endpoints
Observability	OpenTelemetry, LangSmith, Arize Phoenix	Tracing, logging, evaluation
Notebooks	Jupyter, JupyterLab, Google Colab	Exploration and prototyping

Choosing an Operating System

Your choice of operating system shapes your entire development experience. Each option has distinct trade-offs.

macOS

Advantages:

• Unix-based terminal and tooling (similar to Linux)
• Excellent hardware (Apple Silicon M-series) with unified memory that benefits local LLM inference
• Native support for Docker Desktop
• Polished user experience and ecosystem

Limitations:

• Limited GPU options for local model training (though M-series chips are capable for inference)
• Higher cost for comparable hardware
• Some CUDA-dependent tools are not supported

Typical users: Developers who value polish, work across multiple platforms, or prefer macOS for general development.

Linux (Ubuntu / Debian)

Advantages:

• Full CUDA support for NVIDIA GPUs
• Best performance for local model inference and training
• Free and open source
• Docker runs natively with excellent performance
• Preferred for production deployments

Limitations:

• Steeper learning curve for newcomers
• Desktop experience less polished than macOS or Windows
• Driver and dependency management can be complex

Typical users: ML engineers, researchers, and developers targeting production deployments.

Windows + WSL2

Advantages:

• Familiar environment for many developers
• WSL2 provides a full Linux kernel and Ubuntu environment
• Can run both Windows and Linux tools
• Good GPU passthrough for CUDA via WSL2

Limitations:

• WSL2 adds complexity and overhead
• Some tools have Windows-specific quirks
• File system performance between Windows and WSL can be an issue

Typical users: Developers who primarily use Windows but need Linux tooling for AI development.

Recommendation

For LLM development, Linux (Ubuntu) is the most straightforward choice if you have NVIDIA hardware. macOS with Apple Silicon is excellent for development and inference but less suitable for training. Windows + WSL2 is a viable alternative if you are already in the Windows ecosystem. Choose based on your hardware, team standards, and personal preference.

Programming Languages for LLM Development

Language	Use Case	Strengths	Weaknesses
Python	Primary development, prototyping, ML	Largest ecosystem, all major frameworks, easy to learn	Performance, async complexity
TypeScript/JavaScript	Web UIs, API backends, full-stack	Ubiquitous in web, good async support	Limited ML libraries, less mature for AI
Go	High-performance services, APIs	Fast, concurrent, simple deployment	Smaller AI ecosystem
Rust	Performance-critical components	Memory-safe, blazing fast	Steep learning curve, small ecosystem
Java / Kotlin	Enterprise integrations	Mature, large ecosystem	Verbose, slower iteration

Python dominates the LLM ecosystem—all major frameworks (PyTorch, Transformers, LangChain, etc.) are Python-first, and most model providers offer Python SDKs. However, production systems often use a polyglot approach: Python for model orchestration, and Go, Rust, or TypeScript for high-performance API layers.

Python Environment Setup

Python environment management is one of the most important—and most frequently misunderstood—aspects of LLM development. The landscape has evolved significantly, with uv emerging as the default choice for new projects in 2026.

Python Version Selection

Use Python 3.11 or newer. Check your project's dependencies for compatibility. Many LLM libraries require Python 3.9+.

Virtual Environments

A virtual environment isolates your project's dependencies from the system Python and other projects. Without isolation, you risk dependency conflicts that are notoriously difficult to debug.

Tool Comparison

Tool	Best For	Key Characteristics
uv	New projects in 2026	Single Rust binary, 10-100x faster than pip, replaces pip, venv, virtualenv, pyenv, pip-tools, and most of pipx and Poetry
Poetry	Existing projects, mature lockfile	Polished UX, deterministic dependencies, healthy maintenance
Conda / Mamba	Projects with non-Python native dependencies	Handles C/C++ libraries, R, etc.
venv	Fallback when nothing else is available	Ships with Python 3.3+, zero setup
Pipenv	Maintenance mode	Not recommended for new projects

Setting Up with uv

For new projects in 2026, uv is the recommended default:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a new project
uv init my-llm-project
cd my-llm-project

# Create a virtual environment and install dependencies
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
uv add openai python-dotenv pydantic

uv manages Python interpreters, creates virtual environments, resolves dependencies, and installs packages—all in one fast Rust-based tool.

Choosing an IDE

Your IDE is where you spend most of your development time. The right choice dramatically improves productivity, especially with AI-powered coding features.

VS Code

The most popular editor for Python development. Free, highly extensible, with excellent Python support via the Python extension and Pylance. Works on all platforms.

Key extensions for LLM development:

• Python (Microsoft)
• Pylance
• GitLens
• Docker
• Ruff (for linting and formatting)
• Jupyter

Cursor

A VS Code fork focused on AI-powered development. Emphasizes control: AI-generated edits are shown as reviewable diffs before being applied, guided by explicit rules.

Best for: Developers who want AI assistance but prefer to review changes before they are written to files. Solo developers and fast iteration on Python projects.

Pricing: Free tier (2,000 completions + 50 slow requests/month); Pro at $20/month.

Windsurf

Another VS Code fork that applies edits directly in the editor while using broader workspace context (terminal output, recent edits, conversation history). Built on familiar VS Code architecture, preserving muscle memory and keybindings.

Best for: Developers who prefer the agent to apply edits directly as it works, drawing on broader workspace context. Large enterprise monorepos and multi-module architectures.

Pricing: Free tier (25 prompt credits/month); Pro at $15/month.

PyCharm / IntelliJ IDEA

Full-featured IDEs with excellent Python support. JetBrains Air offers agentic development capabilities. More resource-intensive than VS Code but provides deeper code analysis and refactoring tools.

JetBrains Air

An Agentic Development Environment where you delegate coding tasks to AI agents while staying in control of the workflow.

Recommendation

Start with VS Code—it is free, widely used, and has excellent Python support. If you want deeper AI integration, try Cursor (if you prefer control) or Windsurf (if you prefer flow). For larger projects, PyCharm Professional offers superior code intelligence.

Version Control

Git is non-negotiable for any serious development project. It tracks changes, enables collaboration, and provides a safety net for experimentation.

Essential Git Practices

Repository organization:

• One repository per project or service
• Use a .gitignore that excludes virtual environments, cache files, and secrets
• Include a README.md with setup instructions

Branching strategy:

• main or master for production-ready code
• develop for integration (optional)
• Feature branches: feature/your-feature-name
• Bugfix branches: fix/issue-description

Secrets management:

• Never commit API keys, passwords, or other secrets
• Use .env files with python-dotenv for local development
• Add .env to .gitignore
• Use environment variables in production (e.g., via CI/CD secrets)

Commit messages: Write clear, descriptive commit messages in the imperative mood ("Add retry logic for API calls" rather than "Added retry").

Working with LLM APIs

Most LLM development starts with cloud APIs—OpenAI, Anthropic, Google, or open-weight providers via platforms like Together AI or Replicate.

API Keys and Environment Variables

# .env file (never commit this!)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
MODEL_NAME=gpt-4o

# In your code (using python-dotenv)
import os
from dotenv import load_dotenv

load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")

API Client Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      Application Code                          │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Model Client Abstraction                    │
│              (Unified interface for multiple providers)        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Provider-Specific Client                     │
│              (OpenAI SDK, Anthropic SDK, LiteLLM)             │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                        LLM Provider API                        │
│              (OpenAI, Anthropic, Together, etc.)              │
└─────────────────────────────────────────────────────────────────┘

Key Considerations

Rate limits: All cloud providers impose rate limits. Implement retry logic with exponential backoff. Track your usage to avoid surprises.

Model selection: Different models excel at different tasks. Use environment variables or configuration to switch models without code changes.

Cost monitoring: Track token usage per request. Many LLM providers offer usage dashboards; consider logging token counts in your application.

Error handling: APIs can fail for many reasons—rate limits, timeouts, content filtering. Implement graceful fallbacks and user-friendly error messages.

Running Local Models

Running models locally offers privacy, cost predictability, and freedom from rate limits. The tooling ecosystem has matured significantly, making local inference a practical default for many workflows.

Hardware Requirements

For local model inference, VRAM is the single most important specification. A 7B parameter model at Q4_K_M quantization requires 8–16 GB of VRAM. System RAM of 16 GB is the bare minimum for running a 7B model alongside normal development tools. Model weights, KV-cache, and activation memory all compete for GPU memory.

Local Model Runtimes

Runtime	Description	Best For
Ollama	Single-command setup, OpenAI-compatible API	Easiest local inference
LM Studio	GUI installer, user-friendly	Beginners, GUI preference
llama.cpp	High-performance C++ implementation, quantization support	Advanced users, maximum performance
Docker Model Runner	Runs models in Docker containers, no API key needed	Docker-native workflows
vLLM	High-throughput serving, GPU-optimized	Production-grade serving

Getting Started with Ollama

Ollama is the most accessible local runtime:

# Install (macOS/Linux/Windows via WSL)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull qwen3:8b  # 8B parameter model

# Run and chat
ollama run qwen3:8b

# The model exposes an OpenAI-compatible API on localhost:11434

Switching from Cloud to Local

One of the key benefits of the current ecosystem is the standardized OpenAI-compatible API layer. Existing application code can switch from cloud to local inference by updating the base URL, API key placeholder, and model name string.

# Cloud
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Local (Ollama)
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # placeholder, not actually used
)

When to Use Local Models

• Prototyping and development before committing to a cloud provider
• Privacy-sensitive data that cannot leave your machine
• Cost-sensitive workloads with high inference volume
• Offline development (e.g., during travel)
• CI/CD pipelines where API costs would be prohibitive

Vector Database Setup

If you are building RAG applications, you will need a vector database to store and search embeddings. The choice of vector database depends on your scale, latency requirements, and infrastructure preferences.

When to Introduce a Vector Database

Add a vector database when:

• Your application requires semantic search over a document corpus
• You have more than a few thousand chunks to index
• You need metadata filtering or hybrid search
• You are moving beyond a proof-of-concept

Vector Database Options

Database	Description	Best For
Pinecone	Fully managed, cloud-native	Teams that don't want to manage infrastructure
Weaviate	Open-source, self-hosted or cloud	Flexibility and control
Qdrant	Open-source, high performance	High-scale, self-hosted
Milvus	Open-source, GPU-accelerated	Large-scale, performance-critical
pgvector	PostgreSQL extension	Teams already using PostgreSQL

Local Development Setup

For local development, you can run a vector database in Docker:

# docker-compose.yml
services:
  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"
    volumes:
      - qdrant_data:/qdrant/storage

Docker for LLM Development

Docker provides reproducible, isolated environments that work the same way on every machine. It is essential for production deployments and valuable for development.

Why Docker Matters

• Reproducibility: The same Docker image runs identically on your machine, your colleague's, and in production
• Dependency isolation: No conflicts with system packages or other projects
• Local testing: Run your entire stack (API, vector DB, model runner) locally
• Deployment consistency: The same container can be deployed to any cloud

Docker in the LLM Stack

┌─────────────────────────────────────────────────────────────────┐
│                     Docker Compose                             │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐           │
│  │   API       │  │  Vector DB  │  │  Model      │           │
│  │   Service   │  │  (Qdrant)   │  │  Runner     │           │
│  │  (FastAPI)  │  │             │  │ (Ollama)    │           │
│  └─────────────┘  └─────────────┘  └─────────────┘           │
└─────────────────────────────────────────────────────────────────┘

Docker Model Runner

Docker Model Runner (DMR) makes it easy to manage, run, and deploy AI models using Docker. Models run in Docker containers, so no API key is needed and no data leaves your computer. DMR supports pulling models from Docker Hub, OCI-compliant registries, or Hugging Face.

Basic Docker Setup

# Dockerfile for a FastAPI LLM application
FROM python:3.11-slim

WORKDIR /app

# Install uv for fast dependency management
COPY pyproject.toml uv.lock ./
RUN pip install uv && uv sync --frozen

COPY . .

CMD ["uv", "run", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

# docker-compose.yml
services:
  api:
    build: .
    ports:
      - "8000:8000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    volumes:
      - ./:/app

  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"

Project Structure

A well-organized project structure makes your code easier to understand, test, and maintain. The following layout is adapted from production-ready AI project templates.

my-llm-project/
│
├── src/                          # Source code
│   ├── core/                     # Core utilities
│   │   ├── config.py             # Configuration (pydantic-settings)
│   │   ├── logging.py            # Logging setup
│   │   └── utils.py              # Shared utilities
│   │
│   ├── clients/                  # External clients
│   │   ├── llm.py                # LLM client abstraction
│   │   └── vector_db.py          # Vector database client
│   │
│   ├── rag/                      # RAG-specific code
│   │   ├── chunking.py           # Document chunking
│   │   ├── indexing.py           # Index building
│   │   └── retrieval.py          # Retrieval logic
│   │
│   ├── agents/                   # Agent loops
│   │   ├── planner.py
│   │   └── executor.py
│   │
│   ├── workflows/                # Deterministic pipelines
│   │   └── rag_pipeline.py
│   │
│   ├── prompts/                  # Versioned prompts
│   │   ├── templates/
│   │   └── registry.py
│   │
│   └── api/                      # FastAPI endpoints
│       └── routes.py
│
├── tests/                        # Tests
│   ├── unit/
│   └── integration/
│
├── data/                         # Data files
│   ├── raw/
│   ├── processed/
│   └── external/
│
├── notebooks/                    # Jupyter notebooks
│   └── exploration.ipynb
│
├── scripts/                      # Utility scripts
│   └── ingest_data.py
│
├── docker/
│   └── Dockerfile
│
├── .env                          # Environment variables (gitignored)
├── .gitignore
├── pyproject.toml                # Dependencies (uv/poetry)
├── uv.lock / poetry.lock         # Lock file
├── Makefile                      # Common commands
├── README.md
└── AGENTS.md                     # Context for AI coding agents

Directory Explanations

• src/core/: Configuration, logging, and shared utilities that are used across the project
• src/clients/: Abstractions for external services (LLM providers, vector databases, etc.)
• src/rag/: RAG-specific components—chunking, indexing, retrieval
• src/agents/: LLM-driven loops that plan and execute actions
• src/workflows/: Deterministic pipelines that call LLMs on a fixed path
• src/prompts/: Versioned, testable prompt templates
• src/api/: FastAPI or similar entrypoints
• tests/: Unit and integration tests
• data/: Raw, processed, and external data
• notebooks/: Exploratory notebooks (numbered for ordering)
• scripts/: One-off utility scripts
• AGENTS.md: Context for AI coding agents—architecture decisions, deployment topology, project priorities

Configuration Management

Managing configuration across development, staging, and production environments is essential for reliability and security.

Environment Variables

Use environment variables for configuration that varies between environments:

# .env (development)
OPENAI_API_KEY=sk-...
MODEL_NAME=gpt-4o
LOG_LEVEL=DEBUG
VECTOR_DB_URL=http://localhost:6333

# Production (set via CI/CD or orchestration)
OPENAI_API_KEY=${PROD_OPENAI_API_KEY}
MODEL_NAME=gpt-4o
LOG_LEVEL=INFO
VECTOR_DB_URL=${PROD_VECTOR_DB_URL}

Configuration with Pydantic Settings

# src/core/config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    openai_api_key: str
    model_name: str = "gpt-4o"
    log_level: str = "INFO"
    vector_db_url: str = "http://localhost:6333"

    class Config:
        env_file = ".env"
        env_prefix = ""  # No prefix, matches env var names

settings = Settings()

Best Practices

• Never hardcode secrets in source code
• Use different .env files for different environments (.env.dev, .env.prod)
• Validate configuration at application startup
• Use a configuration schema to document expected variables
• Keep configuration out of version control (except for defaults)

Logging and Debugging

Observability should be built in from the beginning, not added as an afterthought.

What to Log

• Request logging: Each incoming request, including user ID and query
• Prompt logging: The prompts sent to LLMs (for debugging and evaluation)
• Token usage: Number of input and output tokens per request
• Latency: Time taken for each operation (embedding, retrieval, generation)
• Errors: All exceptions with full stack traces
• Retrieval results: Which chunks were retrieved and their scores

Structured Logging

Use structured logging (JSON format) for easier parsing and analysis:

import structlog

logger = structlog.get_logger()

logger.info(
    "llm_request",
    model=model_name,
    input_tokens=input_tokens,
    output_tokens=output_tokens,
    latency_ms=latency_ms,
    user_id=user_id,
)

Tracing

For distributed debugging, consider OpenTelemetry or a specialized LLM observability platform like LangSmith or Arize Phoenix. These tools provide end-to-end tracing of LLM calls, retrieval steps, and agent decisions.

Testing LLM Applications

Testing LLM applications is challenging because outputs are non-deterministic. However, a structured approach to testing is essential for production quality.

Types of Tests

Unit tests: Test deterministic components (chunking, parsing, configuration). No LLM calls.

Prompt tests: Test that prompts produce expected outputs for known inputs. Use a small, consistent test set.

Regression tests: Ensure that changes don't break existing behavior. Compare outputs against a baseline.

Evaluation tests: Measure quality metrics (accuracy, relevance, faithfulness) on a held-out dataset.

Integration tests: Test the full pipeline with real (but controlled) API calls.

Testing Strategy

1. Start with a small golden dataset of queries and expected answers
2. Run evaluations offline against this dataset
3. Track metrics (exact match, semantic similarity, human evaluation)
4. Use the evaluation results to decide when to deploy changes

Tooling

• pytest for test discovery and execution
• pytest-cov for coverage reporting
• Custom evaluation harnesses for LLM-specific metrics

Typical Development Workflow

┌─────────────────────────────────────────────────────────────────┐
│ 1. Design                                                      │
│    Define requirements, choose architecture, select models    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 2. Implement                                                   │
│    Write code, prompts, and configurations                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 3. Prompt Testing                                              │
│    Test prompts on sample inputs, iterate                     │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 4. Evaluation                                                  │
│    Run against golden dataset, measure quality                │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 5. Debugging                                                   │
│    Use logs and traces to fix issues                          │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 6. Deployment                                                  │
│    Build container, deploy to staging, then production        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 7. Monitoring                                                  │
│    Track latency, costs, error rates, user feedback           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ 8. Iteration                                                   │
│    Use production data to improve prompts, models, and code   │
└─────────────────────────────────────────────────────────────────┘

This workflow is iterative, not linear. You will frequently loop back from later stages to earlier ones as you learn from testing, evaluation, and production data.

Common Beginner Mistakes

1. Installing Packages Globally

Installing packages with pip install without a virtual environment leads to dependency conflicts across projects. Fix: Always use a virtual environment (uv, poetry, or venv).

2. Hardcoding API Keys

Committing API keys to version control is a security risk. Fix: Use .env files and environment variables.

3. Ignoring Virtual Environments

Skipping virtual environments because they seem like extra work leads to "it works on my machine" problems. Fix: Use uv—it makes environment management fast and painless.

4. Poor Project Organization

Throwing all code into a single file or directory makes the project hard to navigate and test. Fix: Use a consistent project structure from day one.

5. Skipping Version Control

Not using Git from the start means losing history and making collaboration difficult. Fix: Initialize a Git repository on day one.

6. Excessive Dependencies

Adding every library you might need creates bloat and dependency conflicts. Fix: Start with a minimal set of dependencies and add more only when needed.

7. Not Monitoring API Costs

LLM API costs can surprise you. Fix: Log token usage and set up budget alerts.

8. Mixing Development and Production Configurations

Using the same configuration for development and production leads to accidental production issues. Fix: Use separate configuration files or environment variables for each environment.

Production Best Practices

1. Isolate Environments

Use separate virtual environments for each project. Use Docker for production parity.

2. Automate Dependency Management

Use a lock file (uv.lock or poetry.lock) to ensure reproducible installations.

3. Containerize Applications

Use Docker for all services. It ensures consistency across environments and simplifies deployment.

4. Secure API Credentials

Never commit secrets. Use environment variables or secrets management services.

5. Maintain Reproducible Builds

Document every step of your setup. Use Dockerfiles and dependency lock files.

6. Use Version Control Effectively

Commit regularly, write meaningful commit messages, and use feature branches.

7. Monitor Costs

Track token usage and set up alerts for budget thresholds.

8. Document Project Setup

Maintain a README with setup instructions, dependencies, and common commands.

9. Implement CI/CD

Automate testing and deployment to catch issues early.

10. Start Small, Iterate

Begin with a minimal viable setup and add complexity as needed.

Interview Questions

1. What are the essential tools for an LLM development environment?

Answer: The essential tools include Python with a virtual environment manager (uv, Poetry, or venv), an IDE (VS Code, Cursor, or PyCharm), Git for version control, Docker for containerization, and an LLM client library (OpenAI SDK, Anthropic SDK, or LiteLLM). For RAG applications, a vector database is also essential. The specific choices depend on the project requirements and team preferences.

2. Why are virtual environments important in LLM development?

Answer: Virtual environments isolate project dependencies from the system Python and other projects. This prevents dependency conflicts, ensures reproducibility, and makes it easier to share projects with others. In 2026, uv is the recommended choice because it is 10-100x faster than pip and replaces multiple legacy tools in a single binary.

3. Why is Docker important for LLM development?

Answer: Docker provides reproducible, isolated environments that work consistently across development, staging, and production. It eliminates "it works on my machine" problems, simplifies dependency management, and makes deployment straightforward. Docker Model Runner also enables running LLMs locally without API keys.

4. How do you securely manage API keys in an LLM project?

Answer: API keys should never be hardcoded in source code or committed to version control. Use .env files with python-dotenv for local development, and add .env to .gitignore. In production, use environment variables set via CI/CD, orchestration platforms, or secrets management services.

5. Why is Git essential for LLM development?

Answer: Git tracks changes, enables collaboration, provides a safety net for experimentation, and maintains a history of the project. It is essential for any serious development project, regardless of the domain.

6. When should a vector database be introduced into a project?

Answer: A vector database should be introduced when the application requires semantic search over a document corpus, typically when there are more than a few thousand chunks to index. It is essential for RAG applications and becomes necessary as the project moves beyond a proof-of-concept.

7. What IDEs are commonly used for LLM development and what are their trade-offs?

Answer: VS Code is the most popular, free, and extensible. Cursor and Windsurf are VS Code forks with AI-native features—Cursor emphasizes control with reviewable diffs, while Windsurf emphasizes flow with direct edits. PyCharm offers deeper code analysis for larger projects. JetBrains Air provides agentic development capabilities.

8. Why is Python dominant in AI development?

Answer: Python dominates because all major AI frameworks (PyTorch, Transformers, LangChain, etc.) are Python-first, most model providers offer Python SDKs, and the ecosystem is the largest and most mature. However, production systems often use other languages (Go, Rust, TypeScript) for performance-critical components.

9. What does a reproducible development environment mean and why is it important?

Answer: A reproducible development environment means that the same setup produces the same results on any machine. It is achieved through virtual environments, dependency lock files, Docker containers, and documented setup procedures. Reproducibility is critical for collaboration, debugging, and production deployments.

10. How would you organize an LLM project for long-term maintainability?

Answer: Use a consistent project structure with separate directories for core utilities, clients, RAG components, agents, workflows, prompts, and APIs. Use a configuration management system (pydantic-settings) with environment variables. Use version control, virtual environments, and Docker from day one. Document the setup in a README and provide context for AI coding agents in an AGENTS.md file.

Best Practices Checklist

#	Practice	Description
1	Install a supported Python version	Use Python 3.11 or newer; check dependency compatibility.
2	Use isolated virtual environments	Use uv, Poetry, or venv to isolate project dependencies.
3	Manage dependencies properly	Use a lock file (uv.lock or poetry.lock) for reproducibility.
4	Secure API credentials	Use .env files for local development; never commit secrets.
5	Use Git from day one	Initialize a repository, use feature branches, and write meaningful commit messages.
6	Containerize applications with Docker	Use Docker for consistency across environments and easy deployment.
7	Organize projects consistently	Use a clear directory structure with separate concerns.
8	Log requests and token usage	Enable observability from the start for debugging and cost tracking.
9	Monitor API costs	Track token usage and set up budget alerts.
10	Document environment setup	Maintain a README with setup instructions for new team members.
11	Implement CI/CD	Automate testing and deployment to catch issues early.
12	Test with a golden dataset	Use a held-out evaluation set to measure quality before deployment.

Related Articles

• What Is an LLM?
• LLM Learning Roadmap
• Choosing the Right LLM Stack for Your Project
• Build Your First LLM Application
• What Is Prompt Engineering?
• What Is RAG?
• What Is Fine-Tuning?
• What Is LLMOps?
• LLM Security Overview

Key Takeaways

• A well-designed development environment improves productivity, reliability, and the ability to scale from prototype to production.
• Python remains the primary language for LLM development, with uv emerging as the default environment manager for new projects in 2026.
• Version control, virtual environments, and Docker form the foundation of modern AI engineering—use them from day one.
• AI-native IDEs like Cursor and Windsurf can dramatically accelerate development, each with different trade-offs in control versus flow.
• Vector databases become essential when building RAG applications with more than a few thousand documents.
• Proper configuration management, logging, testing, and observability should be incorporated from the beginning, not added as an afterthought.
• Local model inference is increasingly practical, with runtimes like Ollama and Docker Model Runner reducing setup to a single command.
• A reproducible development workflow—documented, automated, and consistent—is critical for team collaboration and production success.

---

This article is part of the LLMDevPro Getting Started Handbook — your guide to building production-grade LLM applications.