The Problem
You have a research agent that works beautifully on your laptop. It calls the Anthropic API, runs tool loops, and returns a detailed summary. You run it in a terminal and it works every time. Now a teammate wants to use it. Then a product manager wants to embed it in a dashboard. Then you want to run it in production for real users.
None of that is possible with a Python script that you run in a terminal. You need an HTTP endpoint. You need it to handle concurrent requests. You need it to stream progress back to the client in real time — because the agent takes 30 seconds and users will not stare at a spinner. And you need to ship it as a container so it runs identically in staging, production, and every engineer's machine.
That's the gap this lesson bridges. FastAPI turns your agent into an HTTP service. Docker packages that service into a container that runs anywhere. Uvicorn runs the service in production with the right concurrency model for agents. Together, these three components are the standard deployment stack for production Python AI services.
The gap between "it works in my terminal" and "it's deployed and serving requests" is not a devops problem — it's an architecture decision you make when you write the API layer and the Dockerfile. Make those decisions wrong and you'll be debugging production timeouts and blocking event loops for weeks.
Async is not optional for LLM services
LLM API calls take 5-60 seconds. If your endpoint is def instead of async def, every in-flight request blocks a thread. A single synchronous endpoint with 4 uvicorn workers can serve exactly 4 concurrent users before requests start queuing. With async def, a single worker handles hundreds of concurrent requests — waiting on I/O costs nothing while other requests are served. Async is not an optimization for agents; it's a correctness requirement.
FastAPI for agent endpoints
FastAPI is the right framework for agent services. It's async-first, generates OpenAPI docs automatically, validates requests with Pydantic, and has native StreamingResponse for SSE.
The request and response models:
from pydantic import BaseModel
class AgentRequest(BaseModel):
user_input: str
session_id: str | None = None
max_tokens: int = 4096
class AgentResponse(BaseModel):
output: str
steps: int
model: str
usage: dict
Pydantic validates every incoming request. If user_input is missing, FastAPI returns a 422 Unprocessable Entity before your agent code even runs. No manual validation needed.
A basic POST /agent/run endpoint:
@app.post("/agent/run", response_model=AgentResponse)
async def run_agent(request: AgentRequest):
result = await agent.run(request.user_input)
return AgentResponse(
output=result.output,
steps=result.step_count,
model=result.model,
usage=result.usage,
)
The async def is critical. The Anthropic SDK's async client (AsyncAnthropic) suspends the coroutine while the API call is in flight — the uvicorn event loop services other requests during the wait. No thread is blocked.
Streaming with SSE:
Most agent tasks take long enough that users need streaming. Use StreamingResponse with a generator that yields SSE-formatted strings:
from fastapi.responses import StreamingResponse
@app.post("/agent/run/stream")
async def run_agent_stream(request: AgentRequest):
async def event_generator():
async with client.messages.stream(
model="claude-opus-4-5",
max_tokens=request.max_tokens,
messages=[{"role": "user", "content": request.user_input}],
) as stream:
async for text in stream.text_stream:
data = json.dumps({"type": "text_delta", "text": text})
yield f"data: {data}\n\n"
yield "data: {\"type\": \"done\"}\n\n"
return StreamingResponse(
event_generator(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
The X-Accel-Buffering: no header tells nginx (if you put it in front) not to buffer the stream — critical for SSE to reach the client in real time.
Background tasks for fire-and-forget:
For long jobs where you want to return a job ID immediately and let the agent run asynchronously:
from fastapi import BackgroundTasks
@app.post("/agent/run/async")
async def run_agent_async(request: AgentRequest, background_tasks: BackgroundTasks):
job_id = str(uuid.uuid4())
background_tasks.add_task(run_agent_job, job_id, request.user_input)
return {"job_id": job_id, "status": "queued"}
Health check:
@app.get("/health")
async def health():
return {"status": "ok", "model": "claude-opus-4-5", "timestamp": time.time()}
Health checks are used by Docker, Kubernetes, and load balancers to decide whether to send traffic to your container. If /health returns 200, traffic flows. If it returns 500, the container is taken out of rotation.
Your agent endpoint is def run_agent(request: AgentRequest) (not async). It calls the Anthropic API which takes 20 seconds. With 4 uvicorn workers, the maximum number of concurrent users before requests start queuing is:
Explanation: A synchronous def endpoint in uvicorn runs in a thread pool. Each in-flight request occupies one thread for its entire duration — 20 seconds for a 20-second LLM call. With 4 workers and sync endpoints, you have exactly 4 threads available, so only 4 requests can run simultaneously. The 5th user waits until one finishes. With async def and the async Anthropic client, a single worker can handle hundreds of concurrent requests — the event loop suspends the coroutine while waiting for the API, freeing it to serve other requests. Async is not optional for services that call LLMs.
Docker containerization
Docker makes your service portable and reproducible. The same container that runs on your laptop runs in production. No "works on my machine" problems.
Multi-stage Dockerfile:
# Stage 1: builder — install dependencies
FROM python:3.12-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
# Stage 2: runtime — minimal image, no build tools
FROM python:3.12-slim AS runtime
WORKDIR /app
# Copy installed packages from builder (not pip, not build tools)
COPY --from=builder /install /usr/local
# Copy only application code
COPY main.py .
# Non-root user for security
RUN adduser --disabled-password --gecos "" appuser
USER appuser
EXPOSE 8080
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
Why multi-stage? The builder stage installs pip, compiles C extensions, and touches dozens of temporary files. None of that belongs in the production image. The runtime stage only copies the compiled packages and your application code. Result: a 200MB image instead of 800MB, no build tools in production that an attacker could exploit, and faster pulls.
The .dockerignore file — never skip this:
.env
.env.*
__pycache__
*.pyc
.git
.gitignore
.DS_Store
*.egg-info
dist/
.venv/
venv/
node_modules/
Without .dockerignore, COPY . . copies your .env file (with your ANTHROPIC_API_KEY) into the image. That key is now baked into a Docker layer, visible to anyone who pulls the image. .dockerignore prevents this.
Environment variables vs. secrets:
Never put secrets in the Dockerfile with ENV ANTHROPIC_API_KEY=sk-.... That bakes the key into an image layer that is visible in docker history. Instead, pass secrets at runtime:
# Development: read from a local .env file
docker run -p 8080:8080 --env-file .env research-agent
# Production: inject from your secrets manager
docker run -p 8080:8080 \
-e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
research-agent
Your Dockerfile should only have non-secret configuration — port numbers, log levels, worker counts:
ENV PORT=8080
ENV LOG_LEVEL=info
ENV WORKERS=1
Build and run:
# Build the image
docker build -t research-agent .
# Run with environment variables from .env file
docker run -p 8080:8080 --env-file .env research-agent
# Test the health endpoint
curl http://localhost:8080/health
# Test the agent endpoint
curl -X POST http://localhost:8080/agent/run \
-H "Content-Type: application/json" \
-d '{"user_input": "What is transformer attention?"}'
Production considerations
Uvicorn workers: how many?
# Stateless agent (no shared in-process state): multiple workers safe
uvicorn main:app --host 0.0.0.0 --port 8080 --workers 4
# Stateful agent (in-memory session cache, shared queue): single worker only
uvicorn main:app --host 0.0.0.0 --port 8080 --workers 1
Multiple uvicorn workers are separate processes. They do not share memory. If your agent stores session state in a Python dict, worker 2 has no access to worker 1's dict — requests to the same session will hit different workers and lose state. Stateful agents must run with --workers 1 and persist state externally (Redis, database).
Timeout handling:
Two timeouts matter: the FastAPI response timeout (how long before the web server gives up on a request) and the Anthropic API timeout (how long before the SDK gives up on an API call). Set both:
# In the Anthropic client, set a hard timeout
client = AsyncAnthropic(timeout=60.0) # 60 seconds max per API call
# In uvicorn, set keepalive (not a hard request timeout — use a reverse proxy for that)
# nginx: proxy_read_timeout 120s;
Graceful shutdown:
When Docker sends SIGTERM (on docker stop or a deploy), uvicorn stops accepting new connections and waits for in-flight requests to complete before shutting down. The default wait is 5 seconds — short for a 30-second agent. Extend it:
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "30"]
This tells uvicorn to wait up to 30 seconds for in-flight requests before forcing shutdown. Users get their responses; the process exits cleanly.
Your .dockerignore file is missing. You run docker build -t myagent . with a COPY . . instruction in your Dockerfile. The build succeeds and the image is pushed to a container registry. What is the security consequence?
Explanation: COPY . . copies the entire build context — including .env — into the image layer. Docker layers are not encrypted; they are tarballs. Anyone with pull access to the image can run docker history myagent or extract the layer tarball and read the .env file, including the API key. The .dockerignore file is the only thing that prevents this. Always add .env and .env.* to .dockerignore before writing any Dockerfile with COPY . ..