Phase 5: Modern LLM Engineering · 60 min · OpenAI Vision API · Claude Vision · Whisper
The Concept
How Vision-Language Models Work
A Vision-Language Model (VLM) extends an LLM with image understanding. The key insight is that images and text can live in the same representational space — if you can convert an image into the same kind of token sequence that text produces, the LLM's attention mechanism can reason over both modalities together.
The process works in three steps. First, a vision encoder (like CLIP or ViT) processes the image by splitting it into patches (typically 16×16 pixels), encoding each patch into a vector, and producing a sequence of visual tokens. A 1024×1024 image yields 64×64 = 4,096 patches, each becoming one visual token. Second, these visual tokens are projected into the LLM's embedding space using a learned projection layer. Third, the visual tokens are concatenated with text tokens and fed into the LLM, which attends to both through standard self-attention — the model doesn't know or care that some tokens came from an image and others from text.
This is why VLMs can answer "what's in this image?" — the visual tokens carry the image's semantic content, and the LLM's language capabilities can describe what those tokens represent. The same attention mechanism that lets a text model connect "the cat" to "sat on the mat" lets a VLM connect a visual token (a patch of fur) to the word "cat."
Image → [Vision Encoder] → Visual tokens → [Projection] → LLM embedding space
Text → [Tokenizer] → Text tokens ─────────────────→ LLM embedding space
Both → concatenated → [LLM with self-attention] → Response
A VLM processes an image as "visual tokens." How does a 1024×1024 image become tokens, and roughly how many tokens does it produce?
Vision encoders like ViT split the image into patches (typically 16×16 pixels). A 1024×1024 image yields 64×64 = 4,096 patches, each encoded into a token vector. These visual tokens are concatenated with text tokens and fed to the LLM, which attends to both modalities through self-attention.
The Major VLMs
| Model | Strengths | Context | Best For |
|---|
| GPT-4o | Strong reasoning + vision | 128k tokens | General multimodal, document analysis |
| Claude 3.5 Sonnet | Excellent chart/diagram understanding | 200k tokens | Documents, code screenshots, diagrams |
| Gemini 1.5 Pro | Native multimodal, video support | 1M+ tokens | Long video, many images, audio |
| Qwen-VL | Open-weight, good OCR | Variable | OCR, document parsing, self-hosting |
Sending Images to VLMs
Most APIs accept images as base64-encoded strings or URLs:
import base64
import httpx
def encode_image(path: str) -> str:
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode()
def analyze_image(image_path: str, prompt: str) -> str:
response = httpx.post(
"https://api.openai.com/v1/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json={
"model": "gpt-4o",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{"type": "image_url", "image_url": {
"url": f"data:image/jpeg;base64,{encode_image(image_path)}"
}}
]
}],
"max_tokens": 500
}
)
return response.json()["choices"][0]["message"]["content"]
Practical Patterns
1. Document Analysis (OCR+)
- Extract structured data from invoices, forms, receipts
- VLMs replace fragile OCR pipelines with single-model understanding
- Output structured JSON with field validation
2. Visual Question Answering
- "What's wrong with this circuit board?" → diagnostic reasoning
- "Is this food safe to eat?" → safety classification
- "Describe the architecture in this diagram" → understanding
3. Image Comparison
- "Are these two products the same?" → e-commerce dedup
- "What changed between these screenshots?" → QA testing
- "Does this match the description?" → listing verification
Audio Multimodal
# Whisper for speech-to-text — runs locally or via API
import whisper
model = whisper.load_model("base")
result = model.transcribe("meeting.wav")
print(result["text"]) # Full transcription with timestamps
Text-to-Speech (TTS): OpenAI TTS, ElevenLabs, or local with piper for edge deployment.
Cost Considerations
Token costs for images
Image tokens are expensive. A single high-res image can cost 1,000-7,000 tokens depending on resolution. Always resize images before sending. A 1920x1080 screenshot costs ~3x more than a 640x360 version — and the model often performs identically on the smaller image.
Image Preprocessing for VLMs
from PIL import Image
import io, base64
def prepare_image_for_vlm(
path: str, max_dim: int = 1024, quality: int = 85,
) -> str:
"""Resize and compress an image before sending to a VLM.
Dramatically reduces token cost with minimal quality loss."""
img = Image.open(path)
# Convert to RGB if needed (handles RGBA, palette, etc.)
if img.mode != "RGB":
img = img.convert("RGB")
# Resize maintaining aspect ratio — longest side = max_dim
w, h = img.size
if max(w, h) > max_dim:
scale = max_dim / max(w, h)
img = img.resize((int(w * scale), int(h * scale)), Image.LANCZOS)
# Compress to JPEG
buffer = io.BytesIO()
img.save(buffer, format="JPEG", quality=quality)
return base64.b64encode(buffer.getvalue()).decode()
# Cost comparison (GPT-4o vision):
# 1920x1080 original → ~3,400 tokens → $0.017/image
# 1024x576 resized → ~1,200 tokens → $0.006/image (65% savings)
# 512x288 resized → ~300 tokens → $0.002/image (88% savings)
# Rule: resize to the smallest dimension where the model still answers correctly
VLM Selection Guide
Use case → Recommended VLM → Why
──────────────────────────────────────────────────────────────────────
Document/invoice extraction → GPT-4o or Claude 3.5 → Structured output, OCR+
Chart/diagram understanding → Claude 3.5 Sonnet → Best at visual reasoning
Long video analysis → Gemini 1.5 Pro → Native video, 1M+ context
Many images at once → Gemini 1.5 Pro → Large context window
Self-hosted / private → Qwen-VL or Llama Vision → Open-weight, no API
OCR-only (no reasoning) → Tesseract + text LLM → Cheaper, no image tokens
Real-time / low latency → Local Qwen-VL → No network round-trip
Cost hierarchy (per image):
GPT-4o $0.01-0.03 │ Claude 3.5 $0.003-0.012 │ Gemini $0.001-0.004
Local $0 (GPU cost only) │ OCR+text $0.0001 (text tokens only)