Phase 6: Computer Use & Browser Agents · 55 min · Python · Anthropic SDK · base64
Vision + Action Loops
An agent that can't verify what it just did is flying blind.
Hiring signal: Visual reasoning in agent loops is the skill that enables QA automation, accessibility testing, and legacy system automation. Any company that does automated UI testing or needs to automate tasks on visual-only interfaces needs engineers who can build reliable vision + action loops.
What you will learn
- Build agents that reason about visual content in screenshots (charts, images, UI state)
- Implement UI state detection: identify interactive elements, error states, loading indicators
- Handle multi-step visual workflows with state verification at each step
The Problem
The agent screenshots a web dashboard. It sees a chart showing an anomaly. The text on the page doesn't describe the anomaly — it's in the chart's visual pattern. The agent needs to reason about what it sees in the image, not just what's in the text.
Or more practically: the agent clicks "Submit" on a form. Did it work? A spinner appeared, then a success message, then it disappeared. Or did a validation error pop up? Without visual verification at each step, the agent has no way to know — and will proceed with a false assumption that breaks the rest of the workflow.
Vision + action loops solve this by making visual state inspection a first-class part of every action cycle.
Vision Capabilities in Agent Loops
Passing screenshots to Claude uses image content blocks with base64-encoded PNG data:
import base64
def screenshot_to_content_block(png_bytes: bytes) -> dict:
b64 = base64.standard_b64encode(png_bytes).decode("utf-8")
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": b64,
}
}
# Pass to Claude along with a specific question
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
messages=[{
"role": "user",
"content": [
screenshot_to_content_block(png_bytes),
{"type": "text", "text": "What is the current UI state? Is there an error message visible? Is any button in a loading/disabled state?"}
]
}]
)
Key pattern: ask specific, structured questions about the screenshot — not "describe what you see." The model's answer will be more useful and more consistent when the question is concrete:
- "Is there an error state visible? If yes, what is the error message?"
- "Is any button in a loading or disabled state?"
- "What step in the form is currently active?"
- "Has the success message appeared?"
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers UI State Detection Patterns, Multi-Step Visual Verification, Build It, What to Practice — plus a hands-on lab, quiz, and project artifact.
Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.
Browse all courses · View pricing · DeVenture Academy