Phase 6: Computer Use & Browser Agents · 55 min · Python · Anthropic SDK · base64
Vision + Action Loops
An agent that can't verify what it just did is flying blind.
Real-world application: Visual reasoning in agent loops is the skill that enables QA automation, accessibility testing, and legacy system automation. Any company that does automated UI testing or needs to automate tasks on visual-only interfaces needs engineers who can build reliable vision + action loops.
What you will learn
- Build agents that reason about visual content in screenshots (charts, images, UI state)
- Implement UI state detection: identify interactive elements, error states, loading indicators
- Handle multi-step visual workflows with state verification at each step
The Problem
The agent screenshots a web dashboard. It sees a chart showing an anomaly. The text on the page doesn't describe the anomaly — it's in the chart's visual pattern. The agent needs to reason about what it sees in the image, not just what's in the text.
Or more practically: the agent clicks "Submit" on a form. Did it work? A spinner appeared, then a success message, then it disappeared. Or did a validation error pop up? Without visual verification at each step, the agent has no way to know — and will proceed with a false assumption that breaks the rest of the workflow.
Vision + action loops solve this by making visual state inspection a first-class part of every action cycle.
Vision capabilities in agent loops
Passing screenshots to Claude uses image content blocks with base64-encoded PNG data:
import base64
def screenshot_to_content_block(png_bytes: bytes) -> dict:
b64 = base64.standard_b64encode(png_bytes).decode("utf-8")
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": b64,
}
}
# Pass to Claude along with a specific question
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
messages=[{
"role": "user",
"content": [
screenshot_to_content_block(png_bytes),
{"type": "text", "text": "What is the current UI state? Is there an error message visible? Is any button in a loading/disabled state?"}
]
}]
)
Key pattern: ask specific, structured questions about the screenshot — not "describe what you see." The model's answer will be more useful and more consistent when the question is concrete:
- "Is there an error state visible? If yes, what is the error message?"
- "Is any button in a loading or disabled state?"
- "What step in the form is currently active?"
- "Has the success message appeared?"
UI state detection patterns
Common patterns agents must detect:
| State | Visual signals | Action |
|---|
| Loading | Spinner, skeleton screen, disabled buttons, progress bar | Wait — poll screenshots until spinner disappears |
| Error | Red borders, alert banners, error icons, red text | Read error message, decide to retry or escalate |
| Success | Green check, confirmation message, redirect | Verify success, proceed to next step |
| Empty | "No results" message, empty table, placeholder text | Adjust query or handle gracefully |
| Ready | Form with unfilled fields, active buttons | Proceed with action |
Implement a UIStateDetector that classifies screenshots before every action:
from enum import Enum
class UIState(Enum):
LOADING = "loading"
ERROR = "error"
SUCCESS = "success"
EMPTY = "empty"
READY = "ready"
UNKNOWN = "unknown"
The detector passes the screenshot to Claude and asks it to classify. Never act on a UI in a LOADING state.
An agent clicks "Submit" on a form. How should it verify the submission succeeded?
In UI automation, silent failures are common — network timeouts, validation errors, session expiry, rate limits. The only way to verify success in computer use is visual verification: screenshot → analyze → confirm expected state. Assuming success without verification leads to downstream failures that are hard to debug.
Multi-step visual verification
For each action in a visual workflow, the pattern is:
- Act — execute the action (click, fill, navigate)
- Screenshot — capture the new screen state
- Verify — did the expected UI transition occur?
- Continue or retry — if verified, proceed; if not, retry or escalate
The VisualVerifier makes this concrete: "After clicking Submit, does this screenshot show a success state?" It uses the model to check the screenshot against a description of the expected post-action state.
An agent encounters a "loading" spinner in its screenshot. It should:
Clicking during a loading state is a race condition that often causes errors or unexpected behavior. The correct pattern is: detect loading state → wait → verify loaded → then act. In Playwright this is wait_for_selector("css=.spinner", state="hidden"). In computer use it's polling screenshots until the spinner is no longer visible. Never act on a UI in transition.
The tryit block below simulates the "detect state, wait if loading, verify after acting" loop from VisualWorkflow using a plain Python state machine instead of real screenshots — it shows how the loop handles a page that starts out loading, then becomes ready, then confirms success.
import itertools
# A scripted sequence of "screenshots" the fake screen will report, in order.
SCREEN_STATES = iter(["loading", "loading", "ready", "success"])
def take_fake_screenshot():
return next(SCREEN_STATES, "unknown")
def wait_for_ready(max_polls=5):
for attempt in range(1, max_polls + 1):
state = take_fake_screenshot()
print(f" poll {attempt}: state={state}")
if state != "loading":
return state
return "timeout"
print("Step 1: wait for the page to finish loading")
state = wait_for_ready()
print(f" -> resolved state: {state}")
print("\nStep 2: perform action (click Submit) and verify")
if state == "ready":
result_state = take_fake_screenshot()
verified = result_state == "success"
print(f" post-action state: {result_state}")
print(f" verified success: {verified}")
else:
print(" skipped action: page was not ready")
Build
Build it
The following implementation provides a ScreenshotAnalyzer, UIStateDetector, VisualVerifier, and a VisualWorkflow that ties them together into a resilient multi-step automation.
"""
lesson_06_03_vision_action.py
Vision + action loops: visual state detection and multi-step verification.
Requirements:
pip install anthropic
"""
import os
import base64
import json
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable
import anthropic
class UIState(Enum):
LOADING = "loading"
ERROR = "error"
SUCCESS = "success"
EMPTY = "empty"
READY = "ready"
UNKNOWN = "unknown"
@dataclass
class ScreenshotAnalysis:
state: UIState
error_message: str | None
active_step: str | None
confidence: str # "high" | "medium" | "low"
raw_response: str
class ScreenshotAnalyzer:
"""Pass screenshots to Claude with structured questions about UI state."""
def __init__(self, client: anthropic.Anthropic):
self.client = client
def _encode(self, png_bytes: bytes) -> dict[str, Any]:
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": base64.standard_b64encode(png_bytes).decode("utf-8"),
},
}
def analyze(self, png_bytes: bytes, question: str) -> str:
"""Ask Claude a specific question about a screenshot."""
response = self.client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
messages=[{
"role": "user",
"content": [
self._encode(png_bytes),
{"type": "text", "text": question},
],
}],
)
return response.content[0].text
def detect_state(self, png_bytes: bytes) -> ScreenshotAnalysis:
"""Classify the current UI state from a screenshot."""
question = """Analyze this screenshot and answer:
1. What is the current UI state? Choose one: loading, error, success, empty, ready
2. If there is an error, what does the error message say?
3. If there is a multi-step form, what step is currently active?
4. How confident are you: high, medium, or low?
Reply in JSON format:
{"state": "...", "error_message": null_or_string, "active_step": null_or_string, "confidence": "high|medium|low"}"""
response_text = self.analyze(png_bytes, question)
try:
# Extract JSON from response
start = response_text.find("{")
end = response_text.rfind("}") + 1
data = json.loads(response_text[start:end])
return ScreenshotAnalysis(
state=UIState(data.get("state", "unknown")),
error_message=data.get("error_message"),
active_step=data.get("active_step"),
confidence=data.get("confidence", "medium"),
raw_response=response_text,
)
except (json.JSONDecodeError, ValueError):
return ScreenshotAnalysis(
state=UIState.UNKNOWN,
error_message=None,
active_step=None,
confidence="low",
raw_response=response_text,
)
class UIStateDetector:
"""Detect and wait for the correct UI state before acting."""
def __init__(self, analyzer: ScreenshotAnalyzer, take_screenshot: Callable[[], bytes]):
self.analyzer = analyzer
self.take_screenshot = take_screenshot
def wait_for_ready(self, max_polls: int = 10, poll_interval: float = 1.0) -> ScreenshotAnalysis:
"""Poll screenshots until the UI is in a non-loading state."""
for attempt in range(max_polls):
png = self.take_screenshot()
analysis = self.analyzer.detect_state(png)
print(f" [State check {attempt+1}] {analysis.state.value} (confidence: {analysis.confidence})")
if analysis.state != UIState.LOADING:
return analysis
print(f" [Wait] Loading state detected, polling again in {poll_interval}s...")
time.sleep(poll_interval)
return ScreenshotAnalysis(
state=UIState.UNKNOWN,
error_message="Timeout: UI still loading after max polls",
active_step=None,
confidence="low",
raw_response="",
)
class VisualVerifier:
"""Verify that an action produced the expected UI transition."""
def __init__(self, analyzer: ScreenshotAnalyzer):
self.analyzer = analyzer
def verify(self, png_bytes: bytes, expected_description: str) -> tuple[bool, str]:
"""
Check if the screenshot matches the expected post-action state.
Returns (succeeded, explanation).
"""
question = f"""Does this screenshot show: "{expected_description}"?
Answer with:
- succeeded: true or false
- explanation: one sentence describing what you actually see
Reply in JSON: {{"succeeded": true/false, "explanation": "..."}}"""
response = self.analyzer.analyze(png_bytes, question)
try:
start = response.find("{")
end = response.rfind("}") + 1
data = json.loads(response[start:end])
return data.get("succeeded", False), data.get("explanation", response)
except (json.JSONDecodeError, ValueError):
return False, response
@dataclass
class WorkflowStep:
name: str
action: Callable[[], None]
expected_state_after: str
max_retries: int = 2
@dataclass
class WorkflowResult:
completed: bool
steps_passed: list[str]
steps_failed: list[str]
error: str | None
duration_seconds: float
class VisualWorkflow:
"""
Execute a multi-step workflow with visual verification at each step.
Detects loading states, verifies actions succeeded, and retries on failure.
"""
def __init__(
self,
detector: UIStateDetector,
verifier: VisualVerifier,
take_screenshot: Callable[[], bytes],
):
self.detector = detector
self.verifier = verifier
self.take_screenshot = take_screenshot
def run(self, steps: list[WorkflowStep]) -> WorkflowResult:
start = time.time()
passed: list[str] = []
failed: list[str] = []
for step in steps:
print(f"\n[Step] {step.name}")
# 1. Wait for UI to be ready
state = self.detector.wait_for_ready()
if state.state == UIState.ERROR:
error_msg = f"Error before step '{step.name}': {state.error_message}"
print(f" [FAILED] {error_msg}")
failed.append(step.name)
return WorkflowResult(
completed=False,
steps_passed=passed,
steps_failed=failed,
error=error_msg,
duration_seconds=time.time() - start,
)
# 2. Execute action (with retries)
succeeded = False
last_explanation = ""
for attempt in range(step.max_retries + 1):
if attempt > 0:
print(f" [Retry {attempt}] {step.name}")
step.action()
time.sleep(0.5) # Brief pause for UI to react
# 3. Verify the expected transition occurred
png = self.take_screenshot()
succeeded, last_explanation = self.verifier.verify(png, step.expected_state_after)
print(f" [Verify] {step.expected_state_after} → {'OK' if succeeded else 'FAILED'}")
print(f" [Detail] {last_explanation}")
if succeeded:
break
if succeeded:
passed.append(step.name)
else:
failed.append(step.name)
return WorkflowResult(
completed=False,
steps_passed=passed,
steps_failed=failed,
error=f"Step '{step.name}' failed after {step.max_retries+1} attempts: {last_explanation}",
duration_seconds=time.time() - start,
)
return WorkflowResult(
completed=True,
steps_passed=passed,
steps_failed=failed,
error=None,
duration_seconds=time.time() - start,
)
if __name__ == "__main__":
print("Vision + Action Loops — Lesson 06-03")
print("=" * 50)
print("\nUIState enum values:")
for state in UIState:
print(f" {state.value}")
print("\nScreenshotAnalysis fields:")
analysis = ScreenshotAnalysis(
state=UIState.READY,
error_message=None,
active_step="Step 1 of 3: Personal Information",
confidence="high",
raw_response='{"state": "ready", "active_step": "Step 1 of 3"}',
)
print(f" state: {analysis.state.value}")
print(f" active_step: {analysis.active_step}")
print(f" confidence: {analysis.confidence}")
print("\nWorkflowStep example:")
print(" WorkflowStep(")
print(" name='Fill email field',")
print(" action=lambda: page.fill('#email', 'user@example.com'),")
print(" expected_state_after='Email field shows user@example.com',")
print(" max_retries=2,")
print(" )")
print("\nTo run a full visual workflow:")
print(" Connect take_screenshot() to a real Playwright page or screen capture")
print(" Provide an ANTHROPIC_API_KEY")
print(" Build WorkflowStep list and call workflow.run(steps)")
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy