Phase 6: Computer Use & Browser Agents · 50 min · Python · Anthropic SDK · computer_use_20241022 tool
Computer Use Fundamentals
Agents that can see and click open a fundamentally different design space.
Real-world application: Computer use is live in production at companies automating legacy workflows, QA testing, and RPA replacement. AI engineers who can work with computer use APIs command premium rates specifically because the skill requires both computer vision intuition and solid engineering discipline around sandboxing and safety.
What you will learn
- Explain the computer use perception-action loop: screenshot → reason → action → screenshot
- Identify the three action types: key, mouse, screenshot
- Evaluate when computer use is appropriate vs. when an API integration is better
The Problem
A customer asks "why did my subscription charge fail?" The support agent needs to: open the billing dashboard, navigate to the customer's account, check the payment method status, view the most recent failed charge event, and copy the error code. With computer use, an AI agent does all of this by seeing the screen and clicking — exactly as a human would — with no API integration required.
This is computer use's core promise: automate anything a human can do with a mouse and keyboard, regardless of whether an API exists.
The perception-action loop
Computer use agents operate on a tight feedback cycle:
- Perceive: take a screenshot of the current screen state
- Reason: the model analyzes the screenshot using vision capabilities — "I can see a login form with an Email field and Password field"
- Act: the model outputs a computer use action —
{type: "text", text: "user@example.com"} into the focused field - Repeat: take another screenshot to verify the action landed correctly
Each iteration is one tool call. Complex tasks require 10–50+ iterations. The model never "sees" the screen except through screenshots — it must navigate interfaces the same way a human would, reading labels, interpreting icons, and understanding layout.
The loop continues until the task is complete or the agent determines it cannot proceed.
Three action primitives
Every computer use action maps to one of three primitives:
Key: keyboard input — shortcuts, navigation, confirmation
{"type": "key", "key": "ctrl+c"}
{"type": "key", "key": "Return"}
{"type": "key", "key": "Tab"}
Mouse: pointer control — move, click variants, scroll
{"type": "mouse_move", "coordinate": [100, 200]}
{"type": "left_click", "coordinate": [100, 200]}
{"type": "right_click", "coordinate": [100, 200]}
{"type": "double_click", "coordinate": [100, 200]}
{"type": "scroll", "coordinate": [100, 200], "direction": "down", "amount": 3}
Type text: insert text at the current focus
{"type": "type", "text": "hello world"}
All mouse actions use pixel coordinates derived from the current screenshot. The model reads the image, estimates coordinates for visible UI elements, and acts. This coordinate-based approach is both computer use's strength (universal) and its main fragility (brittle if resolution changes).
An AI needs to submit a form on a legacy internal web application that has no API and won't get one. What's the right approach?
This is computer use's primary value proposition. Legacy internal applications with no API, no maintainer, and no roadmap for modernization are common in every large organization. Computer use lets AI automate them without any code changes to the target system — the agent operates through the visual interface the humans use.
Computer use vs. API integration
Not every automation task should use computer use. The decision rule is simple:
| Situation | Use |
|---|
| No API exists (legacy system, vendor lock-in) | Computer use |
| API exists but is expensive or rate-limited | Computer use (sometimes) |
| Complex UI that would take weeks to reverse-engineer | Computer use |
| Reliable API available | API integration — always prefer this |
| Speed is critical | API integration |
API integration wins when one exists: it's 10–100× faster, has no coordinate fragility, costs no screenshot tokens, and is far more maintainable. Computer use is the tool of last resort with the broadest possible reach.
The coordinate fragility issue is real: pixel coordinates depend on screen resolution, window size, zoom level, and UI rendering. Best practices for managing this:
- Use a fixed resolution sandbox — 1024×768 is the standard, chosen because it reduces screenshot token cost while remaining functional for most UIs
- Always take a fresh screenshot before clicking — never cache coordinates between actions
- Describe clicks by text labels, not pixel offsets — let the model derive coordinates from a fresh screenshot
A computer use action uses coordinate [x: 100, y: 200] to click a button. The screen resolution changes. What happens?
This is computer use's primary fragility. Pixel coordinates are brittle — they depend on screen resolution, window size, zoom level, and UI rendering. Best practices: (1) Use a fixed resolution sandbox. (2) Always take a fresh screenshot before clicking — never cache coordinates. (3) Look for text labels, not pixel offsets, when describing where to click.
Build
Build it
The following implementation sets up the Anthropic computer use API with the computer_use_20241022 tool, implements the screenshot-action loop, and provides a ScreenState class that manages the current screenshot and element positions.
"""
lesson_06_01_computer_use_intro.py
Computer use fundamentals: screenshot-action loop using the Anthropic API.
Demonstrates the perception-action loop with the computer_use_20241022 tool.
"""
import os
import base64
import json
import time
from dataclasses import dataclass, field
from typing import Any
import anthropic
# ---------------------------------------------------------------------------
# Screen state management
# ---------------------------------------------------------------------------
@dataclass
class ScreenState:
"""Holds the current screenshot and parsed element positions."""
screenshot_b64: str = ""
width: int = 1024
height: int = 768
last_action: dict[str, Any] | None = None
action_count: int = 0
token_usage: dict[str, int] = field(default_factory=lambda: {"input": 0, "output": 0})
def from_file(self, path: str) -> "ScreenState":
"""Load a screenshot from a PNG file."""
with open(path, "rb") as f:
self.screenshot_b64 = base64.standard_b64encode(f.read()).decode("utf-8")
return self
def from_bytes(self, data: bytes) -> "ScreenState":
"""Load a screenshot from raw bytes."""
self.screenshot_b64 = base64.standard_b64encode(data).decode("utf-8")
return self
def as_image_block(self) -> dict[str, Any]:
"""Return the screenshot as an Anthropic image content block."""
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": self.screenshot_b64,
},
}
# ---------------------------------------------------------------------------
# Computer use agent
# ---------------------------------------------------------------------------
class ComputerUseAgent:
"""
Drives the screenshot → reason → act loop using the Anthropic
computer_use_20241022 tool.
"""
TOOL_DEFINITION = {
"type": "computer_20241022",
"name": "computer",
"display_width_px": 1024,
"display_height_px": 768,
"display_number": 1,
}
def __init__(self, task: str, max_iterations: int = 20):
self.client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
self.task = task
self.max_iterations = max_iterations
self.state = ScreenState()
self.action_log: list[dict[str, Any]] = []
def _build_messages(self) -> list[dict[str, Any]]:
"""Build the message list for the current iteration."""
messages = [
{
"role": "user",
"content": [
self.state.as_image_block(),
{"type": "text", "text": f"Task: {self.task}\n\nComplete the task using the computer tool. Take a fresh screenshot before each click to ensure coordinates are current."},
],
}
]
return messages
def _execute_action(self, action: dict[str, Any]) -> None:
"""
Execute a computer use action.
In a real implementation this would drive xdotool, pyautogui, or
a Playwright/Selenium browser. Here we log the action and simulate.
"""
self.state.last_action = action
self.state.action_count += 1
self.action_log.append({
"iteration": self.state.action_count,
"action": action,
"timestamp": time.time(),
})
action_type = action.get("type", "unknown")
print(f" [ACTION {self.state.action_count}] {action_type}: {json.dumps(action)}")
def _take_screenshot(self) -> None:
"""
Capture the current screen state.
In a real deployment this calls xwd, scrot, or Playwright's
page.screenshot(). Here we keep the existing state as a no-op.
"""
print(" [SCREENSHOT] Capturing current screen state...")
def run(self, initial_screenshot_path: str | None = None) -> dict[str, Any]:
"""
Run the perception-action loop until the task is complete
or max_iterations is reached.
"""
if initial_screenshot_path:
self.state.from_file(initial_screenshot_path)
print(f"\nStarting computer use agent\nTask: {self.task}\n{'-'*60}")
for iteration in range(self.max_iterations):
print(f"\nIteration {iteration + 1}/{self.max_iterations}")
# Safety check: always take a fresh screenshot before reasoning
if iteration > 0:
self._take_screenshot()
messages = self._build_messages()
try:
response = self.client.beta.messages.create(
model="claude-opus-4-5",
max_tokens=4096,
tools=[self.TOOL_DEFINITION],
messages=messages,
betas=["computer-use-2024-10-22"],
)
except anthropic.APIError as e:
print(f" [ERROR] API call failed: {e}")
break
# Track token usage
self.state.token_usage["input"] += response.usage.input_tokens
self.state.token_usage["output"] += response.usage.output_tokens
# Process response content blocks
task_complete = False
for block in response.content:
if block.type == "text":
print(f" [REASONING] {block.text[:200]}...")
if any(phrase in block.text.lower() for phrase in
["task complete", "successfully", "finished", "done"]):
task_complete = True
elif block.type == "tool_use" and block.name == "computer":
self._execute_action(block.input)
if response.stop_reason == "end_turn" or task_complete:
print("\nTask complete.")
break
return {
"actions": self.action_log,
"total_actions": self.state.action_count,
"token_usage": self.state.token_usage,
"estimated_cost_usd": (
self.state.token_usage["input"] * 3e-6 +
self.state.token_usage["output"] * 15e-6
),
}
# ---------------------------------------------------------------------------
# Demo
# ---------------------------------------------------------------------------
if __name__ == "__main__":
# Demonstrate the computer use agent setup and action logging.
# In production, provide a real screenshot and connect action execution
# to xdotool, pyautogui, or a Playwright browser instance.
agent = ComputerUseAgent(
task="Navigate to the login form, enter 'user@example.com' in the email field, enter 'password123' in the password field, and click the Sign In button.",
max_iterations=5,
)
# Show the tool definition that enables computer use
print("Computer use tool definition:")
print(json.dumps(ComputerUseAgent.TOOL_DEFINITION, indent=2))
print("\nAction primitive examples:")
actions = [
{"type": "screenshot"},
{"type": "left_click", "coordinate": [400, 300]},
{"type": "type", "text": "user@example.com"},
{"type": "key", "key": "Tab"},
{"type": "type", "text": "password123"},
{"type": "key", "key": "Return"},
]
for action in actions:
print(f" {json.dumps(action)}")
print("\nNote: To run against a real screen, call agent.run() with a")
print("screenshot path and connect _execute_action() to your automation backend.")
print("\nScreenState + token budget tracking initialized.")
print(f"Max resolution: {agent.state.width}x{agent.state.height} (standard for cost/reliability)")
The action primitives above map naturally onto a simple state machine: a cursor position, a focused field, and a log of what happened. The tryit block below simulates that state machine in pure Python — no real screen involved — so you can see how an agent tracks cursor position and typed text across a sequence of actions.
class FakeScreen:
"""A tiny grid-based stand-in for a real screen the agent 'clicks' on."""
def __init__(self, width=10, height=10):
self.width = width
self.height = height
self.cursor = (0, 0)
self.focused_field = None
self.fields = {"email": "", "password": ""}
self.log = []
def click(self, x, y, field=None):
self.cursor = (x, y)
self.focused_field = field
self.log.append(f"click at ({x},{y}) -> focus={field}")
def type_text(self, text):
if self.focused_field is None:
self.log.append(f"type '{text}' ignored: no field focused")
return
self.fields[self.focused_field] += text
self.log.append(f"type '{text}' into {self.focused_field}")
screen = FakeScreen()
actions = [
("click", 2, 3, "email"),
("type", "user@example.com"),
("click", 2, 5, "password"),
("type", "hunter2"),
]
for action in actions:
if action[0] == "click":
_, x, y, field = action
screen.click(x, y, field)
else:
_, text = action
screen.type_text(text)
print("Action trace:")
for entry in screen.log:
print(f" {entry}")
print("\nFinal field values:")
for name, value in screen.fields.items():
print(f" {name}: {value}")
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy