Phase 6: Computer Use & Browser Agents · 55 min · Python · Playwright · Anthropic SDK
Browser Automation with Playwright
DOM-level control is 10× faster and more reliable than clicking pixels.
Real-world application: Playwright is the standard for web automation in AI agent systems. Any company building AI features that involve web interactions — price comparison, research agents, form filling, QA automation — needs engineers who can wrap Playwright as clean agent tools. This shows in portfolios immediately.
What you will learn
- Automate web tasks using Playwright: navigation, form filling, clicking, data extraction
- Build an agent that uses Playwright tools to browse the web on behalf of the user
- Combine Playwright (structured web) with computer use (unstructured GUI) in one agent
The Problem
Computer use takes screenshots and clicks pixels — it's universal but expensive. Every page state requires a screenshot. Every click requires coordinate derivation. And coordinates break whenever the UI changes.
Playwright gives you DOM-level control: find elements by text, CSS selector, or ARIA role — no pixel coordinates. For web automation, Playwright is 10× faster and more reliable than computer use. The key engineering decision: choose the right tool for the interface type.
Playwright fundamentals for agent use
Playwright's Python API is async-first. Core operations:
# Navigation
await page.goto("https://example.com")
await page.wait_for_selector(".main-content")
# Finding and interacting with elements
await page.fill("#email-input", "user@example.com")
await page.click("text=Submit")
await page.select_option("select[name='country']", "US")
# Extraction
title = await page.title()
text = await page.text_content("h1")
links = await page.query_selector_all("a")
Selector hierarchy — prefer semantic over structural:
| Selector type | Example | Stability |
|---|
| Text | text=Submit | High — survives CSS refactors |
| ARIA label | [aria-label="Search"] | High — semantic meaning |
| Test ID | [data-testid="submit-btn"] | High — purpose-built for automation |
| CSS class | .btn-primary | Medium — breaks on redesigns |
| CSS positional | div > form > button:last-child | Low — breaks on restructure |
| Auto-generated ID | #btn-submit-92847 | Very low — changes on deploy |
Always prefer text=, [aria-label=], and [data-testid=] selectors. They're semantic, self-documenting, and survive page redesigns.
Playwright as agent tools
Wrap Playwright operations as tools the Claude API can call. Each tool has a clear contract: input parameters → side effect → return value.
tools = [
{
"name": "navigate_to",
"description": "Navigate the browser to a URL",
"input_schema": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "Full URL to navigate to"}
},
"required": ["url"]
}
},
{
"name": "click_element",
"description": "Click an element by text content or CSS selector",
"input_schema": {
"type": "object",
"properties": {
"selector": {"type": "string", "description": "Text content or CSS selector"}
},
"required": ["selector"]
}
},
{
"name": "fill_form_field",
"description": "Fill a form field identified by its label text",
"input_schema": {
"type": "object",
"properties": {
"label": {"type": "string"},
"value": {"type": "string"}
},
"required": ["label", "value"]
}
}
]
The agent calls these tools. The Playwright browser maintains state between calls — cookies, login sessions, navigation history all persist. This is what allows the agent to log in once and remain authenticated for subsequent tool calls.
An agent needs to click the "Submit" button on a form. Which Playwright selector is most robust?
CSS positional selectors and auto-generated IDs break when the UI changes. Text and ARIA selectors are semantic — they survive page redesigns and framework migrations because they reference what the element means, not where it is. For agent tools, prefer semantic selectors. They're also more readable: text=Submit is self-documenting.
Structured extraction patterns
A web page is 50–200KB of HTML. 90%+ is noise: scripts, styles, meta tags, tracking pixels. Returning raw HTML to the model wastes tokens and buries signal in noise.
Never return raw HTML. Always parse and extract first.
Extraction patterns:
# Get cleaned text content (no HTML tags)
content = await page.evaluate("() => document.body.innerText")
# Extract structured list items
items = await page.evaluate("""
() => Array.from(document.querySelectorAll('.product-card')).map(el => ({
title: el.querySelector('h3')?.textContent?.trim(),
price: el.querySelector('.price')?.textContent?.trim(),
url: el.querySelector('a')?.href,
}))
""")
# Get all links on the page
links = await page.evaluate("""
() => Array.from(document.querySelectorAll('a[href]')).map(a => ({
text: a.textContent.trim(),
href: a.href
})).filter(l => l.text)
""")
# Get form fields
fields = await page.evaluate("""
() => Array.from(document.querySelectorAll('input, textarea, select')).map(el => ({
name: el.name || el.id,
type: el.type,
label: document.querySelector(`label[for="${el.id}"]`)?.textContent?.trim()
}))
""")
Return to the model: title, main text, links, form fields. The model gets what it needs without irrelevant noise.
Why should Playwright tools return structured data rather than raw HTML to the model?
A typical web page is 50–200KB of HTML. 90%+ is irrelevant to the agent's task. Returning raw HTML to the model wastes tokens and buries the signal in noise. Clean extraction — title, main text content, list of links, form fields — gives the model precisely what it needs. This is a cost/quality decision that good agent engineers always make.
Before wiring up a real browser, it helps to see the selector-priority logic in isolation. The tryit block below picks the "best" selector for an element from a set of candidates, using the same stability ranking described above.
SELECTOR_PRIORITY = ["data-testid", "aria-label", "text", "css-class", "css-position", "auto-id"]
def best_selector(candidates):
"""Given a dict of {selector_type: selector_value}, pick the most stable one."""
for selector_type in SELECTOR_PRIORITY:
if selector_type in candidates:
return selector_type, candidates[selector_type]
return None, None
elements = [
{
"name": "Submit button",
"candidates": {"css-position": "div > form > button:last-child", "text": "Submit", "auto-id": "#btn-92847"},
},
{
"name": "Search input",
"candidates": {"css-class": ".search-box", "aria-label": "Search"},
},
{
"name": "Mystery div",
"candidates": {"css-position": "div:nth-child(3)"},
},
]
for el in elements:
selector_type, value = best_selector(el["candidates"])
print(f"{el['name']}: use {selector_type} -> '{value}'")
Build
Build it
Eight agent tools wrap the core Playwright operations. The BrowserAgent class handles tool dispatch and session state.
"""
lesson_06_02_playwright_agent.py
Browser agent using Playwright for DOM-level web automation.
Wraps Playwright as 8 clean agent tools callable by Claude.
Requirements:
pip install anthropic playwright
playwright install chromium
"""
import os
import json
import asyncio
from typing import Any
import anthropic
from playwright.async_api import async_playwright, Page, Browser, BrowserContext
# ---------------------------------------------------------------------------
# Playwright browser manager
# ---------------------------------------------------------------------------
class BrowserSession:
"""Manages a persistent Playwright browser session."""
def __init__(self, headless: bool = True):
self.headless = headless
self._playwright = None
self._browser: Browser | None = None
self._context: BrowserContext | None = None
self.page: Page | None = None
async def start(self) -> None:
self._playwright = await async_playwright().start()
self._browser = await self._playwright.chromium.launch(headless=self.headless)
self._context = await self._browser.new_context(
viewport={"width": 1280, "height": 720},
user_agent="Mozilla/5.0 (compatible; ResearchAgent/1.0)",
)
self.page = await self._context.new_page()
async def stop(self) -> None:
if self._context:
await self._context.close()
if self._browser:
await self._browser.close()
if self._playwright:
await self._playwright.stop()
async def __aenter__(self):
await self.start()
return self
async def __aexit__(self, *args):
await self.stop()
# ---------------------------------------------------------------------------
# Agent tools
# ---------------------------------------------------------------------------
class PlaywrightTools:
"""Eight Playwright operations wrapped as agent-callable tools."""
def __init__(self, session: BrowserSession):
self.session = session
@property
def page(self) -> Page:
if not self.session.page:
raise RuntimeError("Browser session not started")
return self.session.page
async def navigate(self, url: str) -> dict[str, str]:
await self.page.goto(url, wait_until="domcontentloaded", timeout=30000)
title = await self.page.title()
current_url = self.page.url
return {"status": "navigated", "url": current_url, "title": title}
async def click(self, selector: str) -> dict[str, str]:
# Try text selector first, then CSS
try:
locator = self.page.get_by_text(selector, exact=False)
count = await locator.count()
if count > 0:
await locator.first.click(timeout=5000)
return {"status": "clicked", "selector": f"text={selector}"}
except Exception:
pass
await self.page.click(selector, timeout=10000)
return {"status": "clicked", "selector": selector}
async def fill(self, label: str, value: str) -> dict[str, str]:
# Find input by associated label text
locator = self.page.get_by_label(label)
count = await locator.count()
if count > 0:
await locator.first.fill(value)
return {"status": "filled", "label": label, "value": value}
# Fall back to placeholder
locator = self.page.get_by_placeholder(label)
await locator.first.fill(value)
return {"status": "filled", "placeholder": label, "value": value}
async def get_content(self) -> dict[str, Any]:
title = await self.page.title()
url = self.page.url
# Extract clean text — no HTML
body_text = await self.page.evaluate(
"() => document.body.innerText.slice(0, 3000)"
)
return {"title": title, "url": url, "text": body_text}
async def get_links(self) -> dict[str, Any]:
links = await self.page.evaluate("""
() => Array.from(document.querySelectorAll('a[href]'))
.map(a => ({text: a.textContent.trim().slice(0, 100), href: a.href}))
.filter(l => l.text && l.href.startsWith('http'))
.slice(0, 30)
""")
return {"count": len(links), "links": links}
async def extract_table(self, selector: str = "table") -> dict[str, Any]:
data = await self.page.evaluate(f"""
() => {{
const table = document.querySelector('{selector}');
if (!table) return null;
const headers = Array.from(table.querySelectorAll('th')).map(th => th.textContent.trim());
const rows = Array.from(table.querySelectorAll('tbody tr')).map(tr =>
Array.from(tr.querySelectorAll('td')).map(td => td.textContent.trim())
);
return {{headers, rows}};
}}
""")
return {"table": data}
async def wait_for(self, selector: str) -> dict[str, str]:
await self.page.wait_for_selector(selector, timeout=15000)
return {"status": "visible", "selector": selector}
async def screenshot(self) -> dict[str, str]:
import base64
data = await self.page.screenshot(type="png")
b64 = base64.standard_b64encode(data).decode("utf-8")
return {"status": "captured", "size_bytes": len(data), "b64_preview": b64[:50] + "..."}
TOOL_DEFINITIONS = [
{"name": "navigate", "description": "Navigate to a URL",
"input_schema": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}},
{"name": "click", "description": "Click an element by text or CSS selector",
"input_schema": {"type": "object", "properties": {"selector": {"type": "string"}}, "required": ["selector"]}},
{"name": "fill", "description": "Fill a form field by its label",
"input_schema": {"type": "object", "properties": {"label": {"type": "string"}, "value": {"type": "string"}}, "required": ["label", "value"]}},
{"name": "get_content", "description": "Get the page title, URL, and main text content",
"input_schema": {"type": "object", "properties": {}}},
{"name": "get_links", "description": "Get all links on the current page",
"input_schema": {"type": "object", "properties": {}}},
{"name": "extract_table", "description": "Extract a table from the page",
"input_schema": {"type": "object", "properties": {"selector": {"type": "string"}}}},
{"name": "wait_for", "description": "Wait for an element to appear",
"input_schema": {"type": "object", "properties": {"selector": {"type": "string"}}, "required": ["selector"]}},
{"name": "screenshot", "description": "Take a screenshot of the current page",
"input_schema": {"type": "object", "properties": {}}},
]
async def dispatch(self, name: str, inputs: dict[str, Any]) -> str:
"""Dispatch a tool call by name."""
handler = getattr(self, name, None)
if not handler:
return json.dumps({"error": f"Unknown tool: {name}"})
result = await handler(**inputs)
return json.dumps(result)
# ---------------------------------------------------------------------------
# Browser agent
# ---------------------------------------------------------------------------
class BrowserAgent:
"""Agent that browses the web using Playwright tools."""
SYSTEM = """You are a web research agent. Use the provided browser tools to
complete the user's task. Rules:
- Always start by navigating to the relevant URL
- Use get_content() to understand the current page before acting
- Use semantic selectors (text, aria-label) not CSS position
- Never return raw HTML — extract structured data
- When research is complete, synthesize findings into a clear answer"""
def __init__(self):
self.client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
async def run(self, task: str, max_turns: int = 15) -> str:
async with BrowserSession() as session:
tools = PlaywrightTools(session)
messages = [{"role": "user", "content": task}]
print(f"Task: {task}\n{'-'*50}")
for turn in range(max_turns):
response = self.client.messages.create(
model="claude-sonnet-4-5",
max_tokens=4096,
system=self.SYSTEM,
tools=PlaywrightTools.TOOL_DEFINITIONS,
messages=messages,
)
assistant_content = []
tool_results = []
for block in response.content:
if block.type == "text":
print(f"[Turn {turn+1}] {block.text[:200]}")
assistant_content.append({"type": "text", "text": block.text})
elif block.type == "tool_use":
print(f"[Tool] {block.name}({json.dumps(block.input)[:80]})")
result = await tools.dispatch(block.name, block.input)
assistant_content.append({"type": "tool_use", "id": block.id, "name": block.name, "input": block.input})
tool_results.append({"type": "tool_result", "tool_use_id": block.id, "content": result})
messages.append({"role": "assistant", "content": assistant_content})
if tool_results:
messages.append({"role": "user", "content": tool_results})
if response.stop_reason == "end_turn":
break
# Return final text response
for block in response.content:
if hasattr(block, "text"):
return block.text
return "Task completed."
if __name__ == "__main__":
async def demo():
agent = BrowserAgent()
result = await agent.run(
"Navigate to https://example.com, get the page title and main heading text, then list any links you find."
)
print(f"\nResult:\n{result}")
asyncio.run(demo())
Unlock the full lesson
You've read the first 2 sections. The rest of this lesson covers Check Yourself — plus a hands-on lab, quiz, and project artifact.
Create a free account — the opening phases of 29 of 35 courses are free, no credit card.
Browse all courses · View pricing · DeVenture Academy