We stopped making our web agent click like a human. It's 15x faster now.
Our web agent was clicking through forms like a human. Three-second delays between fields. Scrolling to find buttons. Waiting for animations to finish. It worked, but barely.
Then we stripped out all the human mimicry and rebuilt it around DOM inspection and direct API discovery. The difference was dramatic:
- Form submission: 12 seconds → 800ms
- Data extraction: 45 seconds → 2 seconds
- Multi-step workflows: 3 minutes → 15 seconds
- Cost per task: $0.23 → $0.04
The key insight: web agents don't need to behave like humans. They need to behave like programs.
Instead of click('Submit'), we taught our agent to inspect forms and POST directly:
// Old approach: human-like clicking
await page.click('#email-field')
await page.type('user@example.com')
await page.click('#submit-btn')
// New approach: direct form manipulation
const form = await page.$('form')
const formData = new FormData(form)
formData.set('email', 'user@example.com')
const response = await fetch(form.action, {
method: 'POST',
body: formData
})For data extraction, instead of scrolling and clicking through pagination, we inspect the network tab to find the JSON endpoints powering the UI:
// Agent discovers: UI loads from /api/products?page=1&limit=20
// Skips UI entirely, hits API directly
const allProducts = []
for (let page = 1; ; page++) {
const response = await fetch(`/api/products?page=${page}&limit=100`)
const data = await response.json()
if (!data.products.length) break
allProducts.push(...data.products)
}The agent went from processing 50 records per minute to 2,000 records per minute.
Pro tip: Give your agent a "discover APIs" skill that monitors network requests while it browses. Half the time, there's a cleaner programmatic interface hiding behind the UI.
This pattern works for most web automation: e-commerce scraping, form submissions, data collection. The bottleneck becomes page load time, not interaction time.
The security trade-off is real though. Direct API access often bypasses rate limiting and CSRF protection. We run everything in isolated containers and rotate IP addresses to stay respectful.
But the speed gains are worth the complexity. When your agent can process 1,000 tasks in the time it used to handle 50, you're not just optimizing—you're unlocking entirely new use cases.