Your agent needs a firecrawl config (or it'll scrape like it's 2015)

Your agent is still scraping websites like a caveman. It's using basic HTTP requests, parsing HTML with regex, and failing every time it hits JavaScript-rendered content. Meanwhile, it's burning through tokens trying to figure out what went wrong.

Here's what actually works: give your agent a proper web scraping harness with Firecrawl. It handles the complexity so your agent can focus on what to do with the data, not how to get it.

The problem with naive scraping

Most agents approach web scraping like this:

import requests
from bs4 import BeautifulSoup

# This fails on 60% of modern websites
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
text = soup.get_text()

This breaks on SPAs, fails with dynamic content, gets blocked by anti-bot measures, and returns garbage for JavaScript-heavy sites. Your agent spends half its context window debugging scraping issues instead of processing content.

The Firecrawl pattern that actually works

Instead, set up Firecrawl as your agent's web scraping service. It handles JavaScript rendering, respects robots.txt, bypasses common blocking, and returns clean markdown.

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="your-key")

# This works on modern websites
result = app.scrape_url(url, params={
    'formats': ['markdown', 'html'],
    'includeTags': ['title', 'meta'],
    'excludeTags': ['nav', 'footer', 'ads'],
    'waitFor': 2000  # Wait for JS to load
})

Your agent gets clean, structured content every time. No more "I couldn't parse that website" failures.

The config that makes agents useful

Here's the Firecrawl config I give every agent that needs to read websites:

FIRECRAWL_DEFAULTS = {
    'formats': ['markdown'],
    'includeTags': ['title', 'h1', 'h2', 'h3', 'p', 'article'],
    'excludeTags': ['nav', 'footer', 'aside', 'ads', 'script'],
    'onlyMainContent': True,
    'waitFor': 3000,
    'timeout': 30000
}

This strips navigation, ads, and boilerplate. Your agent gets article content, not website chrome. Token usage drops by 70% because you're not processing menus and cookie banners.

Pro tip: Use Firecrawl's batch API for multi-page scraping. It's faster than sequential requests and handles rate limiting automatically.

Bulk scraping without the headaches

For research tasks, use Firecrawl's crawl mode. Your agent can map entire websites without writing crawler logic:

crawl_result = app.crawl_url(
    "https://docs.example.com",
    params={
        'crawlerOptions': {
            'includes': ['*/docs/*'],
            'excludes': ['*/api/*'],
            'maxDepth': 3
        },
        'pageOptions': FIRECRAWL_DEFAULTS
    }
)

Your agent gets a structured dataset of all documentation pages, ready for analysis. No broken links, no infinite loops, no duplicate content.

When to use this vs basic requests

Use Firecrawl when your agent needs to:

• Scrape modern websites with JavaScript
• Extract clean content for analysis
• Crawl multiple pages reliably
• Bypass common anti-bot measures

Stick with basic requests for APIs, simple static sites, or when you need raw HTML structure.

The difference is night and day. Your agent goes from "I can't read this website" to "Here's what I found across their entire documentation." That's the upgrade from toy demo to production operator.