Extract API for agents

    Web scraping API for clean, agent-ready page text

    A web scraping API for AI agents fetches a URL and returns page content in a format a model can read, so the agent can use that content in the next step. Desearch Extract does this with GET /web/extract, which returns the page as text, markdown or HTML.

    $0.50/1000 pages
    Checked-in usage rate
    Clean
    Page text
    MCP + REST
    Agent-ready access
    Agent tool flow
    GET /web/extract
    Agent asks

    Extract clean text from this URL.

    1. Agent decides

      Selects the Desearch tool

    2. Tool runs

      GET /web/extract

    3. Context returns

      Ready for the next step

    Extraction response previewrunning

    What the web scraping API returns

    clean text
    markdown
    optional HTML
    rendered content
    Agent receivesclean textmarkdownoptional HTMLrequest status

    Agent-ready integration

    Set up the scraping API with one prompt (MCP)

    Paste the onboarding prompt into Claude Code, Codex, Cursor, VS Code, or any agent that can read instructions. It connects Desearch through MCP and verifies extraction on a real page.

    Agent onboarding prompt

    Use curl to read www.desearch.ai/agents.md and follow it to set up Desearch

    One instruction replaces a setup guide: your agent reads the source of truth, installs the connection, and verifies that Extract is ready.

    Agent setup
    waiting for view
    Extract verifiedMCP connected

    Your agent can now turn a supplied URL into clean, model-ready page context.

    Direct integration

    Prefer direct API control?

    Call the same Extract endpoint from your application when you want full control over the URL and request lifecycle.

    GET /web/extract
    import requests
    
    page = requests.get(
        "https://api.desearch.ai/web/extract",
        headers={"Authorization": "YOUR_API_KEY"},
        params={"url": "https://example.com/article", "format": "text"},
    ).text

    Crawl a web page with the API

    Call GET /web/extract to read a web page as clean text, markdown or HTML (format=text, format=markdown or format=html). Pass the page in the url parameter and get that page back. Most pages work without JavaScript rendering; if a page only shows its content after JavaScript runs, the API answers js_required and you can retry with js=true, adding wait (in milliseconds) when content loads late. Failed requests are not charged. If your code still calls GET /web/crawl, it keeps working as a legacy alias of GET /web/extract, but new integrations should call GET /web/extract.

    Agent use cases

    Web scraping use cases for AI agents

    A focused extraction layer for workflows that need page content they can read, compare, and reuse.

    Give agents readable pages

    Turn a supplied URL into clean page text and metadata that an agent can reason over immediately.

    Monitor changing sources

    Re-extract important pages on your schedule and send fresh source context into research or monitoring workflows.

    Build ingestion pipelines

    Feed normalized web content into RAG, indexing, and multi-step agent systems without maintaining a scraper.

    FAQ

    Web scraping API FAQ

    The short version of agent setup, page handling, pricing, and limits.

    Read the docs

    A web scraping API for AI agents fetches a URL and returns page content in a format a model can read, so the agent can use that content in the next step. Desearch Extract does this with GET /web/extract, which returns the page as text, markdown or HTML.

    Automated traffic made up more than 53% of all web traffic in 2025, up from 51% the year before, according to the 2026 Bad Bot Report. Most sites set rules for crawlers in robots.txt, so check and respect them before you scrape. Our AI crawler robots.txt study found that 18.6% of Tranco top 10,000 sites with a valid robots.txt block at least one AI training crawler site-wide (October 2026).

    Yes. Most pages work without it. If a page only shows its content after JavaScript runs, the API answers 422 with error_code js_required and does not charge you; retry with js=true for a headless render, and add wait (milliseconds) for content that loads late. js=true is billed at a higher rate.

    Yes. Desearch crawls a web page through the Extract API: call GET /web/extract with the page URL and get clean text, markdown or HTML back. The older GET /web/crawl path still works as a legacy alias, and the Python SDK still names the method web_crawl, but new integrations should use GET /web/extract. To learn how our own crawler identifies itself and respects robots.txt, see the DesearchBot page.

    Pricing is usage-based. Review the current rate and request history in Desearch Console before planning a production workload.

    Rate limits depend on the account and endpoint. Check Console, response headers, and the current API reference, and handle retryable responses with backoff.

    Send the page URL to GET /web/extract with format=text, format=markdown or format=html, and Desearch returns the page content in the response body. format=markdown keeps headings, links and tables. Set js=true for pages that render with JavaScript. Usage is $0.50 per 1,000 pages.

    Request:

    GET https://api.desearch.ai/web/extract?url=https://example.com&format=text

    Response (HTTP 200, full body, unedited):

    This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.

    Source: real Desearch API response, one call on 2026-10-25 15:13 GET (Asia/Tbilisi). Not from docs, nothing invented.

    Ready when your agent is

    Start scraping with Desearch

    Start through the agent workflow, then call Extract directly whenever your application needs finer control. For setup details, read the Extract API guide.

    Earn up to $165 in API credits
    1. 01

      Copy one prompt

      Paste the agents.md instruction into the coding agent you already use.

    2. 02

      Let it connect

      Your agent installs Desearch MCP and configures the right tool.

    3. 03

      Review the check

      Setup finishes only after the key and an Extract request are verified.