Extract API for agents

    Web scraping API for clean, agent-ready page text

    Desearch Extract is a web scraping API built for AI agents. Send any URL to GET /web/extract and get back clean page text or HTML your agent can use right away. Use it over MCP or REST. Usage is billed at $0.50 per 1000 pages, and new accounts get $5 in API credits after attaching a card in Console.

    Get your API keyView Documentation
    $0.50/1000 pages
    Checked-in usage rate
    Clean
    Page text
    MCP + REST
    Agent-ready access
    Agent tool flow
    GET /web/extract
    Agent asks

    Extract clean text from this URL.

    1. Agent decides

      Selects the Desearch tool

    2. Tool runs

      GET /web/extract

    3. Context returns

      Ready for the next step

    Extraction response previewrunning

    What the web scraping API returns

    clean text
    optional HTML
    rendered content
    extraction status
    Agent receivesclean textoptional HTMLrendered contentrequest status

    Trusted by AI innovators worldwide

    View case studies

    Agent-ready integration

    Set up the scraping API with one prompt (MCP)

    Paste the onboarding prompt into Claude Code, Codex, Cursor, VS Code, or any agent that can read instructions. It connects Desearch through MCP and verifies extraction on a real page.

    Agent onboarding prompt

    Use curl to read www.desearch.ai/agents.md and follow it to set up Desearch

    One instruction replaces a setup guide: your agent reads the source of truth, installs the connection, and verifies that Extract is ready.

    Agent setup
    waiting for view
    Extract verifiedMCP connected

    Your agent can now turn a supplied URL into clean, model-ready page context.

    Direct integration

    Prefer direct API control?

    Call the same Extract endpoint from your application when you want full control over the URL and request lifecycle.

    GET /web/extract
    import requests

page = requests.get(
    "https://api.desearch.ai/web/extract",
    headers={"Authorization": "YOUR_API_KEY"},
    params={"url": "https://example.com/article", "format": "text"},
).text

    Crawl a web page with the API

    Call GET /web/extract to read a web page as clean text or HTML (format=text or format=html). Pass the page in the url parameter and get that page back. For pages that load content with JavaScript, set js=true for a headless render, and add wait (in milliseconds, only with js=true) when content loads late; test the pages you depend on. If your code still calls GET /web/crawl, it keeps working as a legacy alias of GET /web/extract, but new integrations should call GET /web/extract.

    Agent use cases

    Web scraping use cases for AI agents

    A focused extraction layer for workflows that need page content they can read, compare, and reuse.

    Give agents readable pages

    Turn a supplied URL into clean page text and metadata that an agent can reason over immediately.

    Monitor changing sources

    Re-extract important pages on your schedule and send fresh source context into research or monitoring workflows.

    Build ingestion pipelines

    Feed normalized web content into RAG, indexing, and multi-step agent systems without maintaining a scraper.

    What builders say

    Selected excerpts from builders using Desearch for research, trading, and agent workflows. Lightly edited for readability.

    View all

    ExponentialReed

    @ExponentialReed

    I just connected mine to Bittensor TAO subnet 22, Desearch, with an API endpoint for X data to help me with subnet analysis.

    The results are getting where they should be.

    53912.9K views

    CryptoZPunisher

    @CryptoZPunisher

    SN22 is becoming an intelligence layer for agents. And behind agents come predictions, then portfolio management.

    Revenue and aligned incentives. The market probably hasn't fully priced this yet.

    43103,886 views

    Numinous AI

    @numinous_ai

    The forecasting environments just got richer.

    Starting this Thursday, miners can choose to take on the cost of news retrieval via Desearch.

    53165,849 views

    vaNlabs

    @vaNlabs

    Desearch integration complete.

    crustty now has full social sentiment guidance, and it's integrated (weighted) in his subnet valuations. All thanks to Bittensor SN22, Desearch.

    2853,537 views

    Angry Dave

    @AngryDavee

    Desearch AI is somewhat like Google but decentralized and smarter.

    AI-powered search to deliver smart, summarized results. APIs for developers to integrate decentralized search.

    206770 views

    FAQ

    Web scraping API FAQ

    The short version of agent setup, page handling, pricing, and limits.

    Read the docs

    An API that fetches a URL and returns its content in a usable format; Extract returns clean text or HTML via GET /web/extract.

    Automated traffic made up more than 53% of all web traffic in 2025, up from 51% the year before, according to the 2026 Bad Bot Report. Most sites set rules for crawlers in robots.txt, so check and respect them before you scrape. Our AI crawler robots.txt study found that 18.6% of Tranco top 10,000 sites with a valid robots.txt block at least one AI training crawler site-wide (October 2026).

    Page handling varies by site and response. Set js=true for a headless render, and use wait (milliseconds, only with js=true) for content that loads late. Test the URLs you depend on and validate important output against the source.

    Yes. Desearch crawls a web page through the Extract API: call GET /web/extract with the page URL and get clean text or HTML back. The older GET /web/crawl path still works as a legacy alias, and the Python SDK still names the method web_crawl, but new integrations should use GET /web/extract. To learn how our own crawler identifies itself and respects robots.txt, see the DesearchBot page.

    Pricing is usage-based. Review the current rate and request history in Desearch Console before planning a production workload.

    Rate limits depend on the account and endpoint. Check Console, response headers, and the current API reference, and handle retryable responses with backoff.

    Ready when your agent is

    Start scraping with Desearch

    Start through the agent workflow, then call Extract directly whenever your application needs finer control. For setup details, read the Extract API guide.

    Get API keyAPI reference
    Earn up to $155 in API credits
    1. 01

      Copy one prompt

      Paste the agents.md instruction into the coding agent you already use.

    2. 02

      Let it connect

      Your agent installs Desearch MCP and configures the right tool.

    3. 03

      Review the check

      Setup finishes only after the key and an Extract request are verified.