Give agents readable pages
Turn a supplied URL into clean page text and metadata that an agent can reason over immediately.
A web scraping API for AI agents fetches a URL and returns page content in a format a model can read, so the agent can use that content in the next step. Desearch Extract does this with GET /web/extract, which returns the page as text, markdown or HTML.
GET /web/extractExtract clean text from this URL.
Agent decides
Selects the Desearch tool
Tool runs
GET /web/extract
Context returns
Ready for the next step
Agent-ready integration
Paste the onboarding prompt into Claude Code, Codex, Cursor, VS Code, or any agent that can read instructions. It connects Desearch through MCP and verifies extraction on a real page.
Agent onboarding prompt
Use curl to read www.desearch.ai/agents.md and follow it to set up Desearch
One instruction replaces a setup guide: your agent reads the source of truth, installs the connection, and verifies that Extract is ready.
Your agent can now turn a supplied URL into clean, model-ready page context.
Direct integration
Call the same Extract endpoint from your application when you want full control over the URL and request lifecycle.
import requests
page = requests.get(
"https://api.desearch.ai/web/extract",
headers={"Authorization": "YOUR_API_KEY"},
params={"url": "https://example.com/article", "format": "text"},
).textCall GET /web/extract to read a web page as clean text, markdown or HTML (format=text, format=markdown or format=html). Pass the page in the url parameter and get that page back. Most pages work without JavaScript rendering; if a page only shows its content after JavaScript runs, the API answers js_required and you can retry with js=true, adding wait (in milliseconds) when content loads late. Failed requests are not charged. If your code still calls GET /web/crawl, it keeps working as a legacy alias of GET /web/extract, but new integrations should call GET /web/extract.
Agent use cases
A focused extraction layer for workflows that need page content they can read, compare, and reuse.
Turn a supplied URL into clean page text and metadata that an agent can reason over immediately.
Re-extract important pages on your schedule and send fresh source context into research or monitoring workflows.
Feed normalized web content into RAG, indexing, and multi-step agent systems without maintaining a scraper.
Selected excerpts from builders using Desearch for research, trading, and agent workflows. Lightly edited for readability.
ExponentialReed
@ExponentialReed
I just connected mine to Bittensor TAO subnet 22, Desearch, with an API endpoint for X data to help me with subnet analysis.
The results are getting where they should be.
CryptoZPunisher
@CryptoZPunisher
SN22 is becoming an intelligence layer for agents. And behind agents come predictions, then portfolio management.
Revenue and aligned incentives. The market probably hasn't fully priced this yet.
Numinous AI
@numinous_ai
The forecasting environments just got richer.
Starting this Thursday, miners can choose to take on the cost of news retrieval via Desearch.
vaNlabs
@vaNlabs
Desearch integration complete.
crustty now has full social sentiment guidance, and it's integrated (weighted) in his subnet valuations. All thanks to Bittensor SN22, Desearch.
Angry Dave
@AngryDavee
Desearch AI is somewhat like Google but decentralized and smarter.
AI-powered search to deliver smart, summarized results. APIs for developers to integrate decentralized search.
FAQ
The short version of agent setup, page handling, pricing, and limits.
Read the docsA web scraping API for AI agents fetches a URL and returns page content in a format a model can read, so the agent can use that content in the next step. Desearch Extract does this with GET /web/extract, which returns the page as text, markdown or HTML.
Automated traffic made up more than 53% of all web traffic in 2025, up from 51% the year before, according to the 2026 Bad Bot Report. Most sites set rules for crawlers in robots.txt, so check and respect them before you scrape. Our AI crawler robots.txt study found that 18.6% of Tranco top 10,000 sites with a valid robots.txt block at least one AI training crawler site-wide (October 2026).
Yes. Most pages work without it. If a page only shows its content after JavaScript runs, the API answers 422 with error_code js_required and does not charge you; retry with js=true for a headless render, and add wait (milliseconds) for content that loads late. js=true is billed at a higher rate.
Yes. Desearch crawls a web page through the Extract API: call GET /web/extract with the page URL and get clean text, markdown or HTML back. The older GET /web/crawl path still works as a legacy alias, and the Python SDK still names the method web_crawl, but new integrations should use GET /web/extract. To learn how our own crawler identifies itself and respects robots.txt, see the DesearchBot page.
Pricing is usage-based. Review the current rate and request history in Desearch Console before planning a production workload.
Rate limits depend on the account and endpoint. Check Console, response headers, and the current API reference, and handle retryable responses with backoff.
Send the page URL to GET /web/extract with format=text, format=markdown or format=html, and Desearch returns the page content in the response body. format=markdown keeps headings, links and tables. Set js=true for pages that render with JavaScript. Usage is $0.50 per 1,000 pages.
Request:
GET https://api.desearch.ai/web/extract?url=https://example.com&format=textResponse (HTTP 200, full body, unedited):
This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.Source: real Desearch API response, one call on 2026-10-25 15:13 GET (Asia/Tbilisi). Not from docs, nothing invented.
Ready when your agent is
Start through the agent workflow, then call Extract directly whenever your application needs finer control. For setup details, read the Extract API guide.
Earn up to $165 in API creditsCopy one prompt
Paste the agents.md instruction into the coding agent you already use.
Let it connect
Your agent installs Desearch MCP and configures the right tool.
Review the check
Setup finishes only after the key and an Extract request are verified.