Extract API
Desearch Extract is a web scraping API built for AI agents. Fetch a public URL and get its content back as text, markdown or HTML. Each call fetches the current version of the page, not a cached snapshot.
Auth
Preferred: Authorization: $DESEARCH_API_KEY (raw key from Console). See Auth and API keys.
curl
JavaScript (desearch-js@1.10.0)
webCrawl() on this package is a deprecated alias. New JavaScript code should call extract().
Python (PyPI desearch-py==1.2.1)
Published Python today exposes web_crawl, not extract. It accepts every format, but not full_page; use REST for that.
Parameters (REST)
| Param | Type | Notes |
|---|---|---|
url | string | Required |
format | text | markdown | html | Default text |
full_page | "true" | "false" | Only with format=text. Default false |
js | "true" | "false" | Render the page in a headless browser first. Default false |
wait | int (ms) | Extra wait after load, only with js=true |
Choosing a format
textreturns the page's main content, like a reader view. It can leave out sections outside the main block, such as tables, notices or sidebars. Setfull_page=trueto get all visible text of the page, one block per line, with table cells separated by|.markdownkeeps headings, links, lists, tables and code. You get the page's main content when the page has one main block, and the whole page without navigation and footers otherwise.htmlreturns the page source as served.
The body is the content itself (text/plain, text/markdown or text/html), not JSON.
JavaScript rendering
Most pages work without it. Use js=true only for pages that build their content in the browser. It is billed at a higher rate (see the pricing page).
You don't have to guess: when a page needs JavaScript and js is off, the API answers 422 with error_code: "js_required" and you are not charged. Retry that URL with js=true.
Errors
Failed requests return JSON with detail and error_code, and are not charged.
| Status | error_code | Meaning | What to do |
|---|---|---|---|
| 404 | page_not_found | The site answered 404 or 410 | Check the URL |
| 404 | no_content | The page has no text | Try format=html or js=true |
| 422 | js_required | The page only shows content after JavaScript runs | Retry with js=true |
| 503 | blocked | The site refused the request or answered with a bot check | Retry later, or with js=true |
| 503 | upstream_timeout | The page took too long to load | Retry later |
| 503 | upstream_failed | The page could not be fetched | Retry later |
A 422 without error_code is a parameter validation error.
Legacy crawl
GET /web/crawl remains a compatibility alias with the same parameters. New integrations should call /web/extract.
The full schema is in the API Reference. How the crawler identifies itself and respects robots.txt is documented at desearch.ai/crawler.
Typed clients: JavaScript SDK · Python SDK.