Extract API

    Extract API

    Desearch Extract is a web scraping API built for AI agents. Fetch a public URL and get its content back as text, markdown or HTML. Each call fetches the current version of the page, not a cached snapshot.

    text
    GET https://api.desearch.ai/web/extract

    Auth

    Preferred: Authorization: $DESEARCH_API_KEY (raw key from Console). See Auth and API keys.

    curl

    bash
    curl --request GET \ 'https://api.desearch.ai/web/extract?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FArtificial_intelligence&format=markdown' \ --header "Authorization: $DESEARCH_API_KEY"

    JavaScript (desearch-js@1.10.0)

    javascript
    const { Desearch } = require("desearch-js"); async function main() { const client = new Desearch(process.env.DESEARCH_API_KEY); const content = await client.extract({ url: "https://en.wikipedia.org/wiki/Artificial_intelligence", format: "markdown", }); console.log(String(content).slice(0, 500)); } main();

    webCrawl() on this package is a deprecated alias. New JavaScript code should call extract().

    Python (PyPI desearch-py==1.2.1)

    Published Python today exposes web_crawl, not extract. It accepts every format, but not full_page; use REST for that.

    python
    import asyncio import os from desearch_py import Desearch async def main(): async with Desearch(api_key=os.environ["DESEARCH_API_KEY"]) as client: content = await client.web_crawl( url="https://en.wikipedia.org/wiki/Artificial_intelligence", format="markdown", ) print(content[:500]) asyncio.run(main())

    Parameters (REST)

    ParamTypeNotes
    urlstringRequired
    formattext | markdown | htmlDefault text
    full_page"true" | "false"Only with format=text. Default false
    js"true" | "false"Render the page in a headless browser first. Default false
    waitint (ms)Extra wait after load, only with js=true

    Choosing a format

    • text returns the page's main content, like a reader view. It can leave out sections outside the main block, such as tables, notices or sidebars. Set full_page=true to get all visible text of the page, one block per line, with table cells separated by |.
    • markdown keeps headings, links, lists, tables and code. You get the page's main content when the page has one main block, and the whole page without navigation and footers otherwise.
    • html returns the page source as served.

    The body is the content itself (text/plain, text/markdown or text/html), not JSON.

    JavaScript rendering

    Most pages work without it. Use js=true only for pages that build their content in the browser. It is billed at a higher rate (see the pricing page).

    You don't have to guess: when a page needs JavaScript and js is off, the API answers 422 with error_code: "js_required" and you are not charged. Retry that URL with js=true.

    Errors

    Failed requests return JSON with detail and error_code, and are not charged.

    Statuserror_codeMeaningWhat to do
    404page_not_foundThe site answered 404 or 410Check the URL
    404no_contentThe page has no textTry format=html or js=true
    422js_requiredThe page only shows content after JavaScript runsRetry with js=true
    503blockedThe site refused the request or answered with a bot checkRetry later, or with js=true
    503upstream_timeoutThe page took too long to loadRetry later
    503upstream_failedThe page could not be fetchedRetry later

    A 422 without error_code is a parameter validation error.

    Legacy crawl

    GET /web/crawl remains a compatibility alias with the same parameters. New integrations should call /web/extract.

    The full schema is in the API Reference. How the crawler identifies itself and respects robots.txt is documented at desearch.ai/crawler.

    Typed clients: JavaScript SDK · Python SDK.