DesearchBot

    DesearchBot is the web crawler operated by Desearch. It fetches public web pages so that people and AI agents searching through Desearch can find them, and cites the source back in every answer. This page documents how to identify it, how to verify that a request really came from us, and how to control what it crawls.

    Identity

    robots.txt token
    DesearchBot
    User-Agent
    Mozilla/5.0 (compatible; DesearchBot/1.0; +https://www.desearch.ai/crawler)
    Signature-Agent
    https://www.desearch.ai/crawler
    Key directory
    https://www.desearch.ai/.well-known/http-message-signatures-directory
    Key ID
    SIH21WRtn5LqTl8zoecrtqPzvQko6jb7RDh21IMjbwU
    Contact
    support@desearch.ai

    Verifying a request

    Anyone can put DesearchBot in a User-Agent header. That string proves nothing on its own, so do not allowlist on it.

    Every request we send is signed with an Ed25519 key using HTTP Message Signatures (RFC 9421), following the Web Bot Auth scheme. The signature covers the host the request is sent to, so it cannot be replayed against a different site, and it holds whatever address the request arrives from.

    Signature-Agent: "https://www.desearch.ai/crawler"
Signature-Input: sig1=("@authority" "signature-agent")
  ;created=1788896662;keyid="SIH21WRtn5LqTl8zoecrtqPzvQko6jb7RDh21IMjbwU"
  ;alg="ed25519";expires=1788896962;tag="web-bot-auth"
Signature: sig1=:4qF0VDVJkCSUrui1Aqg/8p9lYo/URN8oG001r/3uwL5vswanZz6KbRRh1hyEW3mBf8ODALjh22O+9wKlcb0yDg==:

    To check a request:

    1. Read the Signature-Agent header and fetch the key directory below.
    2. Match the keyid from Signature-Input against a key in that directory.
    3. Verify the signature over the covered components, in the order they are listed.

    Our public keys are published at https://www.desearch.ai/.well-known/http-message-signatures-directory. A request that fails this check is not from us, whatever its User-Agent claims — and we would like to hear about it.

    We also crawl from a small, stable set of static addresses and never route through residential proxies. If you would rather just ask, send the timestamp, the host it hit and the source IP address to support@desearch.ai and we will tell you whether it was ours.

    How it crawls

    robots.txt
    We read it before we visit a host, and again on later visits. We follow Allow, Disallow and Crawl-delay written for our token, and fall back to your * group when you have not named us.
    Request rate
    At most one request per second to a host, and slower whenever your robots.txt sets a longer Crawl-delay. That covers every request we make: robots.txt, each sitemap and each page.
    Rendering
    A plain HTTP GET. We do not execute JavaScript, run a headless browser, or load your page assets.
    Discovery
    We start from the sitemap your robots.txt advertises. Where there is none we try /sitemap.xml and /sitemap_index.xml, and stop if neither parses.
    Revisits
    We re-read a sitemap on a schedule set by how often it actually changes, and fetch a page again when its sitemap entry changes.
    Attribution
    Every page we index keeps its link, headline and source, and Desearch cites them in the answers it returns.

    What it never does

    • Bypass paywalls, logins, or metered access.
    • Solve or evade CAPTCHAs and bot challenges.
    • Submit forms, or take any other action that writes to your site.
    • Crawl a path your robots.txt disallows.
    • Rotate residential IP addresses to disguise where it comes from.

    Blocking or slowing it down

    Add any of the following to /robots.txt at the root of your host. There is nobody to contact and nothing to register, and you do not need a rule to allow us: with no rule for our token we follow your * group.

    Block us completely

    We stop at robots.txt and fetch nothing else from your host.

    User-agent: DesearchBot
Disallow: /

    Block part of the site

    Everything outside these paths stays crawlable.

    User-agent: DesearchBot
Disallow: /members/
Disallow: /newsletters/

    Slow us down

    Seconds to wait between our requests. Any value above one second replaces our default.

    User-agent: DesearchBot
Crawl-delay: 10

    Removing a page that is already indexed

    robots.txt controls crawling, not what has already been stored. To drop a page, serve X-Robots-Tag: noindex or add a noindex meta tag and we remove it at the next fetch. For an urgent takedown, email us and we act manually.

    Contact

    Wrong crawl rate, unexpected traffic, a page that should not be indexed, or anything else about this crawler: support@desearch.ai. We reply within two business days.