Find Directories for Directory Submission with Python

Made by Desearch (desearch.ai).

Directory Finder is a small Python script for founders and marketers who want to list a new AI tool. Before any directory submission, it builds the list: which directories take submissions, and does each submit page still load.

The problem

Directory submission starts with research. Searching by hand brings up the same big sites for every wording, plus list articles and services that submit for you, and you still have to open every page to see if it takes submissions. What helps is one deduped list of directories with each submit page read once.

What the app does

• Runs 3 to 5 query variants for your category from config.json.
• Pages through the web results with start and keeps one row per domain.
• Picks results whose link or title looks like a submit page ("submit", "add your", "list your").
• Reads each likely submit page as text and looks for short quotes about free listings, paid options and review.
• Lets you mark each site by hand as a directory, a list or article, or a submission service, and keeps only directories.
• Writes output/directories.csv and output/shortlist.md.

How it works

1. Search. GET /web with query and start for each query variant.
2. Dedupe. One row per domain across all searches.
3. Read. GET /web/extract with format=text and js=false for each likely submit page.
4. Sort. You classify each site in config.json; only directories go to the output.

The code

The search loop, copied from directory_finder.py on main (lines 108 to 113). The script reads the API key from a key file (--key-file).

def fetch(cfg, key, max_extracts):
    s = session(key)
    searches = []
    for q in cfg["queries"]:
        for st in cfg.get("starts", [0, 10, 20]):
            searches.append(call(s, "/web", {"query": q, "start": st}, f"web start={st}"))

The folder has directory_finder.py, an example config.json, samples/raw-sample.json (a trimmed saved run you can rebuild with no key) and offline tests.

View the code

Endpoints

• GET /web
• GET /web/extract

Example output

A run for the category "AI tools" on 2026-10-14 used four query variants, such as "submit your AI tool directory" and "add your AI product to directory". This page shows that stored run. It does not run a live demo.

Most of the directory pages mention a review step, and several mention both a free and a paid option. The shortlist marks only whether a word appears on the page; read each page yourself before you submit.

Run it yourself

git clone https://github.com/Desearch-ai/desearch-py-examples && cd desearch-py-examples/directory-finder
pip install -r requirements.txt
printf '%s' 'YOUR_KEY' > ~/.desearch_api_key && chmod 600 ~/.desearch_api_key   # key from console.desearch.ai
python directory_finder.py config.json --key-file ~/.desearch_api_key

Edit config.json for your category, query variants and starts; steps of 20 (0, 20, 40) brought new domains in testing, since each page held 20 results. To try it with no key and no cost: python directory_finder.py --from-raw samples/raw-sample.json.

What it costs

On 2026-10-14 a web search cost $0.001 and a page read cost $0.0005, read from the X-Desearch-Cost-Usd header. See pricing.

Ideas to extend it

• Re-run every few weeks and compare the CSVs to see which submit pages changed.
• Try other categories by editing the query variants.
• Add a column for the date you submitted to each directory.