Agentic Crawl
Goal-driven web crawling
Collect what your goal needs from the web, and nothing the sites’ rules forbid.
Tell Agentic Crawl what you are looking for and which sites to read. It takes next the page most likely to answer, decides every request before it is sent, and keeps a record of every choice and every refusal. You get the pages, their content as a dataset, and the record.
Toward the goal
| Agentic Crawl | Breadth first | |
|---|---|---|
| Requests to find all 40 tent pages | 90 | 1,121 |
| Requests lost to traps | 3 | 3,291 |
| Requests the rules forbid | 0 | 0 |
From the record
| Decision | Why |
|---|---|
refused /cart | the site’s robots.txt disallows it |
skipped /p/…?sid=731850 | a copy: sid does not change the page |
took /c/backpacks | highest goal score, 0.672 |
From a recorded run on the test corpus described below. Your sites will differ; that is what a pilot measures.
Why a crawler should have a goal
A crawler that reads everything in order spends its budget where the links happen to lead. One that knows what it is for can go straight there, and still has to keep every site’s rules on the way.
Budget
Most pages are not the answer
On our test sites, a breadth-first crawl needed 297 to 1,718 requests to find every page a goal called for. Agentic Crawl needed 51 to 90 for four of the five goals, and stopped by itself once the goal went quiet.
Rules
Checked before every request
The crawl’s scope, each site’s terms, its robots.txt and its rules for AI crawlers are checked before a request is sent, for every page and every redirect. A refusal sends nothing, and is recorded with its reason.
Record
Every choice can be read back
Each step is written as a Brain API decision record: the links it weighed and their scores, what the rules said, what it took and why. The collected pages form an Agentic Dataset that says where each item came from.
How it decides
The agent chooses the order; it never fetches anything itself. Whatever it chooses goes through the same checks as any other request, which it cannot change.
Read every start page
The sites you name are read first, so their links can be judged against the goal before any is followed.
Take the link closest to the goal
Every queued link is scored by how close its own words (its text, its address, the title of the page it sits on) and the page it was found on come to the goal, using an embedding model that runs where the crawler runs. The highest score goes next.
Check the request before it is sent
Scope, the site’s terms, robots.txt and its rules for AI crawlers. If any says no, nothing is sent; the refusal and its reason go on the record.
Skip copies
When two addresses that differ only by a parameter (a session id, a print view) return the same page, the parameter is learned, and further addresses that differ only by it are skipped.
Extract and record
Each page becomes clean, structured text with a record of where and when it was fetched and under which decision.
Stop when the goal goes quiet
After a run of pages well below the best it has found, the crawl ends by itself, and says why.
What has been measured
Every result below comes from a test corpus built for checking crawlers exactly: nine generated sites and 1,799 pages, with every page’s content, every link, rule, trap and change fixed before any crawl starts. The sites’ servers record every request that arrives, so what a crawl did is read from their log, not from the crawler.
Across 36 recorded crawls, no request reached a page that robots.txt, a site’s rules for AI crawlers or its terms put off limits, and every one of the 1,210 refusals gave the reason the corpus itself gives.
Breadth-first and goal-driven crawls, two corpora, three content versions.
For a goal of 40 product pages, the agent found them all in 90 requests; reading in order took 1,121. For backup guides, forum advice and library books: 51 vs 297, 61 vs 1,718 and 75 vs 860.
One run per goal, the same sites and rules for both; a second, held-out corpus gives the same picture.
Session-id links that never end took two thirds of a breadth-first crawl’s requests. The agent fetched three copies, learned that the parameter changes nothing, and skipped the rest.
The tent goal; the same corpus and request budget.
Extracted text matched each page’s known content, and no page lost more than half of it, including pages built from separate sections, which had lost a third of their text before.
1,648 pages with exact gold content; token overlap.
Re-crawled a day and a week later, every page’s “changed” or “unchanged” verdict agreed with what had in fact changed, including on every server that kept saying “unchanged” when it was not.
125 and 325 changed pages; 41 and 81 of them behind stale validators.
Built to be checked
Every decision lands in a hash-chained record that can be verified after the fact. The pages come with where and when they were fetched and under which decision, and the collection is an Agentic Dataset that can be served to other agents.
What we do not claim
The test sites are generated: their words come from small vocabularies, which makes matching easier than on real sites. No result for your sites until a pilot measures it. Pages whose content only a script writes are fetched but not read yet, and forum author cards still slip into extracted text.
The pilot
Name a goal and the sites you own or may crawl. We run Agentic Crawl under those sites’ rules and send you everything it produced.
1 · Goal
What, and where
A goal in plain words (the fields you want, if any) and the sites to read. Your own sites, or ones whose terms allow it.
2 · Run
Under the sites’ rules
Scope, terms, robots.txt and AI-crawler rules are checked before every request. Nothing is fetched that they refuse.
3 · Read
Pages, dataset, record
The pages it kept, their content as a dataset, and the record of every choice and refusal, to check against what you expected.
Request a pilot
Send us a message on LinkedIn with the goal and the sites, and we will set up a pilot with you. The pilot is free.
Who is behind it
Agentic Crawl is built by Alexander Chernov, whose work is on governed AI systems: deciding, with evidence, what an automated system may do.
Every figure on this page comes from a recorded run on the test corpus.
From the same work
Grafferene, a response cache for LLM endpoints that measures before it serves.