Crawl and content map

The crawl

The crawl walks your site page by page and stores what each page is about. Your plan sets how many pages one crawl may read: Starter 100, Growth 250, Scale 500.

Most pages are fetched directly. Sites behind a bot wall need a browser-based fetch through an external provider, which costs real money on our side — so pages are charged at 1 credit each when we fetch them ourselves and 4 when the fallback is needed.

That is why two crawls of the same size can cost differently. One flat rate would mean customers with open sites subsidising customers behind aggressive protection.

A crawl that runs out of credits stops and says where it stopped. Topping up resumes it from that page rather than starting over.

The content map

With the crawl done, the content map puts your pages next to the topics buyers actually ask about, and shows three things:

• Covered — you have a page and it is being cited.
• Written but invisible — you have a page and the models never use it. Usually a format problem: the answer is there but buried, or the page is not structured as an answer.
• Missing — buyers research this topic and you have nothing on it.

The third group is the content plan. The second is usually cheaper to fix and gets results sooner, because the work is rewriting rather than writing.

Handing it to the agent

Any gap in the map can be handed to the agent as a task — it will draft the page, and with an integration connected, put the draft straight into Google Docs, Notion or Slack.