CitedBy

Nobody tells you when the robots that AI assistants send out get a blank page from a website that looks fine in a browser. One line in a settings file can shut them out, and a page that fills in only after its scripts run reaches them empty. CitedBy crawls a site and reports what those crawlers receive, page by page.

Besides the HTML each page sends, the audit reads the sitemap and the llms.txt file in which some sites describe themselves to AI tools. It checks the robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and other AI crawlers, then sends one live request per bot to see whether a firewall turns it away. The usual site checks run alongside: broken links, duplicate titles, heading order and missing FAQ or about pages.

CitedBy's methodology says the crawler waits at least one second between requests and reads public pages only. A check that could not be read is marked "not measured". For agencies, the PDF report can carry the agency's logo and shared links its brand. Developers get a JSON API without a key and an open-source check that fails a deploy when the bots get an empty page.