Whitespace.
See who already built your idea — and the space they left open.

CLUSTER TXT-WEB-FILES · 6 ENTRIES · EARLIEST ENTRY unknown

AI crawler SEO tools

This cluster comprises tools that help website owners manage and optimize how AI and LLM crawlers access their sites, using files like robots.txt and llms.txt, and performing crawl audits. The cluster's dated history is too thin to read a shape from – it predates the index or its members are undated. Among traction signals, Screaming-Frog-SEO-Spider-Full-Version-2026 leads with 1 star.

Entry timeline

ENTRY TIMELINE

AUG 2026: 3SEP 2026: 3
AUG 2026SEP 2026

Each bar is the month a member carries, which is a creation date where the source published one and a crawl date otherwise. Nothing in the index was found before 11 August 2026, and coverage broadened sharply on 14–15 August, so a bar from then on reports how much we found that month, not how much was built; an earlier bar reports how far back a member's own source dated it — not how long we have been watching.

Members by activity

Activity signals, not revenue.

Build origin · 2 ai-built · 1 hand-coded · 3 unlabelled

Operator · 1 open source · 5 unlabelled

Two labels per entry, never collapsed into one: how it was built, and who stands behind it. Counted over the 6 listed below. Unlabelled means nothing has decided yet — not that the answer is small.

  1. 01Screaming-Frog-SEO-Spider-Full-Version-20261★1 mo—SEO website crawler that removes URL crawl limits, renders JavaScript, and automates crawl tasks via command-line scripts.
  2. 02just-a-simple-lens2★1 mo—Crawls a URL as both a search bot and an AI/LLM crawler, scores it on two pipelines, and returns actionable SEO fixes.
  3. 03foxycorp_audit0★1 mo—hand-codedopen sourceA free CI linter that checks websites for AI crawler readiness (llms.txt, robots access, JSON-LD) and reports a score in PR comments.
  4. 04OpenRobots — Free robots.txt & llms.txt generator for AI crawlers0★unknownliveWeb app that generates robots.txt and llms.txt files to allow, block, or audit AI crawlers.
  5. 05BotControl - robots.txt & Crawl Manager0★unknownliveai-builtWeb app that manages robots.txt rules, sitemaps, and provides Common Crawl indexing status for website owners.
  6. 06LLMs.txt Generator - Create AI-Optimized Website Content Files0★unknownliveai-builtWeb app that crawls a website and generates llms.txt files to improve AI searchability.

Ranked by traction, highest first, and paged 20 at a time — this is a window on the cluster, never the whole of it. Ties are broken by a stable id so a product cannot appear on two pages or fall between them. Browse every cluster.