Can AI read your website?
Analyze how LLMs, crawlers, and bots see your content. Get an actionable AI readability score in seconds — right from your browser.
robots.txt & Meta
Full support for Allow/Disallow precedence, wildcards, and 20+ AI user-agents — plus meta directives like noindex, noai, and noimageai.
Semantic Check
Validates heading hierarchy, HTML5 landmarks, image alt-text coverage, JSON-LD structured data, and the lang attribute.
JS Detection
Compares initial HTML to rendered DOM to catch SPA shells that hide content from AI crawlers that don't execute JavaScript.
Every signal that matters — measured in one click
When you click the BotLens icon on any website, it runs a technical audit in under a second. Nothing leaves your browser. Here's exactly what it checks.
Fetch robots.txt & raw HTML
BotLens fetches the site's /robots.txt and the page's server-rendered HTML in parallel. This lets it compare what crawlers see versus what your browser renders after JavaScript runs.
Parse crawler rules
The parser handles the full robots.txt spec: multiple user-agent groups, Allow/Disallow precedence by longest match, wildcards (*), and end-of-path anchors ($). Every rule is evaluated against the current URL for each of 20+ known AI user-agents.
Analyze the DOM
BotLens reads meta directives (robots, googlebot, gptbot, noai, noimageai), heading hierarchy (H1 count, semantic depth), HTML5 landmarks, image alt-text coverage, JSON-LD structured data, and the <html lang> attribute.
Detect JS shells
By comparing the initial HTML payload size to the rendered DOM, BotLens spots SPA shells where crawlers only see a nearly-empty document. A small initial HTML + a large rendered DOM is a red flag for LLM readability.
Compute a weighted score
Signals are weighted (robots/meta 50, semantic 30, JS 20) and combined into a single 0-100 readability score. You get a plain-language verdict — AI Friendly, Partially Restricted, or Blocked / Poor — plus the specific issues to fix.
Who is BotLens for?
Developers
Ship an SPA and want to make sure GPTBot can still index your docs? BotLens tells you if your Next.js or React Router app looks like a hollow shell to crawlers.
SEO practitioners
Traditional tools focus on Google. BotLens surfaces the emerging AI-crawler landscape: which of your clients are blocking ChatGPT? Are they opted out of AI training on purpose?
Content teams
Want your articles cited by Perplexity or answered by Claude? BotLens shows whether your semantic HTML, headings, and structured data give LLMs enough signal to represent you accurately.
Site owners
Curious whether your CMS silently opted you out of AI training? Or blocked ClaudeBot in a stale robots.txt? One click answers both.
Get BotLens
BotLens is live on the Chrome Web Store — one click and you're analyzing.
Frequently asked questions
What is BotLens?
BotLens is a free, open-source Chrome extension that analyzes whether a website can be read by AI crawlers and language models. It gives every page a 0-100 AI readability score based on robots.txt, meta tags, semantic HTML, and JavaScript rendering.
Which AI crawlers does BotLens detect?
BotLens recognizes 20+ AI user-agents including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Applebot-Extended, Bytespider, Meta-ExternalAgent, CCBot, Amazonbot, Cohere, Diffbot, and more.
Does BotLens collect any data?
The Chrome extension collects no data — analysis runs entirely in your browser with no telemetry or external servers. This marketing site uses privacy-friendly Umami analytics for anonymous page counts (no cookies, no personal data). Full privacy policy.
How is BotLens different from a normal SEO tool?
Traditional SEO tools focus on Googlebot and Bingbot. BotLens is built for the AI-native web — it detects blocks against LLM crawlers like GPTBot and ClaudeBot, flags noai and noimageai directives used for AI training opt-outs, and identifies SPA shells that hide content from bots that don't execute JavaScript.
What is GEO (Generative Engine Optimization)?
GEO is the practice of optimizing content so it can be discovered, understood, and cited by generative AI systems like ChatGPT, Claude, and Perplexity — as opposed to traditional SEO which targets search-result rankings. GEO cares about the same technical foundations (semantic HTML, structured data, crawler access) plus new signals like whether you've opted in to AI training via robots.txt and meta directives.
Is BotLens free?
Yes. BotLens is 100% free, open-source under the MIT license, and requires no account. Source code is on GitHub.
Does BotLens work on other Chromium browsers?
BotLens is built on Manifest V3 and should work on any Chromium-based browser: Chrome, Edge, Brave, Arc, Opera, and Vivaldi. Firefox support is on the roadmap.
Ready to optimize for the AI-native web?
Install BotLens and start analyzing any site in one click.
Get the Extension