Artificial Intelligence

Is Your Firm Accidentally Invisible to AI?

In plain terms

A firm can be missing from ChatGPT and Perplexity answers because its site blocks the crawlers those assistants use, and the article says the cause is often not the content. The block usually comes from robots.txt rules, Cloudflare’s AI-crawler settings, the host, security plugins and firewall rules, or geographic and IP blocking. Content that only appears after JavaScript runs is a second way to be unreadable.

Training crawlers such as GPTBot and search crawlers such as OAI-SearchBot are controlled separately, and OpenAI’s documentation says a site that opts out of OAI-SearchBot “will not be shown in ChatGPT search answers.” It ends with a twenty-minute audit: read robots.txt, check Cloudflare and host settings, confirm Google’s Search generative AI control has not been set to opt out, and load key pages with JavaScript disabled. None of this makes a firm visible, the author says. It only stops the firm being invisible.

Crawlers, robots.txt, Cloudflare, JavaScript, and the settings nobody remembers changing

By Dave Taillefer, Business Director, ICONA

Consider a hypothetical firm: eight lawyers, a well-built site, solid rankings for its practice areas in its city, a Business Profile with a few hundred reviews. A partner asks ChatGPT which firms in the city handle a particular kind of matter, and the firm is not there. Perplexity, same. The natural assumption is that the content needs work.

Often it does not. Often the answer is a line in a text file, or a toggle in a firewall dashboard that someone switched on in 2024 when "block AI bots" sounded like protecting the firm's intellectual property. That setting did protect it, in a sense. It also told the systems that now answer a growing share of legal questions to leave the firm out.

Before any conversation about content, structure or "optimizing for AI," the question is simpler: can these systems reach your site at all? This article is how to find out.

There is no such thing as "the AI bot"

The single biggest source of confusion is that the companies running AI assistants operate several different crawlers, for different purposes, and they are controlled separately. Three categories matter.

Training crawlers collect content that may be used to train future models. OpenAI's is GPTBot. Anthropic's is ClaudeBot. Google's is not a crawler at all but a robots.txt token, Google-Extended, that governs whether content from Google's regular crawl may be used to train and ground its Gemini models. Apple has Applebot-Extended, Common Crawl has CCBot, and there are others.

Search and retrieval crawlers build the indexes that assistants consult when they answer a question in real time. OpenAI's is OAI-SearchBot. Anthropic's is Claude-SearchBot. Perplexity's is PerplexityBot, which Perplexity says is used to surface sites in its results and "not used to crawl content for AI foundation models." Google's AI Overviews and AI Mode are governed by the robots.txt rules for Googlebot, the same crawler that has indexed your site for twenty years. Bing's index feeds Microsoft Copilot.

User-triggered fetchers go and read a specific page because a person asked the assistant to. ChatGPT-User, Claude-User and Perplexity-User fall here.

Purpose OpenAI Anthropic Google Perplexity Others
Training GPTBot ClaudeBot Google-Extended (token, not a user agent) — Applebot-Extended, CCBot, Meta-ExternalAgent
Search / retrieval OAI-SearchBot Claude-SearchBot Googlebot PerplexityBot Bingbot (Copilot), Applebot
User-triggered fetch ChatGPT-User Claude-User — Perplexity-User —

OpenAI's documentation is explicit that these are independent: a site can allow OAI-SearchBot in order to appear in ChatGPT search while disallowing GPTBot. It is equally explicit about the consequence of blocking the wrong one: sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." Anthropic's documentation says that disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."

So the blunt "block all AI" robots.txt lists that circulated in 2023 and 2024, many of which included PerplexityBot and, from mid-2024, OAI-SearchBot alongside GPTBot, did not just opt firms out of training. They opted firms out of being found.

Google is a special case

Two things about Google are worth stating precisely, because both are widely misreported.

First, Google-Extended does not affect Search. Google's documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal," and that robots.txt directives for Googlebot are "the control for site owners to manage access to how their sites are crawled for Search." AI Overviews and AI Mode are part of Search. If you block Googlebot to keep your content out of AI answers, you have removed yourself from Google.

Second, there is no AI-specific robots.txt directive for Google. Google's documented controls are the snippet and indexing controls that have existed for years (nosnippet, data-nosnippet, max-snippet, noindex) and, announced in June 2026 and available to every site worldwide since August 31, 2026, a property-level "Search generative AI control" in Search Console that opts an entire site out of AI Overviews, AI Mode and the generative AI features in Discover. Google says opted-out sites will not receive traffic or impressions from those features, and that the setting is not used as a ranking or inclusion signal elsewhere in Search. That is a legitimate choice for some publishers. For a law firm that wants to be found, it is a toggle to confirm has not been flipped.

Where the accidental block usually comes from

The robots.txt file. It is a plain text file at yourfirm.ca/robots.txt, and anyone can read it. Look for User-agent lines naming OAI-SearchBot, PerplexityBot or Claude-SearchBot, or a wildcard block that catches them, followed by Disallow: /. Remember that robots.txt is advisory. Google's own documentation notes its instructions "cannot enforce crawler behavior," and that a page disallowed there can still be indexed if other sites link to it. It is the right tool for saying "please don't"; it is not a security control.

Cloudflare. Cloudflare sits in front of a large share of the web, including many law firm sites, and its posture on AI crawlers has changed three times. In July 2024 it added a one-click "block AI scrapers and crawlers" option for all customers, including free plans; by mid-2025 more than a million customers had switched it on. In July 2025 it began asking every new domain at sign-up whether to allow AI crawlers, with blocking as the default. On July 1, 2026 it replaced the single toggle with three categories, Search, Agent and Training, each with a published definition, and announced that from September 15, 2026 new domains would block Training and Agent by default on pages that display ads, while Search remains allowed. Existing customers keep whatever setting they had. If your site is behind Cloudflare and nobody has looked at the bot settings since 2024, look now. Note also that a crawler serving both search and training purposes gets caught by the Training block.

Your host. Some managed hosts block at the server level. SiteGround, for example, blocks crawlers intended for AI model training by default and allows the ones used in user chat sessions; unblocking training bots requires a support request. WP Engine has said it applies "platform-wide rate limiting on certain high-impact bots"; Search Engine Land's reporting in May 2026 identified GPTBot and ClaudeBot among those affected. Other hosts do not intervene. Ask yours, in writing, what they block or throttle.

Security plugins and firewall rule sets. Generic "bad bot" lists and aggressive rate limiting can throttle legitimate crawlers as a side effect. I have not found evidence that the common WordPress security plugins block AI crawlers by default, so do not assume they do; check your logs instead.

Geographic and IP blocking. Firms sometimes block traffic from outside Canada to reduce form spam. AI crawlers do not originate from Canadian IP ranges; check the published lists before you geo-block.

The JavaScript problem

Even with the door open, there is a second way to be unreadable. In December 2024, Vercel and MERJ published an analysis of crawler behaviour across Vercel's network and found that none of the major AI crawlers rendered JavaScript. OpenAI's and Anthropic's crawlers fetched JavaScript files (11.5 and 23.8 percent of their requests) but did not execute them, and Perplexity's, Meta's and ByteDance's did not render either. Googlebot and Applebot did. No AI vendor has claimed otherwise since, and Google's 2026 guidance still makes a point of saying that it processes content within JavaScript "as long as it isn't blocked," a statement it can make because Google renders and, on the available evidence, most others do not.

For a law firm site this matters in specific places: lawyer biographies loaded into the page after it arrives, practice-area content hidden in tabs or accordions populated by script, "load more" buttons on article listings, cookie or consent walls that hide the page until a script runs, and entire sites built as single-page applications. To Googlebot these look fine. To the crawlers feeding ChatGPT and Perplexity, the page may be a header, a footer and nothing in between.

The test costs nothing. Open a key page with JavaScript disabled in your browser, or view the raw page source rather than the rendered inspector, and read what is actually in the HTML. If your lawyers' credentials and your practice descriptions are not there, they are not there for those systems either.

Related: content locked in PDFs (fee guides, intake packages, process explainers) is a weaker bet than the same content on an HTML page. A PDF has none of the headings and structure that retrieval systems favour, and whether a given AI crawler parses PDFs at all is not something any of them document. If the substance matters, it belongs in HTML on a page.

A word on llms.txt

You will be pitched llms.txt, a proposed file that describes your site to language models. It was proposed in September 2024 by Jeremy Howard of Answer.AI. In June 2025 Google's John Mueller wrote that no AI system currently used it. In June 2026 Google's documentation said outright that Google Search ignores the file and that maintaining one "will neither harm nor help" a site's visibility. I have not found OpenAI, Anthropic, Perplexity or Microsoft stating that their systems read it. Creating one is harmless. Paying for one as a strategy is not a strategy.

Should you block training crawlers?

This is a legitimate choice and I do not think there is one right answer. Blocking GPTBot and ClaudeBot keeps your content out of future training sets, to the extent those crawlers honour robots.txt, and has no direct effect on whether ChatGPT or Claude can find and cite you today, because that runs through OAI-SearchBot and Claude-SearchBot. Google-Extended is the exception: Google says it also governs grounding in the Gemini app, so blocking it can affect whether Gemini draws on your pages, even though it does not touch Google Search, AI Overviews or AI Mode. Some firms will decide their content should not train commercial models, and that is a reasonable position. What is not reasonable is blocking the search crawlers by accident on the way to blocking the training ones.

Two caveats. Robots.txt is voluntary, and compliance is contested: in August 2025 Cloudflare published evidence that Perplexity was using undeclared crawlers to fetch pages that had blocked its declared one; Perplexity disputed the finding and attributed the traffic to a third-party browser service. And the volume is real: Cloudflare reported in July 2026 that 52 percent of the crawler requests it saw were for AI training, up from 22 percent in the spring of 2025, which goes some way to explaining why hosts have started blocking on their customers' behalf. None of that changes the recommendation. Decide deliberately, and allow the retrieval crawlers.

A minimal robots.txt that allows search and blocks training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Drop the last three blocks if you have no objection to training. The three Allow groups only do work if a wildcard block (User-agent: *) exists elsewhere in the file; crawlers follow the most specific group that names them. Do not add anything for Googlebot.

The twenty-minute audit

  1. Read yourfirm.ca/robots.txt. Note every user agent named and what it is allowed to do.
  2. If you use Cloudflare or another edge provider, open its bot or AI-crawler settings and write down what is blocked.
  3. Email your host and ask what they block or rate-limit at the server level.
  4. In Google Search Console, confirm the Search generative AI control has not been set to opt out.
  5. Load your three most important pages with JavaScript disabled. Confirm the substance is present.
  6. Ask whoever manages your server for a log extract filtered to OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot for the past thirty days. Zero visits from a bot you have not blocked is its own finding.
  7. Verify any suspicious bot traffic against the published IP ranges (OpenAI, Anthropic and Perplexity all publish JSON lists), since anyone can claim a user-agent string.

None of this makes you visible. It only stops you being invisible. But every hour spent on content, structure or schema is wasted until the door is open, and the door is closed more often than firms expect.

Sources

Filed under Artificial IntelligenceSEO
Related service Get found in search and AI answers

More in Artificial Intelligence

View all 20
Twenty-Five Years of "Search Is Dead" How to Check Whether AI Actually Mentions Your Firm Why Ranking #1 No Longer Guarantees New Clients