Should You Allow GPTBot, ClaudeBot and PerplexityBot?

OpenAI documents GPTBot for potential model training, OAI-SearchBot for search inclusion and ChatGPT-User for user-triggered visits. Anthropic separately documents ClaudeBot, Claude-SearchBot and Claude-User. Perplexity distinguishes its search crawler from a user-triggered fetcher.

Those categories matter because a publisher may want public pages discoverable in answer search while declining training use. Names and behavior can change, so copy rules from the provider's current official documentation rather than an old crawler list.

The following policy allows OpenAI and Anthropic search crawlers while declining their training crawlers. It is an example, not a recommendation for every publisher.

User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: /

Validate syntax after editing and test the exact public URLs that matter. A broad rule for another user agent can still affect access depending on matching behavior.

robots.txt is a public set of instructions, not authentication. Do not use it to protect customer records, staging sites, documents or paid content. Put sensitive material behind suitable authorization and remove it from public links and sitemaps.

A user-triggered fetcher may be treated differently from an automated indexing crawler. Perplexity, for example, documents that Perplexity-User is tied to user requests and generally ignores robots.txt. Review provider controls and legal or contractual options for your context.

A business seeking visibility in AI-assisted search may choose to allow documented search crawlers to its public marketing and knowledge pages. A publisher with licensing, capacity or training-use concerns may choose narrower access.

Do not make the decision from server traffic alone. Consider content rights, commercial value, infrastructure cost, jurisdiction and whether the relevant answer product sends useful referrals. Revisit the rules when the providers update their documentation.

Consider a software company that wants its public help centre and product pages available in answer search but does not want a training crawler collecting those pages. It could allow the providers' documented search agents, disallow the documented training agents, and keep customer portals protected with authentication regardless of crawler rules.

The company should record the decision beside the rule: agent name, documented purpose, source URL, owner and review date. That record prevents a later developer from collapsing several carefully chosen rules into one broad wildcard because the reason is no longer visible.

After deployment, fetch robots.txt from the production hostname, test representative URLs with the provider-specific agent where practical, and inspect server or CDN logs. A rule in source control is not enough if an edge firewall blocks the same agent or a different robots file is served in production.

A common mistake is copying a list from a blog without checking the provider's current names. Another is assuming that a rule for one agent automatically covers the provider's search, training and user-triggered agents. The official documentation distinguishes those roles, so each relevant agent needs an intentional decision.

Syntax can also undermine the policy. Teams put an Allow or Disallow line under the wrong User-agent group, publish the file on one hostname but test another, or forget a broad wildcard rule higher in their mental model. Robots matching behavior should be tested rather than guessed.

The most serious mistake is using robots.txt as a privacy boundary. The file itself is public, and cooperative crawlers are not the only way a URL can be requested. Authentication, authorization and appropriate data storage remain necessary.

Ask whether the content is yours to expose, whether discovery supports the business model, whether crawling creates meaningful infrastructure cost, and whether a contract or regulation changes the decision. Separate public marketing content from licensed, user-generated or customer material because one rule rarely fits all of them.

Also decide how you will measure the result. Search referrals, citations, crawl logs and server load answer different questions. A change that increases crawling without useful discovery may need revision; a lack of immediate referrals does not prove the agent ignored the site.

This approach is not for someone seeking a universal allow-list. Publishers have different rights and commercial goals. It is also not a replacement for legal advice when content licensing, privacy or contractual duties are involved.