How do I show up in Claude's answers?

Updated July 31, 2026 8 min read
You probably got here asking
  • How do I show up in Claude's answers?
  • Why doesn't Claude ever mention my company?
  • Does Claude have its own search index?
  • Should I block ClaudeBot in robots.txt?
  • What is Claude-SearchBot?
  • Does Claude use Brave Search?
The short answer

Claude's web search runs on Brave Search, so showing up in Claude starts with being visible in Brave's independent 30-billion-page index, not Google's. Then set Anthropic's three robots.txt tokens deliberately: ClaudeBot (model training), Claude-SearchBot (Claude's own search index), and Claude-User (the live fetch that pulls your page into an answer). Blocking Claude-User is what most directly removes you from answers.

On this page
  1. What Claude actually does when it searches
  2. The three Anthropic crawler tokens, and what each one controls
  3. Claude's search index belongs to Brave, not Google
  4. You cannot block Brave with robots.txt, and you cannot see it in your logs
  5. How closely does Claude follow Brave's ranking?
  6. Why this deserves attention at Claude's current size
  7. Yahoo Scout gave Claude a second, much larger surface
  8. Your checklist for this week
  9. Frequently asked questions
Key takeaways
  • Claude's web search has been powered by Brave Search since it launched on March 20, 2025, and Brave runs its own index of 30 billion-plus pages with roughly 100 million daily updates rather than reselling Google or Bing.
  • Anthropic publishes three crawler tokens — ClaudeBot for training, Claude-SearchBot for search indexing inside Claude, and Claude-User for the live fetch that answers a specific question — and all three obey robots.txt.
  • Brave deliberately does not publish a differentiated crawler user-agent, so you cannot target it in robots.txt the way you target GPTBot, and you cannot confirm its visits in your server logs.
  • Per Brave's own help documentation, robots.txt does not prevent a page from being indexed in Brave Search; removal requires a noindex meta directive plus a re-fetch request at search.brave.com/submit-url.
  • Claude went from 0.30% to 2.91% of AI referral traffic between April 2025 and March 2026 on StatCounter's numbers, while First Page Sage's usage-based methodology puts it at 10.3% — the gap is a measurement difference, not a contradiction.

You rank on page one in Google for the thing you sell. Someone opens Claude, asks a question squarely in your category, and gets back three competitors and a forum thread. Nothing on your site is broken. Your pages are indexed, fast, and well linked. You are simply not in the index Claude is searching.

That last sentence is the whole problem, and it is the part almost nobody working on AI visibility has looked at. Claude's web search does not run on Google's index, and it does not run on Bing's. It runs on Brave's. And Brave behaves differently from every other search property you have ever configured — differently enough that the standard SEO playbook gives you the wrong answer twice.

What Claude actually does when it searches

Anthropic added web search to Claude on March 20, 2025. Before that, Claude answered entirely from training data. Now, when a question needs current information, Claude issues a search, gets ranked results back, fetches some of those pages live, and cites them in the answer it writes.

That sequence hides two separate control surfaces, and they are governed by two different companies. The first is whether Anthropic's own crawlers may touch your site — controlled through robots.txt, and clearly documented. The second is whether your pages exist in the index Claude searches over in the first place — controlled by Brave, and barely documented at all. Nearly everyone spends their effort on the first and never looks at the second.

The three Anthropic crawler tokens, and what each one controls

Anthropic publishes three user-agent tokens. They are not interchangeable, and the common instinct — block the AI crawlers, keep the search ones — maps onto them badly unless you read the descriptions closely.

TokenWhat it doesWhat blocking it costs you
ClaudeBotBulk crawler gathering data to, in Anthropic's words, “enhance the utility and safety of our generative AI models.”Your content is excluded from training. No effect on live answers.
Claude-SearchBotCrawls and indexes pages to improve search result quality within Claude.Your pages are less likely to surface well in Claude's own search layer.
Claude-UserLive fetch when a Claude user's query needs your specific page.Anthropic: blocking it “prevents our system from retrieving your content in response to a user query.”

All three respect robots.txt. Anthropic also documents support for Crawl-delay, which is worth knowing because the real motivation for blocking a crawler is usually bandwidth rather than principle. A crawl delay slows the request rate while leaving you in the answer path. Blocking does both at once, and most people only meant to do one.

robots.txt — three tokens, three separate decisions
# Training corpus. Blocking this does NOT remove you from answers.
User-agent: ClaudeBot
Disallow: /

# Claude's own search index. Keep open if you want to be found.
User-agent: Claude-SearchBot
Allow: /

# Live fetch for a user's question. This is the answer path.
User-agent: Claude-User
Allow: /

# Legacy token from third-party blocklists. Not in Anthropic's
# current documentation. Harmless to keep, do not rely on it.
User-agent: anthropic-ai
Disallow: /

# If bandwidth is the real concern, rate-limit instead of blocking:
# User-agent: ClaudeBot
# Crawl-delay: 10

Note the anthropic-ai line. That token appears in a great many copy-pasted blocklists and third-party crawler trackers, but it is not in Anthropic's current published documentation. Treat it as legacy: keeping it costs nothing, relying on it to control anything is a mistake.

Claude's search index belongs to Brave, not Google

Anthropic's subprocessor list formalizes the arrangement: Brave Search is the retrieval backend behind Claude's web search. This is not a white-label of somebody else's results. Brave runs its own crawler and its own index — more than 30 billion pages, with roughly 100 million updates a day. It is not a Bing reseller and it is not a Google reseller.

That independence is the entire reason a site can be strong in Google and functionally invisible in Claude. Two crawlers made two different sets of decisions about your pages. Google's index may know your site intimately while Brave's holds a thin, stale or partial version of it. Nothing in Google Search Console will warn you, because it has no visibility into Brave.

You cannot block Brave with robots.txt, and you cannot see it in your logs

Here is where the SEO mental model breaks outright. Every other crawler in this space publishes a token you can name and address: GPTBot, Googlebot, PerplexityBot, ClaudeBot. Brave deliberately does not publish a differentiated crawler user-agent. The stated reason is to stop sites from special-casing its traffic and serving the crawler something different from what a person would see.

Two consequences follow, and both break something you probably assumed was working:

  • You cannot write a robots.txt rule that targets Brave. There is no token to address. A generic User-agent: * group is the only thing that reaches it, and per Brave's own documentation that will not do what you expect anyway.
  • You cannot verify Brave crawling by user-agent in your access logs. There is no string to grep for. The log audit you would run to confirm GPTBot or Claude-SearchBot activity has no equivalent here.

The inverse matters more for most readers. If your only defense against AI crawlers is a robots.txt blocklist, you have not touched the index Claude actually searches. And if you have been assuming that a Disallow rule somewhere is keeping you out of AI answers, it very likely is not.

How closely does Claude follow Brave's ranking?

This is the open question, and the honest answer is that we have one third-party data point rather than a body of evidence.

A citation-overlap study published by rivalhound compared what Claude cited against what Brave returned organically and found that Claude's cited results matched Brave's top organic results 86.7% of the time. The same study put ChatGPT's alignment with Bing at 26.7%. The proposed explanation is that Claude re-ranks Brave's output comparatively little, while ChatGPT applies much heavier post-processing to whatever Bing hands it.

Attribute that carefully. It is a vendor study, not a peer-reviewed finding, and neither its methodology nor its sample has been independently replicated. But the direction matters even if the number does not hold. If Claude really is passing Brave's ranking through with light re-ranking, Brave organic rank sits unusually close to a direct lever on Claude visibility — rare in a field where most claimed levers are three inferences removed from the outcome.

Why this deserves attention at Claude's current size

The counterargument writes itself: Claude is small. It is. The interesting parts are the shape of the curve and the disagreement about the level.

0.30% to 2.91%Claude's share of AI referral traffic, April 2025 to March 2026 (StatCounter)
10.3%Claude's US share on First Page Sage's usage-based methodology, July 2026
~250MUS users across the Yahoo properties where Scout, built on Claude, launched in 2026

StatCounter's March 2026 figures put Claude at 2.91% of AI referral traffic, up from 0.30% in April 2025 — roughly a tenfold increase in eleven months. It doubled inside a single month, from 1.37% in February 2026 to 2.91% in March. Over the same window ChatGPT dropped below 80% for the first time and Perplexity shed more than 40% from its peak.

First Page Sage's July 2026 report puts Claude at 10.3% of US usage. Both numbers can be correct, because they count different things. StatCounter measures outbound clicks from AI chatbots to websites — it only sees you when a user leaves the chatbot for a site. First Page Sage measures usage: how much people actually use the tool. A tool whose users read the answer and stay put scores low on the first and high on the second. Use StatCounter's number if you care about referral traffic, First Page Sage's if you care about being the answer someone acts on.

Keep the base rate in view either way. Chartbeat data reported in March 2026 found chatbot referrals still account for less than 1% of total publisher pageview referrals, even after triple-digit growth during 2025. Claude is a fast-growing slice of a channel that is still small. Both halves of that are true.

Yahoo Scout gave Claude a second, much larger surface

In January 2026 Yahoo launched Scout, an AI answer engine, in beta across roughly 250 million US users spanning Yahoo Mail, News, Finance, Sports, Search and Shopping. Scout runs on Anthropic's Claude as its foundation model, and it cites its sources.

That changes the arithmetic on the it-is-only-2.91% objection. Claude visibility no longer means only the people who deliberately open Claude. It now includes a mass-market surface inside properties tens of millions already use daily.

One honest caveat: Scout is not Claude's web search wearing a Yahoo skin. Yahoo pairs Claude with Microsoft's Bing grounding API for freshness and with its own knowledge graph. So Brave rank is not the lever for Scout the way it appears to be for Claude's own search. What carries across is everything about how Claude reads, weighs and represents your content once it has it in hand.

Your checklist for this week

  1. Go to search.brave.com and search your brand name, your main product terms, and your three most commercially important queries. Write down where you actually appear and which of your pages Brave has at all.
  2. Where Brave is missing pages or holding stale ones, submit them at search.brave.com/submit-url. That endpoint is also the only mechanism for pushing a noindex change through, so it is worth knowing in both directions.
  3. Open your robots.txt and make three separate decisions instead of one. Training (ClaudeBot) is a policy choice with no effect on answers. Search indexing (Claude-SearchBot) and live fetch (Claude-User) are the answer path — blocking either is a decision to be absent from Claude.
  4. If server load is your actual reason for blocking, set Crawl-delay rather than Disallow. Anthropic documents support for it, and it costs you nothing in visibility.
  5. Audit any AI-crawler blocklist you copied from a template. Check specifically whether it blocks Claude-User without your having intended that, and remember that it does nothing at all about Brave.
  6. Stop treating your Google rank as a proxy for Claude visibility. Different crawler, different index, different result. Check Brave directly, on a schedule, the same way you check your Google rankings.

None of this needs new content. It needs looking at a search engine you have never looked at, and reading three crawler descriptions closely enough to notice they do different jobs.

Frequently asked questions

Does blocking ClaudeBot remove me from Claude's answers?

No. ClaudeBot is Anthropic's bulk training crawler. The token that governs whether Claude can pull your page into a live answer is Claude-User, which Anthropic says prevents its system from retrieving your content in response to a user query when blocked. Claude-SearchBot governs indexing for search quality inside Claude. Blocking ClaudeBot alone leaves the answer path fully intact.

Can I see Brave's crawler in my server logs?

No. Brave deliberately does not publish a differentiated crawler user-agent, specifically so that sites cannot special-case its traffic. There is no string to search your access logs for and no published IP list for the purpose. The only practical check is to query Brave Search directly and see which of your pages appear and how they rank.

How do I get a page out of Brave Search entirely?

Per Brave's own help pages, robots.txt is not used to prevent a page from being indexed in Brave Search. You need a noindex meta directive on the page itself, and then you request a re-fetch at search.brave.com/submit-url so Brave sees the directive and drops the page. A Disallow rule on its own will not remove it, no matter how long you wait.

Is anthropic-ai still a valid robots.txt token?

It appears in many third-party crawler trackers and copy-pasted blocklist templates, but it is not in Anthropic's current published crawler documentation. Treat it as legacy. Leaving it in your robots.txt is harmless and costs nothing, but do not rely on it to control anything. The three documented tokens are ClaudeBot, Claude-SearchBot and Claude-User.

What does Crawl-delay actually do for Anthropic's crawlers?

Anthropic documents Crawl-delay as a supported directive, which gives you a middle option between full-speed crawling and an outright block. If your concern is server load rather than whether Claude can cite you, a crawl delay slows the request rate while keeping your content eligible to appear in answers. A Disallow rule solves the load problem and removes you from answers at the same time.

Does Yahoo Scout make my Brave ranking more important?

Not directly. Yahoo Scout, launched in beta in January 2026 across roughly 250 million US users, uses Anthropic's Claude as its foundation model but grounds freshness through Microsoft's Bing API and Yahoo's own knowledge graph. It is a Claude surface with a different retrieval layer. Scout raises the value of Claude visibility in general rather than the value of Brave rank specifically.

Sources

READY TO SEE THE SIGNALS?

Find out what the engines see on your site.

One audit scores SEO, AEO, and GEO separately, then hands you a tracked action plan instead of a PDF that gathers dust.

View plans