How to Check If ChatGPT Can See Your Website
To check whether ChatGPT search can see your website, confirm that OAI-SearchBot is not blocked in robots.txt, the public page returns a successful response without authentication or a bot challenge, important content appears in the rendered HTML, and the URL is indexable and canonical. Then verify visibility with real search-backed ChatGPT prompts.
Start with the correct OpenAI crawler
OpenAI documents different user agents for different purposes. For inclusion in ChatGPT search summaries and snippets, publishers should allow OAI-SearchBot. GPTBot controls potential use of web content for model training and is a separate decision. Allowing or blocking one should not be treated as an automatic policy for the other (OpenAI Publisher FAQ).
A basic robots.txt policy for ChatGPT search discovery is:
User-agent: OAI-SearchBot
Allow: /
Do not paste that block blindly. Review existing wildcard and path-specific rules, then preserve restrictions for private, authenticated, checkout, account, or internal API routes.
Test 1: inspect robots.txt
Open https://yourdomain.com/robots.txt and search case-insensitively for:
OAI-SearchBotUser-agent: *Disallow: /- any disallow rule covering the page you want cited
Bot-specific rules normally take precedence over a wildcard block for that bot. A simple explicit allow can make policy intent clearer, but a more specific path rule may still restrict sections of the website.
Also inspect CDN and security settings. OpenAI notes that WAFs, bot mitigation, JavaScript challenges, CAPTCHAs, authentication, geo rules, and rate limits can block a crawler even when robots.txt allows it (OpenAI crawler guidance).
Test 2: verify the HTTP response
Request the exact canonical URL, not only the homepage. It should:
- return
200 OKafter reasonable redirects; - avoid redirect loops or region-dependent failures;
- serve an HTML content type;
- avoid a login, consent wall, or challenge as the main response;
- load consistently without intermittent 403, 429, or 5xx errors.
You can perform a simple command-line check:
curl -I -L https://example.com/important-page
This proves ordinary HTTP availability, not that OpenAI has indexed or selected the page. It is still the fastest way to uncover a basic access failure.
Test 3: check indexability and canonical signals
Inspect the HTML for:
<meta name="robots" content="index,follow" />
<link rel="canonical" href="https://example.com/important-page" />
An explicit index,follow value is optional because indexing is normally the default. The important checks are the absence of noindex, a canonical pointing to the intended URL, and no conflicting X-Robots-Tag HTTP header.
OpenAI states that a disallowed page URL may sometimes be surfaced only as a title and link when discovered through another provider or page. It also notes that a crawler must be allowed to access a page before it can read a noindex directive. That is why crawl policy and index policy should be reviewed together rather than treated as substitutes.
Test 4: verify that the useful content is actually rendered
View the page with JavaScript disabled or inspect the rendered HTML through a crawler or browser automation tool. Confirm that the main answer, headings, product facts, organization identity, and links are present as meaningful text.
An empty root element plus a large JavaScript bundle may still render correctly for users, but it increases the number of failure points for automated retrieval. Server-rendered or prerendered cornerstone pages provide a more resilient baseline.
Check that:
- the title and meta description are specific to the page;
- the H1 names the actual topic or product;
- the first substantive paragraph answers the core question;
- important facts are text, not only pixels inside an image;
- JSON-LD matches the visible content;
- internal links use crawlable
<a href>destinations.
Test 5: run a focused readiness audit
A crawler-access test only tells you whether a page can be fetched. Citation readiness also depends on whether the page is understandable and trustworthy once fetched.
Use the free AI visibility checker to inspect up to five public pages for Answer First structure, schema quality, E-E-A-T, information density, FAQs, HTML hierarchy, multimodal evidence, and section development. The published methodology explains how each category is calculated.
Test 6: verify actual ChatGPT search visibility
Access and visibility are not the same outcome. Once the technical checks pass, use a fresh ChatGPT session with web search available and test several queries that naturally relate to the page.
Use a mix of:
- a direct branded query;
- an unbranded category query;
- a problem-first buyer query;
- a comparison query;
- a factual question answered by the target page.
Save the prompt, date, answer, citations, and competing sources. Repeat the checks over time. One missing citation does not prove that the crawler cannot see the page; it may simply mean another source was more relevant or authoritative for that prompt.
Test 7: look for ChatGPT referral evidence
OpenAI says referral URLs from ChatGPT search include utm_source=chatgpt.com. Create an analytics segment for this parameter and the relevant ChatGPT referrer, then track landing pages, engagement, and conversions.
For the exact dimensions, channel rules, and funnel events, follow the GA4 and Plausible ChatGPT referral tracking guide.
No referral traffic does not prove invisibility: many answers do not produce clicks, and a newly accessible page may need time to be refreshed. Referral evidence is the business-impact layer, not a replacement for access logs or prompt observations.
Common false conclusions
“GPTBot is allowed, so ChatGPT search can see us”
Not necessarily. OpenAI documents OAI-SearchBot for search discovery and GPTBot for potential training. Check the correct user agent and the rest of the delivery stack.
“robots.txt allows the bot, so access is guaranteed”
No. A firewall, CDN challenge, authentication requirement, rate limit, or server error can still block retrieval.
“ChatGPT did not cite us once, so the page is invisible”
No. The query may not have triggered web search, the page may not be relevant enough, or another source may have been selected. Test a stable set of prompts repeatedly.
“We must allow training to appear in ChatGPT search”
OpenAI exposes separate controls. A publisher can make a deliberate policy decision for GPTBot while separately allowing OAI-SearchBot for search discovery.
FAQ
Which crawler should I allow for ChatGPT search?
OpenAI currently instructs publishers not to block OAI-SearchBot if they want content included in ChatGPT search summaries and snippets.
Is GPTBot the same as OAI-SearchBot?
No. OpenAI documents GPTBot as a control for potential model-training access, while OAI-SearchBot supports search discovery. Treat them as separate policies.
How can I know whether OpenAI actually crawled my page?
Inspect server or CDN access logs for verified crawler requests and compare them with robots.txt policy. A user-agent string alone can be spoofed, so use the verification information published by the crawler operator where available.
Does allowing OAI-SearchBot guarantee a citation?
No. It removes one access barrier. Citation still depends on relevance, indexing or retrieval, content quality, evidence, authority, and competition for the prompt.
How long does a robots.txt change take to affect ChatGPT search?
There is no guaranteed public refresh time for a specific page. Confirm the rule immediately, monitor logs, and measure prompt results over the following weeks rather than expecting an instant citation.
Sources
Youssef builds GEOCARA and has run visibility probes across AI engines since 2025. He writes from measured probe data, not speculation.
LinkedIn ↗Related GEO guides
Start your free trial
Audit your site and see how AI engines perceive you.