What the free check reads on your website

How the score is calculated

The score is the sum of the weights of the tests that passed, divided by the sum of the weights of the tests that could be measured, times 100 and rounded. Next to the number you'll see how many tests passed and a label: Good, Fair or Weak.

A worked example with an invented site: if it passes everything except llms.txt (15 points) and the sitemap (10), it earns 75 of 100 and scores 75.

You'll see the number, how many tests passed and, if any failed, the two heaviest ones with the evidence we measured. Any other failures still count toward the score even though they don't appear on screen.

The evidence is a fact from your files, not an opinion. If your robots.txt blocks a robot, we quote the first Disallow line in the file, which may not be the one that caused the block.

What a test that isn't measured means

Two tests can end up unmeasured. The speed test depends on a Google service: if our access key is missing, the service doesn't answer, returns no score or times out, we mark it as not measured. The AI crawler test goes unmeasured if your server doesn't deliver robots.txt (no answer in time, or a server error). Either way it leaves the calculation entirely. If it's llms.txt or the sitemap that doesn't arrive, the test counts the file as missing, because a search engine couldn't read it either.

We don't take points off for a problem on our side. Example: if speed couldn't be measured (10 points) and a site fails only llms.txt (15), the scale drops to 90: it earns 75 of 90 and scores 83. The count will read “of 7 checks” instead of “of 8”.

The robots.txt example: OAI-SearchBot and GPTBot

A site can have a robots.txt at its root, at yoursite.com/robots.txt. It says which robots may read what. Google puts it this way: it tells search engine crawlers which URLs they can access on your site.

These two lines block OAI-SearchBot, the robot OpenAI uses to surface websites in ChatGPT search:

User-agent: OAI-SearchBot
Disallow: /

According to OpenAI's documentation, sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.”

GPTBot is a different robot and a different decision: it crawls content that may be used to train OpenAI's models, and disallowing it indicates a site's content should not be used for that. “Each setting is independent of the others”: a site can allow OAI-SearchBot and disallow GPTBot.

The check treats all five robots the same: if any one is blocked, the test fails, GPTBot included. If you blocked it on purpose so your content isn't used for training, that's your call; the check flags it anyway, because it measures access, not intent.

If you fix the file, the check will see the change on its next run. According to OpenAI, its systems can take about 24 hours to adjust. And even with everything open, nobody can guarantee ChatGPT will mention your business.

What to do if a test fails

• AI crawler access. Open yoursite.com/robots.txt and look for Disallow: / under the robots the check names or under User-agent: *. If it wasn't your decision, remove it.
• llms.txt file. Publish a text file at yoursite.com/llms.txt with brief information about your business and links to your important pages. It's a proposal from llmstxt.org; nobody can guarantee an AI will use it.
• Structured data. Add JSON-LD data that describes your business (name, phone, services), in the HTML the server delivers and not added later with JavaScript.
• Title and description. Make sure each page has a title, a meta description and a single H1.
• Sitemap. Publish a sitemap.xml at the root of your site.
• Speed (PageSpeed). Open your page's PageSpeed Insights report and see what slows it down on mobile.
• Declared language. Declare the language on the <html> tag, for example lang="en".
• HTTPS. Install a certificate and redirect all traffic to https://.

What it doesn't measure

This is what we measure in your own site's files, not a prediction of what any search engine will do. So the check doesn't measure:

• Rankings. It doesn't tell you where your site shows up on Google or any other search engine.
• Mentions and citations. It doesn't know whether ChatGPT, Perplexity or Google have cited your business, and nobody can guarantee they will.
• What ChatGPT will say about you. We don't ask any AI assistant questions, so we don't measure its answers.
• The quality of your content. It only checks that the title and description exist, not that they're well written or persuasive.
• The rest of your site. It reads one page, the one you enter, and three files. It asks Google for the speed.
• What JavaScript does. It reads the HTML as the server delivers it and doesn't run scripts.
• Firewalls. It reads your robots.txt, not whether a firewall blocks a robot. For OAI-SearchBot, OpenAI also recommends allowing its published IP ranges, which this check can't see.
• Visits a person asks for. When someone asks ChatGPT to visit a page, OpenAI uses ChatGPT-User and says robots.txt rules may not apply. The check doesn't look at that.

How it works under the hood

When you enter a domain, the check requests that page, follows up to three redirects and reads the three files from the domain where it ended up. Each request to your site waits up to 8 seconds. It identifies itself as R21-AuditBot, in case you want to look for it in your server logs, and it only reads what any visitor can see. If it can't connect, it tells you so and shows no score. If you run several checks in a row, it asks you to wait a minute.