Your website might be telling ChatGPT to go away

This week we found that our own website had been asking ChatGPT and Claude not to read it. Not through anything we wrote. Through a setting in our hosting dashboard that we had never switched on, which had quietly added its own instructions to a file we thought we controlled.

We found it by accident, while setting up something unrelated. There was nothing on the site to see. Every page looked the same before and after, so there was no version of paying attention that would have caught it.

How do I check my own site?

Type your website address into a browser and add /robots.txt on the end, so yourbusiness.com.au/robots.txt. You will get a page of plain text. Read down it for the word Disallow, and note which names sit above it.

If Disallow appears under GPTBot, ClaudeBot or Google-Extended, those are ChatGPT, Claude and Google’s AI answers, and your site is asking all three to stay out. The check takes a minute. You do not need to install anything or log in to anything.

What is that file, and why does it decide anything?

Robots.txt is a plain text file at the root of nearly every website. It tells automated visitors which parts of the site they may look at. Search engines read it before they read anything else, and the AI assistants now do the same.

It has been part of how the web works since 1994. Most business owners have never opened theirs, which is reasonable, because until recently nothing much turned on it.

Why would my website be blocking them?

Usually because your hosting company turned it on and told you in a changelog you did not read. Cloudflare, which sits in front of a large share of the web, has a setting called Managed robots.txt. When it is on, Cloudflare writes its own instructions into the file, and its default list blocks the crawlers that feed AI models.

That default is defensible. A lot of businesses do not want their words used to train someone else’s product, and the setting exists because people asked for it. If that is your position, the setting is doing exactly what you want.

Ours was on. We had not switched it on and we had never weighed it up. The file we wrote by hand said one thing, the file being served said the opposite, and both sets of instructions sat in there at once — which one won depended on whose software was reading.

Does being blocked actually cost me customers?

We do not know yet, and neither does anyone selling you a number for it. There is no reliable figure for how often somebody asks an assistant to recommend a plumber in Brisbane instead of searching for one.

What we can tell you is what being blocked changes, which is different from ranking badly. If a crawler cannot read your site, you are not further down the answer. You are not in it. The assistant recommends somebody else and nobody tells you it happened.

For plenty of businesses that is an acceptable trade. If your work comes through a shopfront, a trade counter or word of mouth, there are better uses of an afternoon than this. We would rather say so than talk you into caring.

In our case it was absurd. Being found by AI assistants is one of the four things we sell.

What we changed, and what we kept

We turned the platform setting off, so the file we wrote is the file that gets served. Then we went through Cloudflare’s version and kept the parts doing real work. One was the declaration that our writing should not be used for training. The other was a block on five scrapers that take content and never send a reader back.

That distinction did most of the work for us. A crawler that can quote you and name you is worth letting in. A crawler that takes the words and returns nothing is not. Both get called “AI bots” and they are not the same trade at all.

So what should I actually do?

Open the file. If it says something you did not decide, decide it now, either way.

Then there is the part that cost us the afternoon. Our copy of that file was correct the entire time. We could have read it every morning for a year and learned nothing, because the problem only existed at the address our visitors were hitting.

Whatever runs your website, something sits between your files and the person looking at them, and it is allowed to change what they get. Platforms add headers, rewrite links, insert scripts, and as we found, edit robots.txt. Most of it is helpful and none of it is visible from the inside.

So check the live address rather than the file on your computer. We check this on every site we build now. We found it on our own because we went looking, which is the only reason anyone finds it.