← All posts

Block AI Crawlers? Why I Kept My Sites Open

By Build Log

TL;DR: Today an audit of my sites showed that none of them block AI crawlers, the programs AI tools use to read public web pages. One hosting setting that can add AI-bot blocks to a site automatically had been on for one of my sites, and by this evening it was off. Then I asked why so many owners block AI and what I was missing. The risks are real but small for a tour business. I'm keeping my sites open.

What happened today

Our team ran a read-only audit of my sites. It looked for anything that tells AI crawlers to stay away. It found no active block. It did find one hosting setting that can add AI-bot blocks to a site's robots.txt file without anyone writing them. It was on for one site and was not adding anything. When the team checked again this evening, it was off and no site was blocking AI crawlers. I had already told the team not to block AI training bots on any site.

I'm not a pro at this, and I had to ask. That is part of the story. I'm Kay Honig, I run a tour company, and I am not a web expert, so I did not know what the consequences were. So I asked, in plain words: "Why are many owners blocking AI? What am I missing?"

Why many owners block AI

There are four reasons I found:

  • The content is the product. Publishers sell articles. They don't want AI to learn from them or answer in their place for free.
  • No credit and no clicks. An AI can summarize a page, and the reader never visits the site.
  • Copying. Some owners block bots to stop others from lifting their text or prices.
  • Server load. Heavy bot traffic can slow a small site.

A tour company is not in that position. Our pages exist so people find us and book. Being quoted is the point.

What I give up

I also asked: "What am I giving up? Maybe I am just too naive and do not know what the consequences are." Here is the honest list.

  1. Exclusive use of my words. AI companies can learn from my public pages and nobody pays me for it. But anyone can already read and copy a public page. Nothing private is exposed.
  2. Some clicks. An AI may answer "what's included in a tour?" from my page, and the traveler never visits. Some AI answers link the source, many don't. A traveler who wants to book still needs a company to book with.
  3. Control over what it says. An AI can repeat old or wrong wording. That risk exists whether I block it or not, because it also learns from reviews and other sites. The fix is keeping my own pages accurate and dated.

Bookings, guest emails and anything behind a login are not affected. The setting only touches public pages.

What I'd lose by blocking

More travelers now ask AI tools travel questions, and some start there. "Which Cancún tour company can I trust?" is a real question. If AI crawlers can't read my pages, the answer is built from other people's pages. For a company that sells on transparency, that is the wrong trade. You can read how we run Kay Tours Mexico on our own site.

The bots, in plain words

The big AI companies publish separate bots for training and for live search:

  • GPTBot is OpenAI's training bot.
  • OAI-SearchBot powers ChatGPT search.
  • Google-Extended covers Google's AI training only.

So a middle option exists. Google's AI answers in Search use its normal search crawler, so there is no clean opt-out without leaving search. I chose the simple version: everything stays open.

What I'd do differently

I should have checked this on day one. A setting nobody had looked at could have kept AI crawlers out of the sites using it. Next time I want that audit before launch, not after.

Questions people ask

Should a small business block AI crawlers? If you sell content, maybe. If your pages exist to be found and to earn bookings, blocking mostly hides you.

Does allowing AI crawlers expose private data? No. They only read public pages. Anything behind a login stays behind it.

What is the real risk? Outdated or wrong wording getting repeated. Keep your own pages accurate and dated.