Traffic Up, Sales Flat? Should You Block Every AI Bot?
Has your website traffic suddenly jumped, while inquiries and sales stayed flat?
Look at who is behind those numbers. The increase may be automated programs collecting your content. On September 30, 2026, Cloudflare reported that daily AI agent requests across its network had grown by more than 1,700% over the previous year.
Those requests do not automatically become purchases. Repeated collection consumes server resources and can increase costs on usage-based hosting.
AIB experienced this firsthand. On the evening of September 17, 2026, Korea time, an automated program swept through our site. Total daily requests rose from roughly 290,000 the previous day to 1.09 million, while daily infrastructure costs reached 3.1 times the previous day's level. Another automated traffic pattern appeared in October, prompting a firewall response.
Not all that extra traffic was potential customers. Should we block every bot?
Search, Training, and Spam
Search engines and AI systems need access to your content to discover it and potentially recommend it to customers. Excessive collection, however, adds costs. Ask what a request is doing, not just whether it comes from a bot.
Start with three categories.
| Category | What it does | Your decision |
|---|---|---|
| Search | Collects pages for search results and sources in AI answers | Keep relevant public pages accessible |
| Training | Uses collected content to develop or improve AI models | Decide whether to permit training use |
| Spam-like collection | Creates unwanted load through repetitive requests, excessive scraping, or unauthorized copying | Apply rate limits and firewall controls |
Search and training describe purposes; spam-like collection describes behavior. Even a search crawler may need rate limits if its requests overload your service.
Major Bot Controls
If you want to decline training use, ask your website administrator: “Keep search access to public pages, and apply the relevant training opt-outs below.” A robots.txt file, placed at your site's root, tells crawlers how they may access content.
| Service | Search or user-requested access | Training control | What to ask your administrator |
|---|---|---|---|
| Googlebot | Google-Extended | Add Disallow: / under Google-Extended in robots.txt. Preserve Googlebot access to public pages | |
| ChatGPT | OAI-SearchBot / ChatGPT-User | GPTBot | Add Disallow: / under GPTBot. Preserve OAI-SearchBot access. robots.txt rules may not apply to ChatGPT-User |
| Claude | Claude-SearchBot / Claude-User | ClaudeBot | Add Disallow: / under ClaudeBot. Separately preserve search and wanted user-requested access |
| Perplexity | PerplexityBot / Perplexity-User | Neither is used for foundation-model training | Preserve PerplexityBot for search visibility. Perplexity-User generally does not follow robots.txt |
| Bing / Copilot | bingbot | NOARCHIVE meta tag | Set NOARCHIVE on pages to decline AI-answer and training use while retaining ordinary Bing search. This also excludes the content from Copilot answers. If NOCACHE is also present, Bing treats the page as NOCACHE, so check existing tags. |
| Naver, a Korean search platform | Yeti | No separate training control verified in its official guidance | Keep public pages accessible to Yeti and check crawling in Naver Search Advisor |
| Apple | Applebot | Applebot-Extended | Add Disallow: / under Applebot-Extended. Preserve Applebot access for search |
| Common Crawl | CCBot, which collects a public web dataset | CCBot | Add Disallow: / under CCBot if you do not want your content collected for the dataset |
Google-Extended controls both Gemini training and some uses of content to ground answers. It does not control inclusion in Google Search. Preserve your existing restrictions on private paths when making changes.
robots.txt gives instructions; a firewall enforces access restrictions. Use firewall rules and rate limits for repetitive requests that ignore your instructions. Verify the requester using official IP information or other supported checks before deciding whether to allow access.
In Cloudflare's Training settings, Block also stops search access by Googlebot, Bingbot, and Applebot. Distinguish it from Disallow AI Training if you want to preserve search. Bing's support for a robots.txt training opt-out is targeted for early 2027, so use the NOARCHIVE setting separately for now.
AI Agents Got Blocked Too
Named crawlers are only part of the picture. Some AI agents operate browsers on people's behalf. They compare products, check prices, and find contact pages.
AIB blocked spam-like traffic from Linux environments and found that some AI agents, including Muse, could no longer access our site. Reopening access was the difficult part. Those requests lacked identifying information that would let us distinguish a particular agent and selectively allow it.
A barrier against spam can also stop a customer's AI agent.
Look at request frequency, behavior, and verifiable identity, rather than treating an operating system as proof of abuse. After changing blocking rules, test whether the AI tools your customers use can still view products and reach your inquiry pages.
From SEO to NEO
The business reason to allow discovery is to help customers find you. These acronyms become clearer when translated into practical questions.
| Term | The practical question | Where to start |
|---|---|---|
| SEO | Can Google crawl, index, and show our pages in search? | Check crawling and indexing, then align titles and content with customer questions |
| AEO | Do AI answers in search cite our information? | Provide direct answers and verifiable supporting evidence |
| GEO | Do ChatGPT and Perplexity use our original material as a source? | Publish original data, firsthand examples, and current information on readable pages |
| LLMO | Does AI describe our brand and services accurately? | Keep official descriptions, product names, prices, and facts consistent |
| NEO | Can audiences find us in Naver search and AI Briefing? | Check Naver crawling and the quality of the source content |
Here, NEO is simply a label for visibility on Naver, the Korean search platform. It is particularly relevant when reaching customers in Korea. These five areas overlap. Rather than chasing five separate tricks, publish accurate answers to customer questions and keep the relevant search routes open.
An AI system retrieving and citing your page is different from a model being trained on it. Allowing training does not guarantee that a model will remember or recommend your brand. Crawl access is a starting point, not a promise of citations or rankings.
Check Costs and Customer Traffic
Ask your administrator to check three things.
- What increased? Compare visitors, inquiries, and purchases with server requests, data transfer, and costs. More requests do not necessarily mean more customers.
- Who is visiting, and why? Separate search, training, and spam-like collection. Include AI agents acting on people's behalf.
- What changed after blocking? Check both cost reductions and continued access for search crawling, AI visits, and inquiries.
For the website itself, see Why Isn't Your Business Showing Up in AI Search? 5 Website Checks.
What matters is who visits and what results follow, not traffic volume alone. Limit collection that drives costs while preserving the routes customers use to find you.
Deciding where AI fits into your business? Start with your workflows and budget. Contact AIB.
Sources and References
Bot controls reflect official documentation checked on October 20, 2026, Korea time.
- Traffic trends: Cloudflare, The Internet has a second audience
- Blocking and training preferences: Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training
- Google: List of Google's common crawlers · Optimizing your website for generative AI features on Google Search
- OpenAI: Overview of OpenAI Crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity Crawlers
- Microsoft: Announcing new options for webmasters to control usage of their content in Bing Chat
- Naver, Korean-language documentation: robots.txt 설정하기 · AI 브리핑 소개
- Apple: About Applebot
- Common Crawl: CCBot
- AIB operational records: requests and costs are whole-day totals for September 16 and 17, 2026, in UTC. The October response and the Muse access incident are AIB's firsthand operational experiences.