Bots have always had a significant presence on the internet, but their activity has exploded with the rise of new AI bots.
According to a study by Imperva, a cybersecurity firm, bot traffic has surpassed human traffic, accounting for 51% of web traffic.
And they’re not alone in their assessment.
Human Security reported that AI traffic nearly tripled in 2025 and is growing eight times faster than human traffic, and Matthew Prince, CEO of CloudFlare, just claimed in an interview that AI bot traffic alone will exceed human traffic by 2027.
Right now, your head is probably spinning, so let’s break this down to understand exactly what these bots are, what they’re doing, and how it will affect your website.
Good bots vs. bad bots
When you hear that bots are visiting your site, you might be alarmed at first, but not all bots are bad. According to Imperva, “good bots” account for 14% of web traffic.
So what exactly is a good bot?
What is a good bot?
A good bot self-identifies and plays by the rules.
An example would be Googlebot, which has crawled the web for decades, finding new pages for Google to index. It lets you know in its User Agent string that it is Googlebot, and if you add a directive to your robots.txt file telling it not to crawl your site, it will listen.
Another important factor is that while Google is building a business around crawling your pages, you get value from their bot traffic as well, because it allows your site to show up in their search results.
What is a bad bot?
A bad bot does not self-identify or play by the rules.
Unlike Googlebot, which tells you, “Hey, I’m Googlebot!” A bad bot says, “I’m a human being.” They disguise themselves as human visitors to blend in with other traffic and avoid identification. Even if you did update your robots.txt file to tell them not to crawl your site, they wouldn’t listen anyway.
Bad bots do not provide any value for you, and instead, exist purely to extract value from your website. For WordPress websites, most of the bad bot traffic you get is referrer spam, SEO manipulation, or vulnerability scanning. In other words, bad bots are trying to advertise to you, add links to your site, or hack into it.
And that leaves us with the big question: are these new AI bots good bots or bad bots?
Are AI bots good bots or bad bots?
To answer this question, we first need to understand exactly what we mean by “AI bots.”
What is an AI bot?
There are three distinct kinds of AI bots.
Crawlers: First, there are crawlers used by the AI platforms to gather training data. An example would be GPTBot.
Fetchers: Second, there are AI bots that fetch data from the internet when users ask AI a question it can’t answer with its training data alone. An example of this is the ChatGPT-User bot.
AI agents: Lastly, there are AI agents, which are deployed by AI users to conduct research and complete tasks online. They do not self-identify and disguise themselves as human visitors.
While crawlers accounted for the majority of AI bot traffic in early to mid-2025, things are changing rapidly as the popularity of AI agents grows. This can be partly attributed to tools such as OpenClaw, a free, open-source AI agent launched in November 2025.
When experts say AI bot traffic is exploding in 2026, they are referring to AI agents, in particular.
So are AI bots good or bad?
Most of the research cited in this post comes from security firms, so they view AI traffic as nefarious, but the truth is much more complex.
To answer this question for yourself, you have to ask, “What’s in it for me?”
AI crawlers will use up bandwidth and can be especially aggressive in how they crawl your site. However, having a presence in their training data can be very valuable. This could cause a platform like ChatGPT to recommend your product when someone describes the problem that it solves.
AI fetchers may request numerous pages from your site at a time, but it is likely because someone wants to learn more about your product or service, perhaps after a recommendation from an AI tool.
AI agents sound like bad bots because they disguise themselves as human visitors, yet they also perform human tasks, making them extremely hard to classify as good or bad.
For example, an AI agent might visit your eCommerce store and buy a product! This happens because a user tells their agent, “I’m out of product XYZ. Can you order more for me?” It might sound strange, but people are adopting this technology and using it in these ways.
As you can understand now, AI bots are not strictly good or bad. Your assessment depends largely on the nature of your website. On a product site, AI bots could drive greater brand awareness and sales, while on a content site, they might summarize articles in-app, reducing readership and ad clicks.
Can we identify good AI agents vs. bad ones?
What might seem obvious is that AI agents need a way to identify themselves so that you can allow the good ones while blocking the bad ones. However, it’s not that simple because a bad agent could always lie and say it’s a good one. There are plenty of smart people working on this issue, so hopefully we’ll get some solutions this year.
If you want to block AI bots from accessing your site, there are steps you can take.
How can I block AI bots from accessing my website?
While you won’t be able to block all AI bots, you can block most of the crawlers.
First, you can use your robots.txt file. This is the “polite” way and is more like telling them your preferences, rather than outright blocking them. It’s a good first step, and you can block some but not others.
Next, for bots that don’t respect your robots.txt directives, you can use a firewall. CloudFlare is excellent for this. They offer an option to block AI crawlers, which you can toggle on to enable. It identifies bots not only by their User-Agent strings but also by their behavior.
This tool in CloudFlare is designed to block AI crawlers from accessing your site, but you could also block fetchers by manually configuring your firewall rules e.g. block requests when the User Agent string contains “ChatGPT-User.”
When it comes to AI agents, they are extremely hard to block because they can bypass CAPTCHAs and other bot detection technologies. While a few of the largest sites in the world, like YouTube and Reddit, are having some success, small website owners like us basically don’t stand a shot at blocking AI agents right now. The internet will need to evolve.
Now, let’s get to the burning question: are all these bots ruining your stats?
Are these bots messing up my analytics?
You might have been thinking this whole time:
“My stats say I had 5,000 visitors last month. Were most of those bots??”
Thankfully, no.
Most analytics tools ignore “good bots” because they self-identify and are easy to recognize. At Independent Analytics, we use an open-source library developed by Matomo to recognize hundreds of different bots.
This bot detection allows us to identify both AI crawlers and fetchers, so they can be ignored and excluded from your analytics. This way, they can still access your site – blocking them is up to you – but they don’t skew your stats. Your analytics remain a faithful representation of what real human visitors are doing on your website.
However, AI agents are complicating things now, and we’ll be seeing more of this throughout 2026. Since they are so effective at disguising themselves as human visitors, they can bypass firewalls and bot-detection technologies and appear in your stats as real visitors.
Again, this is confusing because many of them are bad, while some are truly good. In other words, some are here to hack your site, and others are submitting genuine questions through your contact form.
Truthfully, AI agents probably make up a tiny fraction of your traffic right now, as so much of it is aimed at the world’s largest sites, but this will be changing over time.
Once the internet establishes new standards, the “good” AI agents will be able to identify themselves, and then tools like Independent Analytics can start labeling traffic from these AI agents separately from human visitors.
Right now, there is an arms race, and the AI agents are currently winning. The internet is in transition, and 2026 may be the pivotal year for shaping the future of the web, where both humans and AI agents visit and use our websites alongside each other.
