The big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infrastructure for their search index AND for training Gemini:
> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).
Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly.
That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
We had googlebot blast a random customer system and almost cause an outage, this is when I first learnt that google will use it for AI training also. It's honestly kind of frustrating also because you then search on it and theres (was) nothing on how you are meant to "correctly" tell google to fuck off, and not use it like that.
Google's web scraping functionality has been acting as a ddos for more than two decades. I've seen literally hundreds of reports of them attacking websites and taking them down, where there's nothing you can do but accept the traffic, or get delisted
This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this
Ideally yes, the TLS termination does not need to happen for caching purposes. Challenge is that in practice every business wants to be sticky and try to provide more functionalities which do require TLS termination. Most people either trust CDN's or they do not understand MitM so it does not concerns them. Plus they are getting certificate management and DDOS prevention capabilities.
> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.
It's kind of exhausting seeing Cloudflare playing both sides of the arms race.
I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this.
> This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.
And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.
I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests.
This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.
I see Cloudflare as trying to forge the appropriate path ahead. Neither allowing the free-for-all nor trying to block everything isn't playing both sides, it's the path straight down the middle. Providing tools for producers and consumers to do things with permission and compensation.
Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go this route.
PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.
>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites.
???
Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.
> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge.
You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge.
> If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
Cloudflare blocks legitimate users itself sometimes [1].
[0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward.
> That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.
Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges.
> I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
Yeah, I don't like the Cloudflare challenges, but in the past 5 years I've only had it outright block me once, and that fixed itself after 15 minutes. And I use Firefox on Linux with various privacy extensions, so my browser probably appears at least moderately suspicious.
Whereas I've been trapped in impossible ReCaptcha loops quite a few times, which is still better than vague error messages that magically go away when I switch to something not running Linux. So I'll begrudgingly accept that Turnstile is the least user-hostile product on the market right now.
I think the best answer is, nobody knows. The previous equilibrium for content scraping for search engines on the internet was already at times an uncomfortable one. But I agree that from a game theory perspective, "the AI bots take and give nothing back in return" is not just hyperbole, it's the actual situation. If Google is successful in what seems to be its plans and it becomes a box where you type a question and Google gives you an answer and only a vanishing fraction of the users click through to any underlying website, that instantly eliminates the entire value proposition for vast swathes of the web to actually be on the web.
Something has to happen or Google will end up starved and locked out of everything, by means both technical and legal. Then nobody gets anything.
I don't have the answer as to what happens next, and I doubt anyone else who proclaims one super confidently. But we can do some constraints analysis. There is no world where everyone works for free so Google and other AI engines can get all the value from the content, so we can eliminate those possibilities. I think we can safely discard the world(s) in which all content production just stops. However, off the top of my head, it's hard to get much tighter than that, and that definitely leaves a world where effectively everything everywhere ends up going pay-to-access.
Microtransactions have, to date, failed comprehensively, though, so the constraints on what "everything is pay-to-access" gets weird without them.
And there is never guarantee that there is any solution to any set of constraints. Things can end up overconstrained in reality as easily as a math problem. I don't actually think it'll go that way, but when analyzing this question I think it's important to not let "but $SOMETHING just has to have some way to work, because... uh... it has to!" Let the constraints do the talking. You could end up with a scenario where all content of any value is locked down, and it's fundamentally difficult and expensive to ever access or discover it, and consequently the entire content production industry radically contracts compared to its current size, if there is no pragmatic solution to microtransactions that is low-enough friction to get over the psychological and economic hurdles that have killed it to date. If everything is locked behind "macrotransactions" that's a much smaller commercial web. Probably a much higher quality one, too, but at a pretty stiff cost.
I get a mixed feeling about all this. Cloudflare is unilaterally making all these decisions which impact the whole internet traffic flow. Taking the lead is one thing, however decisions like this should have the direct involvement of Internet Engineering Task Force (IETF) to account for all stakeholders, otherwise we run into the situation of a fragmented internet
This is more of a way to create a unauthorized toll tax on highway. There is a problem indeed, however if the proposed direction by cloud flare is to solve it or benefit out of it is a debatable topic.
The usage patterns of how people pay and use AI is basically the same model the web should be using: you pay a small bit of money to access monetized pages, just how you pay a small bit of money to get AI responses.
It just needs people and browsers to get onboard with protocols. Crawlers will have no choice but to pay for content behind these 402 gateways.
An honor system is a system without any (explicit) external enforcement of rules. For example an unnattended fruit stand, where people are trusted to be honorable and leave payment for the fruit they take.
this is annoying, it makes a big deal about "Back when we announced pay-per-crawl"...
I want pay-per-crawl. I clicked the link for it a year ago, got presented with a "request access" button, I "requested access" and obviously since I'm nobody I heard absolutely nothing. Now they're touting the link again, I checked, still that same "request access" button. I have no idea if anyone even has access to this feature.
I don't care about all this other stuff, I want the AI crawlers to pay me cash. Because boy do those fuckers want to crawl me. I'll gladly double the size of my gerrit/jenkins servers to keep up with the load if these stupid bots want to pay to crawl every jenkins build artifact and every changeset source file on the server, as they really seem to want to do.
> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).
That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this
It's kind of exhausting seeing Cloudflare playing both sides of the arms race.
I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this.
> This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.
And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.
This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.
???
Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge.
> If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
Cloudflare blocks legitimate users itself sometimes [1].
[0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward.
[1]: https://news.ycombinator.com/item?id=43329320
That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.
>Cloudflare blocks legitimate users itself sometimes [1].
I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges.
> I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
Yeah, I don't like the Cloudflare challenges, but in the past 5 years I've only had it outright block me once, and that fixed itself after 15 minutes. And I use Firefox on Linux with various privacy extensions, so my browser probably appears at least moderately suspicious.
Whereas I've been trapped in impossible ReCaptcha loops quite a few times, which is still better than vague error messages that magically go away when I switch to something not running Linux. So I'll begrudgingly accept that Turnstile is the least user-hostile product on the market right now.
Why not go all the way and mine monero instead of just completely wasting the work?
https://github.com/techaroHQ/anubis
What would force their hand?
It’s more likely they’ll strike undisclosed agreements with major sources of discussion like reddit etc.
That’s not to say getting new information as a way of context-providing is not going to happen but that’s not scraping.
Something has to happen or Google will end up starved and locked out of everything, by means both technical and legal. Then nobody gets anything.
I don't have the answer as to what happens next, and I doubt anyone else who proclaims one super confidently. But we can do some constraints analysis. There is no world where everyone works for free so Google and other AI engines can get all the value from the content, so we can eliminate those possibilities. I think we can safely discard the world(s) in which all content production just stops. However, off the top of my head, it's hard to get much tighter than that, and that definitely leaves a world where effectively everything everywhere ends up going pay-to-access.
Microtransactions have, to date, failed comprehensively, though, so the constraints on what "everything is pay-to-access" gets weird without them.
And there is never guarantee that there is any solution to any set of constraints. Things can end up overconstrained in reality as easily as a math problem. I don't actually think it'll go that way, but when analyzing this question I think it's important to not let "but $SOMETHING just has to have some way to work, because... uh... it has to!" Let the constraints do the talking. You could end up with a scenario where all content of any value is locked down, and it's fundamentally difficult and expensive to ever access or discover it, and consequently the entire content production industry radically contracts compared to its current size, if there is no pragmatic solution to microtransactions that is low-enough friction to get over the psychological and economic hurdles that have killed it to date. If everything is locked behind "macrotransactions" that's a much smaller commercial web. Probably a much higher quality one, too, but at a pretty stiff cost.
The usage patterns of how people pay and use AI is basically the same model the web should be using: you pay a small bit of money to access monetized pages, just how you pay a small bit of money to get AI responses.
It just needs people and browsers to get onboard with protocols. Crawlers will have no choice but to pay for content behind these 402 gateways.
I want pay-per-crawl. I clicked the link for it a year ago, got presented with a "request access" button, I "requested access" and obviously since I'm nobody I heard absolutely nothing. Now they're touting the link again, I checked, still that same "request access" button. I have no idea if anyone even has access to this feature.
I don't care about all this other stuff, I want the AI crawlers to pay me cash. Because boy do those fuckers want to crawl me. I'll gladly double the size of my gerrit/jenkins servers to keep up with the load if these stupid bots want to pay to crawl every jenkins build artifact and every changeset source file on the server, as they really seem to want to do.