I have no idea why the makers of LLM crawlers think it’s a good idea to ignore bot rules. The rules are there for a reason and the reasons are often more complex than “well, we just don’t want you to do that”. They’re usually more like “why would you even do that?”
Ultimately you have to trust what the site owners say. The reason why, say, your favourite search engine returns the relevant Wikipedia pages and not bazillion random old page revisions from ages ago is that Wikipedia said “please crawl the most recent versions using canonical page names, and do not follow the links to the technical pages (including history)”. Again: Why would anyone index those?
Because you are coming from the perspective of a reasonable person
These people are billionaires who expect to get everything for free. Rules are for the plebs, just take it already
I’m imagining a sci-fi spin on this where AI generators are used to keep AI crawlers in a loop, and they accidentally end up creating some unique AI culture or relationship in the process.
I guess this is what the first iteration of the Blackwall looks like.
Gotta say “AI Labyrinth” sounds almost as cool.
“I used the AI to destroy the AI”
And consumed the power output of a medium country to do it.
Yeah, great job! 👍
We truly are getting dumber as a species. We’re facing climate change but running some of the most power hungry processers in the world to spit out cooking recipes and homework answers for millions of people. All to better collect their data to sell products to them that will distract them from the climate disaster our corporations have caused. It’s really fun to watch if it wasn’t so sad.
We had to kill the internet, to save the internet.
We have to kill the Internet, to save humanity.
Surprised at the level of negativity here. Having had my sites repeatedly DDOSed offline by Claudebot and others scraping the same damned thing over and over again, thousands of times a second, I welcome any measures to help.
I think the negativity is around the unfortunate fact that solutions like this shouldn’t be necessary.
Burning 29 acres of rainforest a day to do nothing
Bitcoin?
Considering how many false positives Cloudflare serves I see nothing but misery coming from this.
Lol I work in healthcare and Cloudflare regularly blocks incoming electronic orders because the clinical notes “resemble” SQL injection. Nurses type all sorts of random stuff in their notes so there’s no managing that. Drives me insane!
In terms of Lemmy instances, if your instance is behind cloudflare and you turn on AI protection, federation breaks. So their tools are not very helpful for fighting the AI scraping.
Can’t you configure exceptions for behaviours?
I’m not sure what can be done at the free tier. There is a switch to turn on AI not blocking, and it breaks federation.
You can’t whitelist domains because federation could come from and domain. Maybe you could somehow whitelist /inbox for the ActivityPub communication, but I’m not sure how to do that in Cloudflare.
Imagine how much power is wasted on this unfortunate necessity.
Now imagine how much power will be wasted circumventing it.
Fucking clown world we live in
On on hand, yes. On the other…imagine frustration of management of companies making and selling AI services. This is such a sweet thing to imagine.
My dude, they’ll literally sell services to both sides of the market.
I…uh…frick.
I just want to keep using uncensored AI that answers my questions. Why is this a good thing?
Because it only harms bots that ignore the “no crawl” directive, so your AI remains uncensored.
Good I ignore that too. I want to towards a world where information is shared. I can get behind the
Get behind the what?
Perhaps an AI crawler crashed Melvin’s machine halfway through the reply, denying that information to everyone else!
That’s not what the
no followcommand meansdon’t worry, information is still shared. but with people. not with capitalist pigs
Because it’s not AI, it’s LLMs, and all LLMs do is guess what word most likely comes next in a sentence. That’s why they are terrible at answering questions and do things like suggest adding glue to the cheese on your pizza because somewhere in the training data some idiot said that.
The training data for LLMs come from the internet, and the internet is full of idiots.
That’s what I do too with less accuracy and knowledge. I don’t get why I have to hate this. Feels like a bunch of cavemen telling me to hate fire because it might burn the food
LLM is a subset of AI
And soon, the already AI-flooded net will be filled with so much nonsense that it becomes impossible for anyone to get some real work done. Sigh.
Some of us are only here to crank hog.
AROOO!
You have Thirteen hours in which to solve the labyrinth before your baby AI becomes one of us, forever.
While AI David Bowie sings you rock lullabies.
So the web is a corporate war zone now and you can choose feudal protection or being attacked from all sides. What a time to be alive.
There is also the corpo verified id route. In order to avoid the onslaught of AI bots and all that comes with them you’ll need to sacrifice freedom, anonymity, and privacy like a good little peasant to prove you aren’t a bot… and so will everyone else. You’ll likely be forced to deal with whatever AI bots are forced upon you while within the walls but better an enemy you know I guess?
Not exactly how I expected the AI wars to go, but I guess since we’re in a cyberpunk world, we take what we get
Next step is an AI that detects AI labyrinth.
It gets trained on labyrinths generated by another AI.
So you have an AI generating labyrinths to train an AI to detect labyrinths which are generated by another AI so that your original AI crawler doesn’t get lost.
It’s gonna be AI all the way down.
All the while each AI costs more power than a million human beings to run, and the world burns down around us.
This is the great filter.
Why isn’t there detectable life out there? They all do the same thing we’re doing. Undone by greed.
The same way they justify cutting benefits for the disabled to balance budgets instead of putting taxes on the rich or just not giving them bailouts, they will justify cutting power to you before a data centre that’s 10 corporate AIs all fighting each other, unless we as a people stand up and actually demand change.
Vote Blue No Matter Who
Any Democrat is Better than Any Republican
Plenty of Democrats are voting to put trump nominees in office, plenty are voting on partisan spending bills. The CR vote should tip you off that any democrat is not better than any republican… half of them are complicit too. 10 Senate Dems just financed this authoritarian takeover.
Not a single Democrat voted to confirm Hegseth and 3 Republicans also didnt but he still got confirmed.
Every single Democrat was present and voted no for the Budget which passed the House and it still passed.
Even if 10 dems voted not to shutdown government and enter congressional recess, the CR only exists because Republicans wrote it and won’t compromise.
Any Democrat is Better than Any Republican.
Scheumer rubberstamped autocracy by not filibustering the CR. I think anyone who protects the constitution and their constituents is better than someone who doesn’t. Not that any repuclicans fit the bill, but its not like we can just trust any old democrat. Look at Gavin Newsome sliding to the right to maintain power. That the kinda dems we want?
No
Then you will get Republicans
Blue no matter who is precisely how we got Trump.
In my country blue is conservatives… But I agree with the sentiment! It worked for California, it can work for your whole country, let the Dems stop fearing they’ll lose elections, give them comfortable margins and then massively support progressives who can bring in the good stuff, they won’t have a chance if the party core thinks the very future of elections is on the line, but if they think they’ll likely win anyway, you might just be able to push through a progressive candidate and end the Neoliberal decay.
To be fair, California is kind of dysfunctional and constantly trips over its own regulations when trying to get anything built. For instance, needing excessive environmental impact review for things like trains that will obviously help the environment, or limiting ferry boats crossing the bay to protect the environment even though it likely results in more people driving instead.
It’s so rare to get positive reinforcement like this, these days. Thank you.
LLMs tend to be really bad at detecting AI generated content. I can’t imagine specialized models are much better. For the crawler, it’s also exponentially more expensive and more human work, and must be replicated for every crawler since they’re so freaking secretive.
I think the hosts win here.
Yeah that’s how we achieve singularity.
That’s just BattleBots with a different name.
You’re not wrong.
Ok, I now need a screensaver that I can tie to a cloudflare instance that visualizes the generated “maze” and a bot’s attempts to get out.
You probably just should let an AI generate that.
They should program the actions and reactions of each system to actual battle bots and then televise the event for our entertainment.
Then get bored when it devolves into a wedge meta.
Somehow one of them still invents Tombstone.
Putting a chopped down lawnmower blade in front of a thing, and having it spin at harddrive speeds is honestly kinda terrifying…
Generating content with AI to throw off crawlers. I dread to think of the resources we’re wasting on this utter insanity now, but hey who the fuck cares as long as the line keeps going up for these leeches.
this is some fucking stupid situation, we somewhat got a faster internet and these bots messing each other are hogging the bandwidth.
Lol website traffic accounts for like 1% of bandwidth budget. 1 netflix movie is like 20k web pages.
nothing can be improved while capitalism or authority exist; all improvement will be seized and used to oppress.
The problem you aren’t recognizing is that, until humans are no longer driven by self preservation, there will always be oppression in any system. They all have and will continue to breakdown. It’s easy to blame capitalism but even socialist systems eventually cave under the weight of greed and power. We are the problem mon frère.
there will always be complications, sure. PERFECT may not be achievable.
but apparently we can’t have GOOD, we can’t have DECENT, we can’t have OKAY until the last owner is strangled with the entrails of the last enforcer. not for more than the about ten seconds it takes them to notice.
How can authority not exist? That’s staggeringly broad
given what domains we’re hosted on; i think we’ve both had a version of this conversation about a thousand times, and both ended up where we ended up. do you want us to explain hypothetically-at-but-mostly-past each other again? I can do it while un-sober, if you like.
Not who you responded to but yeah I want to hear a drug fuelled rant I don’t even care what topic
Not the one you responded to but how about the tiredness-fuelled rant that I replied to the other person with.
then don’t ask for that version, dear. I did say ‘by request’, as a way to mix up a conversation I assumed both the person I was speaking to and myself have already had multiple times.
This is the most boring drug fuelled rant I’ve heard in my life
you haven’t, but it would be. that’s why I offered to add the drugs.
I… Don’t think I’ve heard anyone claim authority shouldn’t exist.
So here’s a little bit of lemmy lore for you. You’re instance lemmy.ml is considered to be a tankie instance by some users. lemmy.dbzer0.com is an anarchist instance. The user you responded to probably made a generalisation based on this and assumed you were familiar with anarchist/communist/socialist/leftist discourse. From this comment I assume they were wrong.
So on behalf of no-one but myself: Hello! Welcome to Anarchism! The belief that authority should not exist. This belief comes from a lot of different places and wears a lot of different faces. Most short explanations aren’t sufficient and long explanation bore most. If you don’t mind a little learning here is a link: https://anarchistfaq.org/afaq/sectionA.html#seca1 and another one https://crimethinc.com/2016/09/28/feature-the-secret-is-to-begin-getting-started-further-resources-frequently-asked-questions#faq or if you like videos: https://youtu.be/lrTzjaXskUU.
Also a little bit about authority: people use authority to mean many things (this is even bought up in the video I linked above). But as far as anarchists are concerned (in general (no specific statement can be made about a group so vast)) authority is the act of coercing people to follow orders or commit involuntary acts. You’re boss can coerce you to neglect your health by threatening to fire you. Your government can force you to obey gender roles by threatening to jail you. A rich person can make you do whatever demeaning thing they want by dangling money in front of you (for reference see mrbeast) because otherwise your landlord will kick you out. This is authority and it is wrong. Those in authority can make mistakes, become greedy and start to think they have the power to do whatever they want (mostly because they can). This leads to suffering. My meaning of life is to minimise suffering. Anarchy is the belief that no-one should hold power over others. That all leadership should be scrutinised. It rejects blind faith in single people and encourages to think for yourself so no-one can do you wrong. And if you can’t be bothered, it encourages you to find people who genuinely care about you and let them stand up for you.
Thanks, I don’t really care for generalizing instances and didn’t really have a choice when I made my account.
But also your definition is impossibly broad as you well and I’m pretty sure not the general consensus. The video doesn’t define it as such either.
For one thing, by your definition we can have absolutely no meaningful human relationships. I can explain this more later when I have time if you don’t see what I mean
My explanation of authority wasn’t meant as a definition but rather a brief summarisation of a complex concept, Andrew does a better job actually explaining it. Like pointing out that Authority is confused with a lot of different concepts like respect or influence. Which I’m starting to suspect is happening here. meaningful human relationships are based on mutual respect. This is not authority as it is voluntary, reciprocated and revoked as soon as the other party steps over the line. This is what I believe is the basis of society and what we need to return to in order to live a truly free life. In modern society in most interactions respect has been replaced with authority. People in positions of power even use them synonymously.
sorry, assumed from combination of comment+instance.
you can totally have human relationships! you just try to base things on consent. it’s hard to see from here, because everything is so steeped in fuckery, and it’s like trying to imagine slack in a system where a thousand things are keeping the same thing under high tension, but it does tend to work when the forces of oppression briefly collapse. it’s also not a binary. you can look at the way systems function better with more autonomy, and why. this has been studied. I can talk on that at length, but that version of the topic is full of serious doorstoppers and even harder to understand in detail.
‘a paradise born in hell’ by solnit goes into how it tends to emerge during disaster response/recovery, and how centralized authoritarian responses tend to… not help so much, if at all. it’s pretty well written, highly recommend.
glad to offer you a new experience.
That’s not really relevant here. This is more of a “genie is out of the bottle and now we have to learn how to deal with it situation”. The idea and technology of bots and AI training already exists. There’s no socioeconomic system that is going to magically make that go away.
I think the point you’re missing is that without the monetary incentive that arises under capitalism, there would be very little drive for anyone to build these wasteful AI systems. It’s difficult to imagine a group of people voluntarily amassing and then using the resources necessary for “AI” absent the desire to cash in on their investment. So you’re correct that an alternative economic system won’t “magically” make LLMs go away. I think it unlikely, however, that such wasteful nonsense would be used on any meaningful scale absent the perverse incentives of capitalism.
It’s difficult to imagine a group of people voluntarily amassing and then using the resources necessary for “AI” absent the desire to cash in on their investment.
I mean Dmitry Pospelov was arguing for AI control in the Soviet Union clear back in the 70s.
Just another way the state capitalist soviet union was closer to capitalism than socialism.
It is called regulation in sane parts of the world.
Sadly, those areas seem to be diminishing rapidly until more people enter the Find Out phase.
I don’t need it to not exist. I need it to stay the fuck out of everyone’s lives unless they work in a lab of some kind.
see, it’s not actually useful. it’s a tomogatchi. do you remember those? no, you fucking don’t.
everyone remembers tomogatchi, they were like a digital houseplant.
cool, but where do you get them? you can’t, right? because they were stupid?
You can still buy new ones. Take a nap.
somehow I get the impression all this defense of tomogatchis is not about tomogatchis.
the used market
so you’re saying it’s a niche toy you can get if you really want one, but nobody’s pushing that shit on you, and if you never want to talk about one again in your life, you can probably do that?
I would like if large language models were in this position. I don’t think you understand the degree to which our productive capacity and infrastructure are committed to this technology. it’s a lot. basically all the cutting edge computer chips being made are specialist chips for processing large (whatever) models, including the engineering to back it. that means everything else is bumped down a generation or five.
then there’s the amount of electricity being put towards these things-we are in a climate disaster, we do not have green energy, and these things are drawing in the high GW/low TW of energy.
water is also a lot more precarious than a lot of people want to think about. these things are using lots and lots of good drinkable water to cool those specialist chips, then just being throws out, because I assume it’s cheaper than cooling the hot water back down.
if these things didn’t ruin the internet, and were just something you ran on the beefy desktop you otherwise use for games, I wouldn’t mind, but that’s not where we are.
Lost on Lemmy?
He is not wrong. Unless people start to take steps , the dependency of tech will be used to chain most of us. Granted, these chains will be the kindest and gentlest chains seen in a long time.
Social revolution lives on in decentralized services, like this; the true battles will be later though. This year is a mild warm up. I can’t imagine the challenges that await many
no, responding to a comment about exactly that thing.
Especially since the solution I cooked up for my site works just fine and took a lot less work. This is simply to identify the incoming requests from these damn bots – which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot – and simply IP ban them. This is considerably simpler, and doesn’t require an entire nuclear plant powered AI to combat the opposition’s nuclear plant powered AI.
In fact, anybody who doesn’t exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to “learn” about products rather than sticking to its own domain.
Fuck 'em. Route those motherfuckers right to /dev/null.
and try to slam your site with like 200+ requests per second
Your solution would do nothing to stop the crawlers that are operating 10ish rps. There’s ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It’s not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
Yep. After you ban all the easy to spot ones you’re still left with far too many hard to ID bots. At least if your site is popular and large.
It’s worked alright for me. Your mileage may vary.
If someone is scraping my site at a low crawl rate I honestly don’t care so long as it doesn’t impact my performance for everyone else. If I hosted anything that was not just public knowledge or copy regurgitated verbatim from the bumf provided by the vendors of the brands I sell, I might oppose to it ideologically. But I don’t. So I don’t.
If parallel crawling from multiple organizations legitimately becomes a concern for us I will have to get more creative. But thus far it hasn’t, and honestly just wholesale blocking Amazon from our shit instantly solved 90% of the problem.
Geez, that’s a lot of requests!
It sure is. Needless to say, I noticed it happening.
the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don’t want a chance to accidentally block one.
This is fair in those applications. I only run an ecommerce web site, though, so that doesn’t come into play.
Cloudflare offers that too, but you can’t always tell
It’s what I’ve been saying about technology for the past decade or two … we’ve hit an upper limit to our technological development … that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they’re always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society … capitalism is literally our world’s bottleneck and it’s being choked off by an oddly shaped gold bar at this point.
