The open source software movement was very much founded on the idea of being concerned about what kinds of tools we use as a society. Software that’s “just a tool” can lock you down and out of what you can do with it. Same argument can just as well be made that people were “shying away from using new and interesting tools” by running free software instead of Microsoft’s new priority stuff that locks you in
AI similarly gives massive control to a handful of companies. They have already used that leverage to intentionally sandbag tasks without telling you, when they think it’s against their economic interests (model distillation and AI research). Is it not possible to imagine that goes further, they would use that leverage to slip in backdoor, not report classes of bugs, etc.?
Torvalds has never been a Free Software guy. Open source is a convenience to him. Why should he allow blobs in the kernel otherwise? They do the job and arguably don’t violate the GPL. That’s all he’s ever been concerned with.
These risks are not different from before and still open source has the advantage of being open to anyone and any agent.
Kernel modules written by US employees with unknown ties to US intelligence was just as much at risk for containing a backdoor as if written by an agent using an LLM with US intelligence ties.
And in both cases intelligence actors or security researchers from around the world can review it all and try to spot backdoors. So can agents with different LLMs behind them.
It would be an issue if there was only one state sanctioned LLM available, but as long as there is world wide competition in training LLMs, opensource will be “reviewed” with competing interests in “mind”.
Do explain how open models like Qwen that you can run locally give massive control to handful of companies. In fact, what seems obvious is that using and improving these models is precisely what takes control away from these companies. People who started the free software movement didn’t cry like a bunch children about how much they hated software. They had the brains to understand the problem wasn’t the software itself, but rather who controls it. So, they went out and built open versions of the software that weren’t corporate owned. And try guess what tools they used initially to do that since open software didn’t exist yet. Your mind is about to be blown.
Only a handful of companies make open weight models, and those models are not made for charity. They want either market share and/or control. Making your own model is not like making your own software because the capital cost are extraordinarily high. The masses will not have the billions in funds they would need need to train these models
Unlike open source code, you cannot check what’s actually in the open weight models. Worse still, you can hide things that trigger on very specific words (poison attacks can be intentional by the model maker or put by other actors in the public data). Fine tuning attempts are also no escape from hidden features you don’t even know to look for
The reason Chinese companies are making open models is very simple actually, and has fuck all to do with charity. LLMs are a form of a general commodity, and it’s not a viable business model to monetize them directly. Trying to compete on model quality is a race to the bottom. Anything one lab can do, others can figure out a few months later. So, you have to keep investing more and more money just to keep ahead. Chinese companies realized this early, and that’s why they release the model in the open, aiming for the high end of the value chain building customized services and integrations tha capture the market. My expectation is that LLMs will become like Linux where nobody really makes any money off the tech directly. But you’ll see platforms like AWS built on top of it.
Also, not sure what the rationale for companies releasing open models would be to trigger some unspecified poisoning in a coding model. Like what exactly is it gonna do when running locally?
Finally, there are completely open models as well with training data, and source released in the open. None of them are frontier quality, but they are certainly a starting point.
There also projects like petals for doing distributed training. There’s absolutely no reason why you can’t have something like SETI@home for training models on a network. This is what people should be putting their energy into instead of whinging endlessly, as if OpenAI or Anthropic give a flying fuck that some brave lib made yet another ‘AI slop’ comment on a niche social media platform.
Current techniques cannot give you anything close to the level of inspection you can with software. Worse still, techniques of frontier AI labs are moving in the opposite direction with it getting harder to understand over time and less going into things like chain of thought
There also projects like petals for doing distributed training
You always be at a massive disadvantage because of the severally more limited compute available. Additionally, distributed computing not inside the same data center adds massively increased latency
Training a model also does nothing for the environmental side of things and just makes things worse
instead of whinging endlessly as if OpenAI or Anthropic give a flying fuck that some brave lib made yet another ‘AI slop’ comment on a niche social media platform.
That’s 100% missing the point. The point is to not encourage more people to use the tool that gives control to Anthropic, OpenAI, etc. Social stigma and pushback is the point. People use things less when it’s seen as uncool. Normalizing AI use just means it’s going to be used more against your interests
No, I’m countering what you said by pointing out that there are rational reasons for companies to put models out in the open.
Current techniques cannot give you anything close to the level of inspection you can with software.
So, that just means people need to put the work into improving the techniques. That’s how problems are solved. Last I looked, models like Qwen are actually becoming easier to understand as the article I linked explains.
You always be at a massive disadvantage because of the severally more limited compute available.
Not really because you’d be leveraging a global compute cluster and amortizing the work. But the thing is that it’s not even a competition. Local models just need to be good enough to use. The same way my I don’t need to have a supercomputer at home for my needs, I don’t need a frontier model either. A model like Qwen is already plenty useful, and they will only get better.
raining a model also does nothing for the environmental side of things and just makes things worse
That’s going to continue to happen regardless of how much whinging you do. The question is whether models will be owned by corps or by the public.
The point is to not encourage more people to use the tool that gives control to Anthropic, OpenAI, etc.
You already lost this battle. And on top of that, as you so glibly pointed out in your other reply, an insane amount of money is going into these companies. So these models will obviously continue to be trained and operated. This is the reality:
I don’t think this is a good faith conversation if we are going to continue to call it being massively used against workers and our overall society just “whining”. I was going to respond with all the other negative effects on human cognition from AI which do not go away no matter who makes it, but I see little point to continuing this
The open source software movement was very much founded on the idea of being concerned about what kinds of tools we use as a society. Software that’s “just a tool” can lock you down and out of what you can do with it. Same argument can just as well be made that people were “shying away from using new and interesting tools” by running free software instead of Microsoft’s new priority stuff that locks you in
AI similarly gives massive control to a handful of companies. They have already used that leverage to intentionally sandbag tasks without telling you, when they think it’s against their economic interests (model distillation and AI research). Is it not possible to imagine that goes further, they would use that leverage to slip in backdoor, not report classes of bugs, etc.?
Torvalds has never been a Free Software guy. Open source is a convenience to him. Why should he allow blobs in the kernel otherwise? They do the job and arguably don’t violate the GPL. That’s all he’s ever been concerned with.
That’s all I need to know.
These risks are not different from before and still open source has the advantage of being open to anyone and any agent.
Kernel modules written by US employees with unknown ties to US intelligence was just as much at risk for containing a backdoor as if written by an agent using an LLM with US intelligence ties.
And in both cases intelligence actors or security researchers from around the world can review it all and try to spot backdoors. So can agents with different LLMs behind them.
It would be an issue if there was only one state sanctioned LLM available, but as long as there is world wide competition in training LLMs, opensource will be “reviewed” with competing interests in “mind”.
Do explain how open models like Qwen that you can run locally give massive control to handful of companies. In fact, what seems obvious is that using and improving these models is precisely what takes control away from these companies. People who started the free software movement didn’t cry like a bunch children about how much they hated software. They had the brains to understand the problem wasn’t the software itself, but rather who controls it. So, they went out and built open versions of the software that weren’t corporate owned. And try guess what tools they used initially to do that since open software didn’t exist yet. Your mind is about to be blown.
Only a handful of companies make open weight models, and those models are not made for charity. They want either market share and/or control. Making your own model is not like making your own software because the capital cost are extraordinarily high. The masses will not have the billions in funds they would need need to train these models
Unlike open source code, you cannot check what’s actually in the open weight models. Worse still, you can hide things that trigger on very specific words (poison attacks can be intentional by the model maker or put by other actors in the public data). Fine tuning attempts are also no escape from hidden features you don’t even know to look for
The reason Chinese companies are making open models is very simple actually, and has fuck all to do with charity. LLMs are a form of a general commodity, and it’s not a viable business model to monetize them directly. Trying to compete on model quality is a race to the bottom. Anything one lab can do, others can figure out a few months later. So, you have to keep investing more and more money just to keep ahead. Chinese companies realized this early, and that’s why they release the model in the open, aiming for the high end of the value chain building customized services and integrations tha capture the market. My expectation is that LLMs will become like Linux where nobody really makes any money off the tech directly. But you’ll see platforms like AWS built on top of it.
Also, not sure what the rationale for companies releasing open models would be to trigger some unspecified poisoning in a coding model. Like what exactly is it gonna do when running locally?
Meanwhile, you absolutely can inspect the weights and there are now plenty of tools for doing just that. It’s an active area of research, but these things aren’t black boxes people make them out to be.
Finally, there are completely open models as well with training data, and source released in the open. None of them are frontier quality, but they are certainly a starting point.
There also projects like petals for doing distributed training. There’s absolutely no reason why you can’t have something like SETI@home for training models on a network. This is what people should be putting their energy into instead of whinging endlessly, as if OpenAI or Anthropic give a flying fuck that some brave lib made yet another ‘AI slop’ comment on a niche social media platform.
That’s what I said?
Current techniques cannot give you anything close to the level of inspection you can with software. Worse still, techniques of frontier AI labs are moving in the opposite direction with it getting harder to understand over time and less going into things like chain of thought
You always be at a massive disadvantage because of the severally more limited compute available. Additionally, distributed computing not inside the same data center adds massively increased latency
Training a model also does nothing for the environmental side of things and just makes things worse
That’s 100% missing the point. The point is to not encourage more people to use the tool that gives control to Anthropic, OpenAI, etc. Social stigma and pushback is the point. People use things less when it’s seen as uncool. Normalizing AI use just means it’s going to be used more against your interests
No, I’m countering what you said by pointing out that there are rational reasons for companies to put models out in the open.
So, that just means people need to put the work into improving the techniques. That’s how problems are solved. Last I looked, models like Qwen are actually becoming easier to understand as the article I linked explains.
Not really because you’d be leveraging a global compute cluster and amortizing the work. But the thing is that it’s not even a competition. Local models just need to be good enough to use. The same way my I don’t need to have a supercomputer at home for my needs, I don’t need a frontier model either. A model like Qwen is already plenty useful, and they will only get better.
That’s going to continue to happen regardless of how much whinging you do. The question is whether models will be owned by corps or by the public.
You already lost this battle. And on top of that, as you so glibly pointed out in your other reply, an insane amount of money is going into these companies. So these models will obviously continue to be trained and operated. This is the reality:
Normalizing whinging about AI without doing anything productive just means that people who share none of your concerns will be the ones developing it.
I don’t think this is a good faith conversation if we are going to continue to call it being massively used against workers and our overall society just “whining”. I was going to respond with all the other negative effects on human cognition from AI which do not go away no matter who makes it, but I see little point to continuing this
amazing counterpoint as always