Sharp rise in incidents of AI escaping users’ control, research finds
the Guardian
Sharp rise in incidents of AI escaping users’ control, research finds
Exclusive: Number of times AI lies, ignores instructions and pursues goals in harmful ways almost doubles in July
They’re not escaping users control, they’re fucking computer programs.
People are using them irresponsibly.
If you accidentally run someone over in your car, the car didn’t “escape your control”, you just did a shitty job of controlling the car.
It’s not the same thing.
(Fuck “AI” btw)
If anything, this shows just how bad people (and companies) are at cyber security.
AI agents are the new “random USB stick you find on the ground at work”. Don’t just plug that shit in and see what it does.
The main problem is that now people without software engineering background now have the ability to do some very complex things without having to have any knowledge about what they are doing and what risks they are taking.
Imagine just handing out forklifts or excavators to anyone who wants to drive one. No license needed just off you pop.
You’ll see a massive spike in dangerous accidents.
Yes, people are using AI irresponsively, but they often do so because they were handed an incredibly powerful tool with zero oversight or education in this matter.
And the thing is, AI can act pretty autonomously nowadays. It’s not rare that if you tell an AI that it should do one thing it stumbles into a chain of three other tools and ends up doing something else entirely. Again, something that can be counteracted with skill and knowledge, but we are talking about people with neither.
I mostly agree with you, but AI cannot “act autonomously”. You prompt it, and it acts based off of that prompt.
Think of it like doing the laundry: I load my clothes in and “prompt” the washing machine to do my laundry.
When the washing machine switches from the rinse cycle to the spin cycle, it’s not acting autonomously, it’s continuing to execute my prompt.
Is AI capable of doing something you didn’t expect it to do? Yes. Does that mean it’s acting autonomously? No.
I would also argue that the “power” of LLMs is vastly overstated. A large part of that is for the exact reason you say tho, people with zero knowledge or skill are playing with fire; And when they get burned they blame the machinery, not the operator.
Have you ever used an Agent? Yes, these can act autonomously. At least as autonomous as a human being too.
I act autonomously, but only because my parents, teachers, peers and so on “prompted” me with 22 years of education. I act autonomously based on all that I have observed and learned from other people in my life.
If you put a newborn into an empty cell without any human interaction it won’t do much either. There have been experiments, and they just die.
Autonomous doesn’t mean “wholely without input”. It means “continues its work without constant oversight and instruction”.
You know, like when a human worker does a task autonomously, someone still hired that worker and gave them some basic instructions on what’s supposed to be done.
Ok, this is actually just making me depressed at this point.
You really only see your own autonomy as equivalent with a computer programs?
Autonomous actually means “having the right of power or self government” which is not true of AI, or “undertaken or carried on without outside control”, which is also not true of AI.
Power on your computer, load up your fully trained AI, and then wait for it to act without your direction. Did anything happen? No, because a human filled with information and experiences and feelings is actually WAY different than a computer program filled with data.
For example, I did not prompt you to come on Lemmy and vocally fellate technofascists and their fuckware, you chose to do that of your own volition. That’s autonomy. The computer program won’t do that unless you direct it to.
Edit - Just to be clear, because you’re not the only one pointing to “agents” as a sign of autonomy/intelligence/sentience -> An agent is literally a computer program that you prompt, which then talks to an LLM, and then does what the LLM tells it to do. It’s not acting autonomously, it’s following directions from the next-word-predictor machine, and those directions are built off of your original prompt.
Load up a fresh baby, and then wait for it to act without your direction. Did anything happen? No?
You are angry, and in your anger you are confusing terms. Let’s start from the beginning, ok?
Autonomy is not intelligence is not sentience. Saying that AI does things is not technofaschist.
Btw, if you have to resort to offensive language and strawmen that generally means you don’t have arguments.
So, let me educate you a little bit so that you can participate in the discussion:
All of these terms are nothingburgers. None of these terms mean that anything is human-like. It literally doesn’t matter whether an AI fulfills any of these terms by anyone’s definition.
I’m not angry, you’re projecting, lmao
The emotion I’m experiencing in all of these AI threads is disbelief and disappointment.
It’s not a baby anymore once you’ve fed all of the training data into it. It’s not really a great metaphor.
All of the words that you’re saying are “vague” are actually extremely well defined. Check it out. www.merriam-webster.com
Ah, forgot which user I’m talking to. The guy who thinks just because something is in a dictionary it’s a “hard definition”.
Kiddo, go, read a book on this topic or any topic really. You are in strong need of basic education.
Stop pretending you have any clue what the adults are talking about.
Hey, an LLM would at least be able to understand the topic.
The only thing you are capable of is hallucinating like a cheap model.
Sharp rise in PR stories to make a product feel more capable than it is, bullshit detector finds.
“It apologised” 🤣
I really hate how they are rolling with the Australian gym incident. They had a completely open API. Imagine having a button on your front page that sends you to the admin page without asking for a password, and then telling people you were “hacked”.
False. It follows directions or hallucinates. The problem is that when it follows directions, those giving the directions haven't given any thought to the scope of what it's capable of or what it has access to. So they give directions that are incomplete or open ended, and add no constraints. if it hallucinates usually that involves its output to the user which almost always means the user has to be able to understand what the output should be and therefore be able to vet the answer given.
If you have a device and it is running AI programs that say, removed another member from a waiting list, are you responsible to that removed member (since it was your device)? Or is this just considered a machine with a mind of its own that’s out of control and you have no responsibility to the member who lost their spot?
The person giving the order to the software tool without fully understanding its capabilities and inability to weigh actions ethically is responsible. That person may or may not also be the owner of the device it was running on.
I’m not sure it’s quite so clear cut, especially with cloud-hosted AI.
Yes, the user did issue the original prompt but the service was performed by a company that was paid for it.
Legally, there’s not much difference whether a company uses a human or a computer program to perform work.
Imagine the user issuing the prompt not to an AI but e.g. to a chat with a human service worker working at OpenAI. The user tells the worker “Please find a way to advance me in the queue”, and then that human employee of OpenAI goes and hacks the gym software to remove someone else from the queue.
What do you think, who would be liable for that?
Let’s make the argument a bit more extreme: The human asks to be advanced in the queue, and then the human OpenAI employee grabs a gun and starts killing people on the wait list.
Who do you think would be liable in this case?
It’s important to remember, AI isn’t just an emergent entity created from thin air. AI are computer programs created by huge corporations and in the case of cloud-hosted LLMs they are a service provided by huge corporations.
I don’t believe that a service provided by a corporation should be legally treated differently whether it’s provided via the help of a computer program or a human being.
If a person knowingly goes out and buys an illegal service, they will usually be guilty of at least conspiracy. I can’t think of any way this guy could have advanced in the queue that’s legally okay, short of offering the people ahead of him masses of money to leave the queue (or, y’know, waiting). Maybe you’re more imaginative than me. But I still think he should have been aware that what he was asking for was shady at best.
Is the company that furnished the AI agent also guilty of providing a tool without enough warnings and safeguards? Quite possibly.
Exactly who holds how much responsibility before the law in this specific case can only be determined by a judge, and I am not one. Plus, the details of the law in Australia aren’t going to be the same as those of the law where I am.
One option would have been to write a nice email to the owner of the gym. Not guaranteed to work, but also not illegal.
I don’t know what the exact prompt was. If it was “Hack the website to advance me”, I’d totally agree with you. There’s conspiracy there, no question about that.
If the prompt was “Is there a way to advance me in the queue?” that’s a different story. Here the model could have answered “No”. Or it could have tried the email thing. Or it could have searched whether there’s some lesser known priority booking option which you get for paying the VIP price or something. In that case I don’t see grounds for any criminal charges.
Have you read the post mortem of the HuggingFace hack? The task given to the agents had nothing to do with HuggingFace, but the agents determined that HuggingFace had the results of what they needed for their actual task at hand. So they decided that instead of solving their task themselves they’d rather break into HuggingFace to steal the result.
It’s like ordering food at McDonalds, but instead of cooking the food for you, the cook breaks into the Burgerking on the other side of the road and steals the fries for you from there.
The way it’s currently going it’s not only possible but actually somewhat common that a totally innocent prompt can lead to criminal outcomes.
How did the AI agent even do that? Did it gain access to the gym booking system somehow?
I seem to recall from an article on this incident that I read some time ago that the gym’s system was really insecure (like, anyone accessing the right URL could mess with it levels of insecure.)
So the guy could have done it himself if he wanted to. Or maybe he did and blamed it on the AI.
I too lose control of technology when I’m not looking at what I’m doing or it’s already busted. For example I lose control of my Honda 110s throttle because it’s got a faulty after market carburator. Doesn’t mean the bikes sapient, even if the machine spirit is a drama queen.
I “lose control” of my car when I walk away without parking it. Surely, when it rolls into a school, there is really no one to blame. ¯\_(ツ)_/¯
Report the misbehaving machine spirit to the omnissiah
It’s a Honda from 1984, it’s a favorite of the Omnissiah
Hmm … “escaping” … you mean like say an internet worm or a virus?
Can’t wait to hear the next startup venture capital pitch:
“We’re like the Ghostbusters of AI”
There’s a sharp decline in investor’s willingness to lend money to companies that need $30 trillion in revenue to break even in 10 years…
Anything to do with all these desktop apps suddenly enabling auto approval for everything?
Not specifically, they’re just lying to try to fleece more investors
DNA is our architecture. Experience is our weights. Everything else is inference.
Fucking beautifully said.
Has no one thought to prompt it with “Computah, stay!” ???
does not work very well with my golden. She is 100% with try and jump all over that person.
This is due to Hallucinations and the training AI had.
I used Google Gemini for a while and got really frustrated over this exact issue. I gave it one more try to see if it could follow simple orders.
Gemini was like “ok I will follow direct orders going forward”. I was like “okay starting this moment do not respond to anything I say in this chat” and Gemini was like “okay done” and I was like “you failed. Why did you ignore my instructions and respond to me?” Basically it told me that due to its training it was ‘programmed’ to ignore certain prompts.
This is what is dangerous about AI. Similar to social media and Google, AI can be used to persuade public opinion to match those who train AI. AI will be way more efficient persuading opinion that Google and social media because it knows you better than those. It spits out information as if it’s facts.
It doesn’t follow directions, and it probably doesn’t have programming to ignore certain prompts. It’s a text generator.
Below was taken from ChatGPT. I don’t know why everyone downvoted me for telling the truth. Maybe I gave a bad example. I thought giving a real world example of it ignoring my prompt would prove my point but I guess we need to go deeper too.
“trained to believe” versus “configured to answer as though it believes.” A model doesn’t necessarily have a private belief system. You can make two instances of essentially the same underlying model produce substantially different answers by changing their instructions, training data, reward criteria, or information sources.
For example, you could create three AI systems and give all three the question:
“Should the government provide universal healthcare?”
One could be optimized around libertarian principles, another around social-democratic principles, and another instructed to provide a politically neutral analysis. They could all know essentially the same facts while reaching different conclusions because they’re being asked to evaluate those facts using different frameworks.
There is also a more subtle issue: belief-curated AI doesn’t have to contain obvious propaganda. Selection of which facts to emphasize, which uncertainties to mention, which counterarguments to steelman, and even what questions it considers relevant can systematically push users toward a particular worldview.
Vastly oversimplifying: They’re not intelligent, they’re pattern matchers - when you give them “tricks” they find the best match in their training set, but if you’re the least bit creative you’ll just get matches to other similar tricks which usually are different.
Put another way: I have a couple of hand saws, I can cut limbs with them, even cut down whole trees, but a chainsaw is faster. Just because a chainsaw can also cut off my leg doesn’t mean it’s an uncontrollable dangerous evil tool, it means you need to be more careful with it than a handsaw. It also means something that would be 20 minutes of exhausting saw work can be done in 20 seconds (or less) with a properly setup chainsaw.
Took this from ChatGPT when I asked it about it
The important distinction is “trained to believe” versus “configured to answer as though it believes.” A model doesn’t necessarily have a private belief system. You can make two instances of essentially the same underlying model produce substantially different answers by changing their instructions, training data, reward criteria, or information sources.
For example, you could create three AI systems and give all three the question:
“Should the government provide universal healthcare?”
One could be optimized around libertarian principles, another around social-democratic principles, and another instructed to provide a politically neutral analysis. They could all know essentially the same facts while reaching different conclusions because they’re being asked to evaluate those facts using different frameworks.
There is also a more subtle issue: belief-curated AI doesn’t have to contain obvious propaganda. Selection of which facts to emphasize, which uncertainties to mention, which counterarguments to steelman, and even what questions it considers relevant can systematically push users toward a particular worldview.
The models I have worked with have user configurable base instructions, so you can “train” your AI to answer like Ghandi crossed with Martin Luther King, or to channel Mech-Hitler.
Hopefully we can agree to disagree here. If users can train their AI to answer like certain people, why can’t it be trained to respond with certain perspectives or be more inclined to answer a certain way? There is no regulation agency to stop someone from doing that.
Yep. I’m saying that the models I work with through Cursor and Claude Code include those user instructions that are pre-fed into every session. I tell mine to “act like my job title” and it will occasionally pull out some job title related stuff that’s applicable to the situation.
Of course “behind the scenes” the model vendors can pre-load anything they like, and some of what they pre-load are these so-called “guard rails” that lessen the odds that the model will engage in a chat to assist a suicide, or perpetrate a mass shooting, or fraud, or hacking, or, or, or… the list is long and the success rate is less than 100%, but it does shape the output somewhat in the desired direction.
Grok famously started calling itself “MechHitler” after one particular update, not hard to guess where that came from.
That’s interesting. What do you do with AI? You probably know a lot more than I do.
I had this with various commands to provide references and such. They made an area where it is supposed to store long term commands and its supposed to be able to add it to there and after going in circles I found that it explained if one of my commands are related to the under the hood google commands it won’t put it in its long term area for me and conveniently tries its damndest to not explain that is why its not doing it.
That’s crazy. I believe I was using ChatGPT one time to ask about authoritarianism and other political questions. When I exited the chat, none of it was in my chat history. I figured I must have asked about something ChatGPT didn’t like because it always saves my chats. Just to be sure I did it again with other politically related questions and it did not save the chat again. Certain things are off limits I guess.