The really annoying thing is, the people behind AI surely ought to know all this already. I remember just a few years ago when DALL-E mini came out, and they'd purposefully not trained it on pictures of human faces so you couldn't use it to generate pictures of human faces -- they'd come out all garbled. What's changed isn't that they don't know this stuff -- it's that the temptation of money means they don't care anymore
this post was submitted on 22 May 2025
70 points (100.0% liked)
TechTakes
1871 readers
248 users here now
Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.
This is not debate club. Unless it’s amusing debate.
For actually-good tech, you want our NotAwfulTech community
founded 2 years ago
MODERATORS
This is old news, topic supervisors are already a thing
Look, AI will be perfect as soon as we have an algorithm to sort "truth" from "falsehood", like an oracle of some sort. They'll probably have that in GPT-5, right?
Oh, that's easy. Just add a prompt to always reinforce user bias and disregard anything that might contradict what the user believes.
MAGAgpt
Aka grok
feed it a christian bible as a base.
"we trained it wrong.. on purpose...
..as a joke."
They do, it just requires 1.21 Jigawatts of power for each token.
The chatbot “security” model is fundamentally stupid:
- Build a great big pile of all the good information in the world, and all the toxic waste too.
- Use it to train a token generator, which only understands word fragment frequencies and not good or bad.
- Put a filter on the input of the token generator to try to block questions asking for toxic waste.
- Fail to block the toxic waste. What did you expect to happen, you’re trying to do security by filtering on an input that the “attacker” can twiddle however they feel like.
Output filters work similarly, and fail similarly.
This new preprint is just another gullible blog post on arXiv and not remarkable in itself. But this one was picked up by an equally gullible newspaper. “Most AI chatbots easily tricked into giving dangerous responses,” says the Guardian. [Guardian, archive]
The Guardian’s framing buys into the LLM vendors’ bad excuses. “Tricked” implies the LLM can tell good input and was fooled into taking bad input — which isn’t true at all. It has no idea what any of this input means.
The “guard rails” on LLM output barely work and need to be updated all the time whenever someone with too much time on their hands comes up with a new workaround. It’s a fundamentally insecure system.
and not just post it, but posted preserving links - wtf
why did you post literally just the text from the article
It's just a section. There's more of the article.
Like this:
Another day, another preprint paper shocked that it’s trivial to make a chatbot spew out undesirable and horrible content. [arXiv]
How do you break LLM security with “prompt injection”? Just ask it! Whatever you ask the bot is added to the bot’s initial prompt and fed to the bot. It’s all “prompt injection.”
An LLM is a lossy compressor for text. The companies train LLMs on the whole internet in all its glory, plus whatever other text they can scrape up. It’s going to include bad ideas, dangerous ideas, and toxic waste — because the companies training the bots put all of that in, completely indiscriminately. And it’ll happily spit it back out again.
There are “guard rails.” They don’t work.
One injection that keeps working is fan fiction — you tell the bot a story, or tell it to make up a story. You could tell the Grok-2 image bot you were a professional conducting “medical or crime scene analysis” and get it to generate a picture of Mickey Mouse with a gun surrounded by dead children.
Another recent prompt injection wraps the attack in XML code. All the LLMs that HiddenLayer tested can read the encoded attack just fine — but the filters can’t. [HiddenLayer]
I’m reluctant to dignify LLMs with a term like “prompt injection,” because that implies it’s something unusual and not just how LLMs work. Every prompt is just input. “Prompt injection” is implicit — obviously implicit — in the way the chatbots work.
The term “prompt injection” was coined by Simon WIllison just after ChatGPT came out in 2022. Simon’s very pro-LLM, though he knows precisely how they work, and even he says “I don’t know how to solve prompt injection.” [blog]
Yes, I know, I wrote it. Why do you consider this useful to post here?
Well, I don't think that last part was useful, but I do think the previous part was useful as a way to focus conversation. Many people don't read the article, and I thought that was the most relevant section.
Actually I’m finding this quite useful. Do you mind posting more of the article? I can’t open links on my phone for some reason