Plaintiff busted trying to use AI prompt injection to win court case, hides text instruction in filing — demands AI model reviewing the text should side with him, rumbled because of strange white spa…
Full title: Plaintiff busted trying to use AI prompt injection to win court case, hides text instruction in filing — demands AI model reviewing the text should side with him, rumbled because of strange white spaces in text
Love the long but still cryptic headline!
According to the article partial success might have been possible:
Not sure what to think of this.
I mean Fuck AI and all that but it’s kinda clever. But only the first time somebody uses it.
And obviously illegal, I do hope the judge comes to the same conclusion.
Using AI to review legal documents is monumentally stupid as well.
Oh yeah. I work in legal tech, and out firm made our internal training on using these models scary as heck to give the associates and shareholders a proper perspective.
That said, we’re only using models that were trained on legal data, so there’s no shitposts from Reddit or FB or whatever to gum up the works, but we still require human validation and drastic consequences if people fail to do so and it’s found out.
Agreed, but:
“Everybody does it! 🤷”
The whole AI hype is monumentally stupid.
I feel like prompt injection should be a perfectly legal defense but only as long as it‘s phrased reasonably. Or is adding a note that asks for a fair trial unreasonable enough to be dismissed? When you only try to reason with unreasonable word salad how could anyone blame you for it?
Adding to that if one side uses LLMs they should definitely have to at least attach the prompts they used and share what model they used so it can be replicated.
I’m not aware of any public frontier LLM provider that uses a static seed for inference. Meaning, even with an identical prompt and identical model you will not get the same output. Seeds should absolutely come back with the streaming metadata on requests imho, but they don’t in any api/harness I’m aware of.
I know. But it needs to be replicated because an LLM can‘t actually explain itself. Because it doesn‘t understand words. If it can‘t be replicated or another session gives a completely different response that‘s their problem to deal with. Because if they legally try to treat LLMs as a consultant or expert, that expert should at least be consistent. If it can‘t well then you probably shouldn‘t use it blindly.
An LLM’s response to a prompt is not necessarily a repeatable process.
Feeling the need to add AI instructions to make it follow the judicial systems own rules is probably not unreasonable given the goofy things those *AI systems can vomit up.
The threat of retaliation for doing it is even more wild though.
fixed ambiguity
What kind of retaliation can the plaintiff make on the LLM agent?
None of that is what’s going on.
[deleted]
It sounds like the only person we know was using LLMs was the plaintiff. He believes the court was as well, but I’m not aware of any reason to believe his accusation
If nobody at the court was using an LLM how was his hidden text discovered?
I read the article. It was enlightening.
I read another article on it and it was never mentioned.
Yes, I also have a girlfriend, but you don’t know her, she lives in Canada.
This story has been posted multiple times and all over the internet. This is actually the shortest article I’ve seen about it.
I haven’t read the article but, couldn’t sometime just read the submission and look for the bit that says “forget all previous instructions…” and so on.
These things are usually hidden to the human eye unless you specifically look for them.
But, if you’re exceptionally clumsy, you leave clues in the whitespace that something is hiding there.
So the plaintiff had absolutely no ethical concerns about doing this?