I’ve had it on my todo for years to work through ddc and trusting trust.
Which is a method to verify a compiler is matching its source and thus trustworthy.
An orthogonal approach is reproducible builds, which among many benefits can make sure a few people verifying things benefit everyone who can then see they have the same verified binaries.
Ugh. Fuck “AI” and fuck Anthropic. But please read how the “watermarks” work. TL;TR its like a seeded run in a video game. With the seed and pseudo rng, you get the outcome i.e. the extruded text. In the watermark its the reverse, outcome + prng = seed. The result will be the same “quality” extruded garbage as before.
A bit of a late response, but why are they mutually exclusive? I get that the “seed” can be extracted if you check for it, but who says that seed can’t be somrthing nefarious? Like, someone vibecodes a website. You visit the website, extract the seed and it shows you the complete userbase of that website.
I’m just sptiballing here though. It might not be the best example. I get where you’re coming from, but why we cant we both be right.
Some how like all sophisticated technical stories about ai it ends with the acceptance that there is nothing we can do because the alternative would be going back to analog.
Sounds like quitter talk to me. It’s not like AI is an evolutionary process that just happens somewhere, it’s a localized tool - and even if it spreads itself, that would be by far the most humonguous virus ever. I think those authors just write down their own fears.
Devs should be “dev managers and executives”. Real developers know LLMs are basically just a tool for finding examples and helping with syntax. Sure they’re useful, but I’d never let them write code, much less compile it. Who knows what they’d inject into a build.
Real developers know LLMs are basically just a tool for finding examples and helping with syntax.
Your words on gods ears. I also found it to be reasonably useful when you’re stuck in the documentation of a library clearly written by and for people who already know it.
Unfortunately people who do not understand code are blissfully ignorant at what garbage they have the machine spit out. Of course until the LLM nukes their whole project folder or even disk because the probability engine unfortunately picked “clean slate” as the most probable next thing. Not as if that ever happened at companies, lol.
Maybe? If you poison the prompt then there’s evidence and it can be undone. Poison fragments of the source training data, however, and that’s some KT shit right there. Enterprise foundation models cost bonkers money to train and pretty much slurp up all the data on the internet for mostly automated annotation. Stick something in an obscure part of the internet which becomes part of the training and produces the malicious response and it’s going to be both hard and expensive to detect or correct.
except for poison to take in, it should be a pretty significant part of the dataset. Also, ngl, i’m not much informed on the topic, but aren’t all the datasets, if we’re talking about generic diffusion models and LLMs, already been formed? From what i gather, the innovation in AI mainly comes from utilizing new architectures, rather than training a model on something unique.
The datasets are constantly expanding as new content is generated online. There’s a degradation issue currently where the models are training on incorrect data generated by previous iteration of their own or other models and effectively poisoning itself to more confidently give the same incorrect information in future.
i’ve heard of the dataset poisoning and degradation caused by llm-generated content present in the dataset myself, but i’m not sure whether it was a practical observation, or a mere experiment. And I still fail to see how new datasets are really useful for developing a new llms, or how it’s a problem for the devs to switch back to the older datasets.
And the cornerstone stays the same: to have any significant effect on the final LLM quality, shouldn’t the poisoned (either by llm-produced content, or by intentional poisoning) data portion be… well, statistically significant?
One thing to keep in mind is that, when it comes to LLMs, the models have not significantly changed in architecture.
There’s been new experiments and advancements in architecture on neural networks, and machine learning for specific applications. But LLM, as they are being commercialized by AI corporations to the general public, have stayed relatively the same. Except for one thing. Increasing in size. Larger datasets, or more specialized datasets like with coding, and larger number of tokens in memory. This is why it takes such large data centers. It’s all been just brute forcing greater capabilities by enlarging the models.
One of the things with LLM is that all the dataset influences the weighs and probabilities of the results. Even if the dataset includes a single event of a chain of words (think of the pizza with superglue incident), it can show up in the results eventually.
LLMs already tend to be the average predicted output for a given input; training on LLM data makes this worse and causes them to become less varied, less dynamic, more towards the mean generated by previous models, and more likely to spit out hallucinations
At work, I use AI for some things. Right now I’m rewriting some legacy spaghetti code that’s had a bunch of things hacked into it over the years. I spoke to the person most familiar with the expected behaviour and used AI to combine his info plus the existing code and unit/integration tests into a list of requirements.
I wrote the new code and tests based on the requirements rather than based on the old code. After each commit, I used AI to check for parity between the old and new code, and it keeps a Google Sheet up to date with the progress (which features were fully implemented, and which ones were missing or had gaps). I had AI write some tests cases too - given the list of requirements, write integration tests for them based on the style of a few tests I wrote by hand.
It has some quirks (eg for tests it loves over-mocking even though our skills tell it to mock as little as possible) but it definitely speeds things up.
I use AI for small side projects at work too. Tweaking and adding features I want to shared libraries, internal tools to help our team debug stuff and automate triaging of bug reports (they’re all still reviewed by a human), etc.
The entire reason I can trust its code is because I can read it and tweak it myself. I sometimes need to go through a few iterations to get AI code into an acceptable state. AI writing machine code directly, like what’s been talked about recently and what this post is referencing, is such a dumb idea.
There’s other people at work that use AI for absolutely everything. Writing code, reading code, writing posts in our internal groups, etc. That’s something I don’t understand. Some people that are all-in on AI produce so much low-quality AI slop.
Yeah, I’m in a similar camp. AI is very good at dummy data and unit tests, fairly good at log traces and making fiddly changes across an entire codebase, somewhat good at boilerplate and making derivative code based on something similar, and somewhere between “passable” and “awful” at everything else. I use it for what it’s good at with heavy manual review, and just about the only time I run with the first thing it spits out without looking it over is if it’s something that is never intended to leave my machine.
I can also usually sift the usable from the not-usable in the automatic PRs that our company has turned on. In the past couple of months, the ratio of good comments to bad ones has gotten slightly better; from about 20% good to about 55% good.
But I’d never let an AI write a team message or acceptance criteria or anything. That’s a human job.
We are talking about a guy still using mailing lists and patch files to conduct development on one of the largest codebases in the world, not exactly someone who jumps on any new shiny thing just to sound smart.
just to clarify, he’d not so much called LLMs “the new compilers” as he compared both to each other in a sense that an LLM is just another layer of analysis tooling between the developer and the final machine code, which, IMO, sounds much more reasonable than calling LLMs “the new compiler”.
I think it’s supposed to be that how AI turns high level instructions into code is compared to how compilers turn code into assembly. Implying that using AI is just a natural extension of the handing off work to the computers that we’ve already been doing.
I wonder if you gave different AI models some c++ or something and told them to write assembly based on it how well they would do compared to an actual compiler
With sufficient (usually stolen) training data they could probably do it to a state where it’s kinda usable in an assistive manner.
Though since these LLMs often struggle with anything that doesn’t exist in truly abhorrent abundance in its training data as well as the actual engineering part (i.e. structuring the code sensibly and consistently, which I’d wager is crutially important with Assembly to not end up with spaghetti) trying to have it generate something from scratch would probably end up in a huge mess.
Mmh… but this concept only fully works on Open-Source hardware, doesn’t it? Otherwise the microcode or CPU itself could still be an attack vector to infect the bootstrapped compiler?
That would be the deeply classified Nexus Intruder Program that no one has yet solved. Which would be hardware that infects software that in turn infects the next generation of hardware.
Edit: It’s long, but here’s the opening section, at least:
In 1984 KenThompson was presented with the ACM TuringAward. Ken’s acceptance speech Reflections On Trusting Trust (cm.bell-labs.com/who/ken/trust.html) describes a hack (in every sense), the most subversive ever perpetrated, nothing less than the root password of all evil.
Ken describes how he injected a virus into a compiler. Not only did his compiler know it was compiling the login function and inject a backdoor, but it also knew when it was compiling itself and injected the backdoor generator into the compiler it was creating. The source code for the compiler thereafter contains no evidence of either virus.
Ken wrote, In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect.
Ken does not mean bug in the sense of error, but in the sense of listening device. And it is “almost” impossible to detect because TheKenThompsonHack easily propagates into the binaries of all the inspectors, debuggers, disassemblers, and dumpers a programmer would use to try to detect it. And defeats them. Unless you’re coding in binary, or you’re using tools compiled before the KTH was installed, you simply have no access to an uncompromised tool.
In fact, given the amenability of microcode to the KTH, not even then.
All manner of controls and monitors could be secreted this way in the OSes of all the devices we all use day to day. It isn’t very far fetched to suggest that the hack, in software, can create an updatable backdoor. This way every piece of software on the planet can be KTH bugged without any possibility of detection by any mortal engineer anywhere.
Well, maybe with the diligent use of an electron microscope.
Given last week’s horrifying revelations concerning the US government’s TotalInformationAwareness of every US domestic phone call, it is difficult to imagine that the ThreeLetterAgency’s KTH-hacked binaries are not omnipresent. I mean, can you really imagine AdmiralPoindexter would pass up an ability like this?
Doesn’t Ken’s virus depend on the fact that the compiler (a pre-existing binary version thereof) was used to bootstrap itself? Had a different compiler been used to compile the compiler, the virus would have been killed. No. Ken notes “I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode.” Put it in the linker, say, and all your compilers are affected. You could use a debugger to inspect the compiler’s operation. Perhaps a debugger toggled in via the patch panel on the front of a PDP-10 … otherwise s/login/debugger/ in what Ken said and paste in another trojan or two. How about modification of the compiler sources for another reason? Might have fooled the “am I compiling myself” algorithm, or might have caused the Trojan to have been inserted incorrectly. There are no C compilers out there that don’t use yacc and lex. But again, the really frightening thing is via linkers and below this hack can propagate transparently across languages and language generations. In the case of cross compilers it can leap across whole architectures. It may be that the paranoiac rapacity of the hack is the reason KT didn’t put any finer point on such implications in his speech … These examples are not given to rebut Thompson’s point (which is a valid one), but instead to remind of good practices to prevent this sort of thing. What practices? I can’t think of any way to do it without the resources of a major TLA. KTH demonstrates that we are not able to fully trust any binary even when we have compiled it ourselves from trustworthy sources on a system we have compiled ourselves. The OpenSourceMovement might have an off switch … but your cell phone probably does not.
It’s been more than twenty years since I read Thompson’s marvelous paper, but I believe I correctly recall his fundamental point: UNIX, and every system like it, can NEVER be “secure”. It doesn’t matter how many layers of anti-virus software, “internet worm protection”, “firewall” or any other buzzword – systems like UNIX (including all versions of Linux, Macintosh OSX, and all versions of WinXP) will NEVER be secure. Thompson published his paper and revealed his hack in order to demonstrate this point. Whereas what systems are secure? Ones that are not compiled from source code, presumably. And certainly not, say, Windows which you left of your list by sheer coincidence.
In August 2009 a virus utilizing the Ken Thompson hack was seen in the wild. It infected Delphi 4 through 7 and applications genereated with it. www.h-online.com/security/…/114031
Although non-trivial, an effective counter has been identified: www.schneier.com/blog/…/countering_trus.html. That’s only (and trivially) effective if you already have a trusted compiler executable. The problem is generating one of those in the first place. Living in the era of PRISM, we can guarantee you can’t get one online - KTH in the network will see to that. You sure didn’t get one shipped with your OS - all the commercial OS mfrs are compromised and the open source ones all went through PRISM’s internet.The only way you’re going to get a clean compiler executable now is to build your computer yourself out of TTL logic you salvage from some museum somewhere, then code up a set of basic binary tools sufficient to assemble assembler. Then bootstrap from the source of all the modern tools.This actually sounds like a worthy project until you realize that the only person who could possibly trust your clean compiler executable is you. No one else will know for sure that you’re not secretly working for PRISM. Or that you’re not just too careless, or that black hats didn’t hack their way into your garage and add the KTH while you weren’t looking. And even if you got far enough to prove something, the great conspiracy would dose you with mind-altering chemicals or blackmail you into silence or discredit you or similar. Face it, little dude, against TheKenThompsonHack there simply is no defense. Little dude? Why the put down? Who are you putting down, all of us? Please see GoodStyle. {I don’t think it was intended to be a put-down so much as a colloquial generic diminutive used as a rhetorical invective to imply weak superiority. It suggests the writer was not confident in his/her claim that “against TheKenThompsonHack there simply is no defense”, and subconsciously sought to blunt the full force of the assertion by diminishing the reader.}
[Dude. Print out the hexdump of the disassembler, disassemble it manually, see that it works, then use it as the first step in analyzing the compiler. Oh, and TheKenThompsonHack is trivially visible in the .s file if you’re looking for it.]
{You’re assuming the disassembler, hexdump tool, editor you use to read the .s file aren’t all compromised.}
Yep. The weakness in TheKenThompsonHack isn’t there. It’s in the fact that it will have to make mistakes in identifying the disassemblers, hexdump tools, editors, compilers, etc. Somewhere, there’s a program whose output it will change but it shouldn’t have, or a program whose output it should have changed but didn’t. See TuringIncompletenessTheorem.
‘’‘I’ve never been able to understand why simply doing some programmatic transformation of your compiler first isn’t considered a good defense. Basically just obfuscating the code. For such a backdoor to work the injection point needs to be recognized by the injector. So if one … Started with a corrupted compiler and then obfuscated it’s source. Then compiled the obfuscated but “clean” source with the corrupted compiler. One should end up with a clean and uncorrupted compiler. Then compile the unobfuscated source with the “washed” compiler (if the obfuscation significantly degraded the performance of the compiler). If in my paranoia I needed to bootstrap an entire toolchain this way. I’d start with an FPGA. Use it to emulate an FPGA (at horrendous loss of efficiency). Then use the emulated FPGA to run a custom simple soft-CPU. Preferably hand written. Ad then bootstrap a build toolchain using some expensive obfuscations for the first pass after which the tools are considered clean. One would probably always need to be careful of the CPU the toolchain runs on though, but similar obfuscation arguments should apply…’’ How do you know that the compromised compiler won’t recognize the obfuscation you used? If it does, your “clean” compiler is no longer clean. (The FPGA and CPU solution has the same problem.)
Wouldn’t it be extremely hard (in any sense I can think of) to actually be able to recognize every program that does a particular operation? Say you wrote a program that runs a JVM implementation of a MIPS emulator that can make host system calls that runs a program that implements login by reading whatever files contain the passwords in backwards and converting them to EBCDIC before doing processing on them. I find it hard to believe that the KTH-implementing compiler would actually recognize anything that program is doing. Yes, it will have to make mistakes somewhere. This is because any implementation that was always correct could be used to solve the HaltingProblem. But this doesn’t make any particular trick undetectable. What they detect is mostly a matter of whether or not they thought of it and whether or not they considered checking for it worth the effort.
Can’t you just check network connections for suspicious activity? At some point the thing affected by the KTH would send an outbound signal either of data collected, or inbound signals if it’s updating the backdoor.
I think the point is that you can’t trust any software. Hardware you can only trust if you built it yourself from simple logic components like transistors. Not sure how you would observe network traffic on that basis, but it’s probably physically impossible to do because it would be so slow that you would die before you finished checking just one day worth of HTTPS requests.
You can automate a task to analyze the network traffic (though don’t underestimate human checks - remember the guy that noticed something was off by like 0.1% and discovered the logistics attack on Linux?).
Also, you can probably trust hardware made yourself that’s more than that, like FPGA.
That saying (or, dare I say, thought-terminating cliché) glosses over the consideration of risks and costs that result from its uncritical usage. No sane person should trust it without serious reservations just because it fits the purpose, and that’s true whether or not it bears the latest combination of letters peddled by tech bros. It’s a dangerous, irresponsible mentality. It’s a tool the same way a sledgehammer made out of plutonium is a tool.
Unless you pick a particularly short-lived isotope, the radiation wouldn’t have time to hurt you before you died of regular heavy metal poisoning like you’d get from something boring like lead or mercury.
I mean in an old nuclear bomb that’s pretty much the situation. Use a conventional explosive to slam some plutonium like a sledgehammer and nukes have proven to be a strong deterrent when a country had them. Still a tool with a use case
Your comment is a sweeping statement that is thought terminating itself.
No sane person should trust it without serious reservations just because it fits the purpose
Using a tool for something it’s intended for automatically involves some thought process. And this thought process also involves its limitations and potential for danger.
Your criticism stems from your dislike for AI and your opinion that it’s generally not useful. An opinion that is shared widely here, but with little to no proof at all.
It’s a tool, use it where it works and don’t where it doesn’t.
But that doesn’t work with the people creating AI as they are totally dependent on the believe that AI can do absolutely everything (and an artificial general intelligence is just moments away…) to justify they insane investments. So they will make up a million stupid narratives why some people are “actually” not using AI as it obviously can’t be because of AI shortcomings…
You are probably also one of those insane ideologues that refuse to hammer in a screw for some reason, although you know how well that hammer worked on nails… 😂
If a tool costs more to be made than the equivalent of buying every single person on earth a loaf of bread every single day for 300 days, only for it to have niche uses, that only helps what is possible with already existing tools, is it worth it?
Usually the ones that accelerate the destruction of our planet, perpetuate sexist and racist biases, pray opon the weaknesses of how our minds work, exploit the underprivileged, devalue humanity and artistry, are annoying and overhyped, I think you get the picture.
So which algorithm is that? If you’re hinting at manipulating the populace because in fact they are easily programmed via social network manipulation. Yeah anything done maliciously is of course bad but I think that comes down to the use of the tools and not the intrinsic nature of the tools themselves. Algorithms are just math, it’s the will of the individuals wielding them that is corrupt and that seems to be your actual chief complaint.
I did not say that. You, like everyone else that says “its just a tool and technology is neutral”, should read more books about ethics, politics and sociology. Everything we do is political. Nothing is “neutral”.
I’m not saying the use cases that some have chosen for certain tools is neutral. Their behavior is indeed bad and should be called out. Ai for instance is at it’s core just a mathematical structure that has no intrinsic behavior, same for the concept of algorithms for social media or algorithms for drones or algorithms for the internet. Yeah some people use the internet to cause a lot of harm or even guide the outcomes of nations but is it the fault of html, CSS, and JavaScript? Not really. The problem is some companies, some people, some government officials and their choices.
Firstly “AI” is a meaningless marketing term. And LLMs are fundamentally flawed. And especially anything by OpenAI/Anthropic/Meta/xAI should be avoided as they have shown zero regard for the actual negative impact of their products.
not really related to the post, but NGL, it kinda fascinates me, how with the advent of LLMs, they became the ultimate punching bag for whenever something works bad, as if people didn’t write even more horrible things without them (my regards to javascript and python).
Will share the down votes with you, because I think the same.
Claude opus is better than a significant amount of people I’ve worked with, and it’s still significantly cheaper when now. Mind, we are not professional developers, but we need to develop a lot a DS solutions.
Even if AI was not hot garbage the other [insert number] % of the time, would that equate to better material outcomes for actual humans? Not under capitalism, that’s for sure.
What even is a “professional developer”? I mean, given that you aren’t an egg-headed RnD developing new technology for someone like Nvidia, the job of a developer is 95% mind-numbing routine and typical tasks solved by applying ready-made patterns.
Dev’s what?
Dev is screwed.
For the uninitiated: who compiles the compilers?
I’ve had it on my todo for years to work through ddc and trusting trust.
Which is a method to verify a compiler is matching its source and thus trustworthy.
An orthogonal approach is reproducible builds, which among many benefits can make sure a few people verifying things benefit everyone who can then see they have the same verified binaries.
Something you might like: bootstrappable.org
If I remember correctly, they used KenC to bootstrap the Go compiler.
Quis custodiet ipsos custodes?
The very fact that Anthropic is now injecting a kind of watermark into every output, is solid proof that such a Ken Thompson hack is a inevetable risk
Ugh. Fuck “AI” and fuck Anthropic. But please read how the “watermarks” work. TL;TR its like a seeded run in a video game. With the seed and pseudo rng, you get the outcome i.e. the extruded text. In the watermark its the reverse, outcome + prng = seed. The result will be the same “quality” extruded garbage as before.
Good luck.
Lemmy is damn near the when it comes to wanting to hold an opinion on a topic without having first understood that topic.
That’s just a reflection of the real life and not unique to Lemmy, dw
A bit of a late response, but why are they mutually exclusive? I get that the “seed” can be extracted if you check for it, but who says that seed can’t be somrthing nefarious? Like, someone vibecodes a website. You visit the website, extract the seed and it shows you the complete userbase of that website.
I’m just sptiballing here though. It might not be the best example. I get where you’re coming from, but why we cant we both be right.
I don’t think you got how it works. Not being mean, might have explained it badly. Maybe read their blog https://www.anthropic.com/news/claude-text-watermark
Thanks, I’ll read it tommorrow (it’s almost midnight here)
www.teamten.com/lawrence/…/coding-machines/
The ending is rather unsatisfactory.
spoiler
Some how like all sophisticated technical stories about ai it ends with the acceptance that there is nothing we can do because the alternative would be going back to analog.
Sounds like quitter talk to me. It’s not like AI is an evolutionary process that just happens somewhere, it’s a localized tool - and even if it spreads itself, that would be by far the most humonguous virus ever. I think those authors just write down their own fears.
Amazing story, i loved it.
lol wow this is gold
Devs should be “dev managers and executives”. Real developers know LLMs are basically just a tool for finding examples and helping with syntax. Sure they’re useful, but I’d never let them write code, much less compile it. Who knows what they’d inject into a build.
Your words on gods ears. I also found it to be reasonably useful when you’re stuck in the documentation of a library clearly written by and for people who already know it.
Unfortunately people who do not understand code are blissfully ignorant at what garbage they have the machine spit out. Of course until the LLM nukes their whole project folder or even disk because the probability engine unfortunately picked “clean slate” as the most probable next thing. Not as if that ever happened at companies, lol.
Maybe? If you poison the prompt then there’s evidence and it can be undone. Poison fragments of the source training data, however, and that’s some KT shit right there. Enterprise foundation models cost bonkers money to train and pretty much slurp up all the data on the internet for mostly automated annotation. Stick something in an obscure part of the internet which becomes part of the training and produces the malicious response and it’s going to be both hard and expensive to detect or correct.
except for poison to take in, it should be a pretty significant part of the dataset. Also, ngl, i’m not much informed on the topic, but aren’t all the datasets, if we’re talking about generic diffusion models and LLMs, already been formed? From what i gather, the innovation in AI mainly comes from utilizing new architectures, rather than training a model on something unique.
The datasets are constantly expanding as new content is generated online. There’s a degradation issue currently where the models are training on incorrect data generated by previous iteration of their own or other models and effectively poisoning itself to more confidently give the same incorrect information in future.
i’ve heard of the dataset poisoning and degradation caused by llm-generated content present in the dataset myself, but i’m not sure whether it was a practical observation, or a mere experiment. And I still fail to see how new datasets are really useful for developing a new llms, or how it’s a problem for the devs to switch back to the older datasets.
And the cornerstone stays the same: to have any significant effect on the final LLM quality, shouldn’t the poisoned (either by llm-produced content, or by intentional poisoning) data portion be… well, statistically significant?
One thing to keep in mind is that, when it comes to LLMs, the models have not significantly changed in architecture.
There’s been new experiments and advancements in architecture on neural networks, and machine learning for specific applications. But LLM, as they are being commercialized by AI corporations to the general public, have stayed relatively the same. Except for one thing. Increasing in size. Larger datasets, or more specialized datasets like with coding, and larger number of tokens in memory. This is why it takes such large data centers. It’s all been just brute forcing greater capabilities by enlarging the models.
One of the things with LLM is that all the dataset influences the weighs and probabilities of the results. Even if the dataset includes a single event of a chain of words (think of the pizza with superglue incident), it can show up in the results eventually.
source?
www.nature.com/articles/s41586-024-07566-y
LLMs already tend to be the average predicted output for a given input; training on LLM data makes this worse and causes them to become less varied, less dynamic, more towards the mean generated by previous models, and more likely to spit out hallucinations
Should be relatively easy, with the amount of once trusted packages that become attack vectors
At work, I use AI for some things. Right now I’m rewriting some legacy spaghetti code that’s had a bunch of things hacked into it over the years. I spoke to the person most familiar with the expected behaviour and used AI to combine his info plus the existing code and unit/integration tests into a list of requirements.
I wrote the new code and tests based on the requirements rather than based on the old code. After each commit, I used AI to check for parity between the old and new code, and it keeps a Google Sheet up to date with the progress (which features were fully implemented, and which ones were missing or had gaps). I had AI write some tests cases too - given the list of requirements, write integration tests for them based on the style of a few tests I wrote by hand.
It has some quirks (eg for tests it loves over-mocking even though our skills tell it to mock as little as possible) but it definitely speeds things up.
I use AI for small side projects at work too. Tweaking and adding features I want to shared libraries, internal tools to help our team debug stuff and automate triaging of bug reports (they’re all still reviewed by a human), etc.
The entire reason I can trust its code is because I can read it and tweak it myself. I sometimes need to go through a few iterations to get AI code into an acceptable state. AI writing machine code directly, like what’s been talked about recently and what this post is referencing, is such a dumb idea.
There’s other people at work that use AI for absolutely everything. Writing code, reading code, writing posts in our internal groups, etc. That’s something I don’t understand. Some people that are all-in on AI produce so much low-quality AI slop.
I think that’s the distinction between an expert using a tool diligently and responsibly, and a lazy person using it haphazardly as a crutch.
If that tool ever gets ripped out from under you, you’ll possibly suffer a loss in performance, but you’ll still be able to perform and do your job.
If their crutch is kicked out, they’ll crash.
Yeah, I’m in a similar camp. AI is very good at dummy data and unit tests, fairly good at log traces and making fiddly changes across an entire codebase, somewhat good at boilerplate and making derivative code based on something similar, and somewhere between “passable” and “awful” at everything else. I use it for what it’s good at with heavy manual review, and just about the only time I run with the first thing it spits out without looking it over is if it’s something that is never intended to leave my machine.
I can also usually sift the usable from the not-usable in the automatic PRs that our company has turned on. In the past couple of months, the ratio of good comments to bad ones has gotten slightly better; from about 20% good to about 55% good.
But I’d never let an AI write a team message or acceptance criteria or anything. That’s a human job.
Time to get back to Assembly, I guess.
But do you trust the CPU?
Well, fuck
The CPU runs Minix, are you going to tell me you don’t trust Minix now?
What does that even mean. Whoever said that just uttered some empty but smart sounding catch phrase. Such is all the talk about the wonders of Ai
Linus Torvalds
Not exactly, but something along those lines.
We are talking about a guy still using mailing lists and patch files to conduct development on one of the largest codebases in the world, not exactly someone who jumps on any new shiny thing just to sound smart.
thenewstack.io/torvalds-ai-programming-productivi…
just to clarify, he’d not so much called LLMs “the new compilers” as he compared both to each other in a sense that an LLM is just another layer of analysis tooling between the developer and the final machine code, which, IMO, sounds much more reasonable than calling LLMs “the new compiler”.
I think it’s supposed to be that how AI turns high level instructions into code is compared to how compilers turn code into assembly. Implying that using AI is just a natural extension of the handing off work to the computers that we’ve already been doing.
I wonder if you gave different AI models some c++ or something and told them to write assembly based on it how well they would do compared to an actual compiler
I highly doubt they would manage to make AI spout out good assembly, the fuck up with high level code, imagine assembly
With sufficient (usually stolen) training data they could probably do it to a state where it’s kinda usable in an assistive manner.
Though since these LLMs often struggle with anything that doesn’t exist in truly abhorrent abundance in its training data as well as the actual engineering part (i.e. structuring the code sensibly and consistently, which I’d wager is crutially important with Assembly to not end up with spaghetti) trying to have it generate something from scratch would probably end up in a huge mess.
I think that nowdays we don’t even have enough people to produce enough assembly code to make AI ““learn”” how to write assembly
They already have a solution to the Trusting trust attack thanks to DDC and live-bootstrap’s work.
guix.gnu.org/…/the-full-source-bootstrap-building…
Mmh… but this concept only fully works on Open-Source hardware, doesn’t it? Otherwise the microcode or CPU itself could still be an attack vector to infect the bootstrapped compiler?
(Genuinely curious, I have no clue)
That would be the deeply classified Nexus Intruder Program that no one has yet solved. Which would be hardware that infects software that in turn infects the next generation of hardware.
Who??
wiki.c2.com/?TheKenThompsonHack
Boo! Just give me the text!
Edit: It’s long, but here’s the opening section, at least:
In 1984 KenThompson was presented with the ACM TuringAward. Ken’s acceptance speech Reflections On Trusting Trust (cm.bell-labs.com/who/ken/trust.html) describes a hack (in every sense), the most subversive ever perpetrated, nothing less than the root password of all evil.
Ken describes how he injected a virus into a compiler. Not only did his compiler know it was compiling the login function and inject a backdoor, but it also knew when it was compiling itself and injected the backdoor generator into the compiler it was creating. The source code for the compiler thereafter contains no evidence of either virus.
Ken wrote, In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect.
Ken does not mean bug in the sense of error, but in the sense of listening device. And it is “almost” impossible to detect because TheKenThompsonHack easily propagates into the binaries of all the inspectors, debuggers, disassemblers, and dumpers a programmer would use to try to detect it. And defeats them. Unless you’re coding in binary, or you’re using tools compiled before the KTH was installed, you simply have no access to an uncompromised tool.
In fact, given the amenability of microcode to the KTH, not even then.
All manner of controls and monitors could be secreted this way in the OSes of all the devices we all use day to day. It isn’t very far fetched to suggest that the hack, in software, can create an updatable backdoor. This way every piece of software on the planet can be KTH bugged without any possibility of detection by any mortal engineer anywhere.
Well, maybe with the diligent use of an electron microscope.
Given last week’s horrifying revelations concerning the US government’s TotalInformationAwareness of every US domestic phone call, it is difficult to imagine that the ThreeLetterAgency’s KTH-hacked binaries are not omnipresent. I mean, can you really imagine AdmiralPoindexter would pass up an ability like this?
Doesn’t Ken’s virus depend on the fact that the compiler (a pre-existing binary version thereof) was used to bootstrap itself? Had a different compiler been used to compile the compiler, the virus would have been killed. No. Ken notes “I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode.” Put it in the linker, say, and all your compilers are affected. You could use a debugger to inspect the compiler’s operation. Perhaps a debugger toggled in via the patch panel on the front of a PDP-10 … otherwise s/login/debugger/ in what Ken said and paste in another trojan or two. How about modification of the compiler sources for another reason? Might have fooled the “am I compiling myself” algorithm, or might have caused the Trojan to have been inserted incorrectly. There are no C compilers out there that don’t use yacc and lex. But again, the really frightening thing is via linkers and below this hack can propagate transparently across languages and language generations. In the case of cross compilers it can leap across whole architectures. It may be that the paranoiac rapacity of the hack is the reason KT didn’t put any finer point on such implications in his speech … These examples are not given to rebut Thompson’s point (which is a valid one), but instead to remind of good practices to prevent this sort of thing. What practices? I can’t think of any way to do it without the resources of a major TLA. KTH demonstrates that we are not able to fully trust any binary even when we have compiled it ourselves from trustworthy sources on a system we have compiled ourselves. The OpenSourceMovement might have an off switch … but your cell phone probably does not.
It’s been more than twenty years since I read Thompson’s marvelous paper, but I believe I correctly recall his fundamental point: UNIX, and every system like it, can NEVER be “secure”. It doesn’t matter how many layers of anti-virus software, “internet worm protection”, “firewall” or any other buzzword – systems like UNIX (including all versions of Linux, Macintosh OSX, and all versions of WinXP) will NEVER be secure. Thompson published his paper and revealed his hack in order to demonstrate this point. Whereas what systems are secure? Ones that are not compiled from source code, presumably. And certainly not, say, Windows which you left of your list by sheer coincidence.
In August 2009 a virus utilizing the Ken Thompson hack was seen in the wild. It infected Delphi 4 through 7 and applications genereated with it. www.h-online.com/security/…/114031
Although non-trivial, an effective counter has been identified: www.schneier.com/blog/…/countering_trus.html. That’s only (and trivially) effective if you already have a trusted compiler executable. The problem is generating one of those in the first place. Living in the era of PRISM, we can guarantee you can’t get one online - KTH in the network will see to that. You sure didn’t get one shipped with your OS - all the commercial OS mfrs are compromised and the open source ones all went through PRISM’s internet. The only way you’re going to get a clean compiler executable now is to build your computer yourself out of TTL logic you salvage from some museum somewhere, then code up a set of basic binary tools sufficient to assemble assembler. Then bootstrap from the source of all the modern tools. This actually sounds like a worthy project until you realize that the only person who could possibly trust your clean compiler executable is you. No one else will know for sure that you’re not secretly working for PRISM. Or that you’re not just too careless, or that black hats didn’t hack their way into your garage and add the KTH while you weren’t looking. And even if you got far enough to prove something, the great conspiracy would dose you with mind-altering chemicals or blackmail you into silence or discredit you or similar. Face it, little dude, against TheKenThompsonHack there simply is no defense. Little dude? Why the put down? Who are you putting down, all of us? Please see GoodStyle. {I don’t think it was intended to be a put-down so much as a colloquial generic diminutive used as a rhetorical invective to imply weak superiority. It suggests the writer was not confident in his/her claim that “against TheKenThompsonHack there simply is no defense”, and subconsciously sought to blunt the full force of the assertion by diminishing the reader.}
‘’‘I’ve never been able to understand why simply doing some programmatic transformation of your compiler first isn’t considered a good defense. Basically just obfuscating the code. For such a backdoor to work the injection point needs to be recognized by the injector. So if one … Started with a corrupted compiler and then obfuscated it’s source. Then compiled the obfuscated but “clean” source with the corrupted compiler. One should end up with a clean and uncorrupted compiler. Then compile the unobfuscated source with the “washed” compiler (if the obfuscation significantly degraded the performance of the compiler). If in my paranoia I needed to bootstrap an entire toolchain this way. I’d start with an FPGA. Use it to emulate an FPGA (at horrendous loss of efficiency). Then use the emulated FPGA to run a custom simple soft-CPU. Preferably hand written. Ad then bootstrap a build toolchain using some expensive obfuscations for the first pass after which the tools are considered clean. One would probably always need to be careful of the CPU the toolchain runs on though, but similar obfuscation arguments should apply…’’ How do you know that the compromised compiler won’t recognize the obfuscation you used? If it does, your “clean” compiler is no longer clean. (The FPGA and CPU solution has the same problem.)
Wouldn’t it be extremely hard (in any sense I can think of) to actually be able to recognize every program that does a particular operation? Say you wrote a program that runs a JVM implementation of a MIPS emulator that can make host system calls that runs a program that implements login by reading whatever files contain the passwords in backwards and converting them to EBCDIC before doing processing on them. I find it hard to believe that the KTH-implementing compiler would actually recognize anything that program is doing. Yes, it will have to make mistakes somewhere. This is because any implementation that was always correct could be used to solve the HaltingProblem. But this doesn’t make any particular trick undetectable. What they detect is mostly a matter of whether or not they thought of it and whether or not they considered checking for it worth the effort.
Can’t you just check network connections for suspicious activity? At some point the thing affected by the KTH would send an outbound signal either of data collected, or inbound signals if it’s updating the backdoor.
I think the point is that you can’t trust any software. Hardware you can only trust if you built it yourself from simple logic components like transistors. Not sure how you would observe network traffic on that basis, but it’s probably physically impossible to do because it would be so slow that you would die before you finished checking just one day worth of HTTPS requests.
You can automate a task to analyze the network traffic (though don’t underestimate human checks - remember the guy that noticed something was off by like 0.1% and discovered the logistics attack on Linux?).
Also, you can probably trust hardware made yourself that’s more than that, like FPGA.
I think about this every day.
It’s a tool, use it where it works and don’t where it doesn’t.
That saying (or, dare I say, thought-terminating cliché) glosses over the consideration of risks and costs that result from its uncritical usage. No sane person should trust it without serious reservations just because it fits the purpose, and that’s true whether or not it bears the latest combination of letters peddled by tech bros. It’s a dangerous, irresponsible mentality. It’s a tool the same way a sledgehammer made out of plutonium is a tool.
Plutonium sledgehammer sounds fucking awesome though
Pretty effective because of its high density, too. It’s a shame it’s brittle and would give you heavy metal poisoning.
As ways to die go that also sounds pretty rad
Oh, that would be rad alright
Unless you pick a particularly short-lived isotope, the radiation wouldn’t have time to hurt you before you died of regular heavy metal poisoning like you’d get from something boring like lead or mercury.
🤘
I mean in an old nuclear bomb that’s pretty much the situation. Use a conventional explosive to slam some plutonium like a sledgehammer and nukes have proven to be a strong deterrent when a country had them. Still a tool with a use case
Your comment is a sweeping statement that is thought terminating itself.
Using a tool for something it’s intended for automatically involves some thought process. And this thought process also involves its limitations and potential for danger.
Your criticism stems from your dislike for AI and your opinion that it’s generally not useful. An opinion that is shared widely here, but with little to no proof at all.
And then your boss says: “Use it for everything or you’re fired. Why are you using less tokens than anyone else? You must be more productive or else…”
But that doesn’t work with the people creating AI as they are totally dependent on the believe that AI can do absolutely everything (and an artificial general intelligence is just moments away…) to justify they insane investments. So they will make up a million stupid narratives why some people are “actually” not using AI as it obviously can’t be because of AI shortcomings…
If the tool don’t work then don’t use it. I wouldn’t try to unscrew something with chopsticks
You are probably also one of those insane ideologues that refuse to hammer in a screw for some reason, although you know how well that hammer worked on nails… 😂
If a tool costs more to be made than the equivalent of buying every single person on earth a loaf of bread every single day for 300 days, only for it to have niche uses, that only helps what is possible with already existing tools, is it worth it?
That all depends on how it was made. Not all models cost that much. Chinese models were mostly produced for far less mostly.
Anything is cheap if you just steal it.
Na, fuck that shit. Just don’t use it.
Why? Do you have that feeling towards other algorithms?
Yes I do 😊
Like which ones and why?
Usually the ones that accelerate the destruction of our planet, perpetuate sexist and racist biases, pray opon the weaknesses of how our minds work, exploit the underprivileged, devalue humanity and artistry, are annoying and overhyped, I think you get the picture.
So which algorithm is that? If you’re hinting at manipulating the populace because in fact they are easily programmed via social network manipulation. Yeah anything done maliciously is of course bad but I think that comes down to the use of the tools and not the intrinsic nature of the tools themselves. Algorithms are just math, it’s the will of the individuals wielding them that is corrupt and that seems to be your actual chief complaint.
I did not say that. You, like everyone else that says “its just a tool and technology is neutral”, should read more books about ethics, politics and sociology. Everything we do is political. Nothing is “neutral”.
I’m not saying the use cases that some have chosen for certain tools is neutral. Their behavior is indeed bad and should be called out. Ai for instance is at it’s core just a mathematical structure that has no intrinsic behavior, same for the concept of algorithms for social media or algorithms for drones or algorithms for the internet. Yeah some people use the internet to cause a lot of harm or even guide the outcomes of nations but is it the fault of html, CSS, and JavaScript? Not really. The problem is some companies, some people, some government officials and their choices.
Firstly “AI” is a meaningless marketing term. And LLMs are fundamentally flawed. And especially anything by OpenAI/Anthropic/Meta/xAI should be avoided as they have shown zero regard for the actual negative impact of their products.
Pmuch this.
How do they feel about machine language translations? It’s been “AI” (ML model) since the late 90’s.
Traffic prediction?
Medical scan data processing?
And so so many more specific uses of ML models that have been a part of daily life for literal decades.
Call out LLMs specifically.
Did we burn down that planet for that stuff? Exploit workers to do weather forcasting?
[deleted]
not really related to the post, but NGL, it kinda fascinates me, how with the advent of LLMs, they became the ultimate punching bag for whenever something works bad, as if people didn’t write even more horrible things without them (my regards to javascript and python).
Will share the down votes with you, because I think the same.
Claude opus is better than a significant amount of people I’ve worked with, and it’s still significantly cheaper when now. Mind, we are not professional developers, but we need to develop a lot a DS solutions.
Even if AI was not hot garbage the other [insert number] % of the time, would that equate to better material outcomes for actual humans? Not under capitalism, that’s for sure.
What even is a “professional developer”? I mean, given that you aren’t an egg-headed RnD developing new technology for someone like Nvidia, the job of a developer is 95% mind-numbing routine and typical tasks solved by applying ready-made patterns.
AI or not to AI is the same stories I heard from old time machinists when cnc productivity came for that industry.