AI’s recursive self-improvement might not come so quickly after all
MIT Technology Review
AI’s recursive self-improvement might not come so quickly after all
AI agents are not yet creative enough to carry out genuinely innovative open-ended AI research, it seems.
What I would have never have guessed. I’m shocked I say. /s
I’m pretty sure this sort of thing was tried with the prior era of neural nets too. When the field hits a ceiling they grasp at the make-AI-teach-itself straw. It’s the Hail Mary pass. What if we keep stacking AIs on top of each other. Maybe they’ll somehow break out of their own limitations.
There’s a cadence. Once in a while a breakthrough happens. The tech is incorporated into the world. There are variations of the tech, but all have the same fundamental ceiling.
The AI Effect takes place. People forget about AI for a while. Time passes. A breakthrough paper is published. AI is upon the world once again.
Only this time with LLMs, it’s seemingly passed the Turing Test so people think it’s close to the fictional AGI. Not just recognizing handwriting, speech, or images. Or putting an annoying animated character on your desktop. This time it’s being freakishly good at predicting what the next words should be based on known sum total of human knowledge. Making it be creative isn’t it this time. That’s the ceiling.
[deleted]
not a reflection on the quality of OP’s submission, but man… like every day now I wish we had an active “noshitsherlock” sub for headlines like these
It’s not being said for the benefit of those who already know.
Someone should DLSS this meme
Using DLSS-5 Anything with varied settings I present to you:
4 inference steps:
20 inference steps:
Turing Test is fundamentally based on fooling humans. Not sure it’s smart to pin humankind’s future on what a birthday magician can do.
How is the Turing test related to the article ?
I mean that describes a majority of engineers. No small feat
If the standard of ML talks at conferences I’ve attended is anything to go by then a top machine learning conference is functionally a daycare for the most annoying people you’ve ever met
Tbh a lot of ai conference papers would absolutely be writable from most engineers. A lot of papers are : we trained a model and got these results. They didn’t publish the weight though, so you would be unable to check the paper.
Lol what? Of course it won’t, If the AI slop ends with recursive edits it’s just going to cause degradation and collapse. I swear techbros have reality confused with their favorite fantasy fiction books.
EDIT: To demonstrate, 90% accuracy of 90% is 81%. Even the best most specific models on earth are not capable of self improvement because they will never reach much less exceed their training data’s capability even if the largest most perfect dataset existed. They might think that by simply adding more layers of machines running in parallel and killing off models which underperform creating a system similar to evolutionary adaptation that it might eventually reach that 91%, but our current approach and level of technology have never demonstrated that capability not even theoretically.
What happened in the computer programming space (with testable outputs) is that the first pass 80% accuracy nailed down an 80% success rate - wrote code that successfully met requirements 4/5 trials. Then, the agents were able to repeat the 1/5 failing trials with “sufficient heat” to both find their problems and create workable solutions, again 4/5 trials - so 80% success rate becomes 96% success rate, and so on… Back in early 2025, programming LLM agents would get themselves caught in iterative loops - trying, failing, trying again, failing again, then trying the first approach again - failing indefinitely. By mid 2026, I don’t see that behavior anymore - if the first “light pass - quick attempt” solution doesn’t succeed, they dig in deeper - do more research specifically focused on the problem areas identified in the first failure and try again, generally successful by the 2nd try, almost always by the 3rd - I haven’t had to break a “trying the first unworkable solution again because I can’t think of anything else to do” loop in over 6 months.
Not all problem spaces are as clear-cut as software creation, but many have similar rules that just take a bit more training to learn.
They Don’t pass 4/5.
They pass 0/5 because they are 80% (that number is way too optimistic btw) accurate to human output on every one of the five attempts.
They also can’t be forced to learn and retake the trial because they don’t have any contextual awareness, they just guess the next word in a sequence.
Even if a machine made 4 self edits sucessfully, it would be permanently disfigured by the one failure and no longer be capable of making good edits.
That’s cool but you’re describing the models from 2 years ago and also not considering harnesses, which account for most of the progress of the last year or so.
You’re entirely right. This won’t replicate to other fields like writing and creative arts in general, but software engineering is just not that hard and can basically be brute forced with a good harness.
It’s a done deal and there is no world where people will write professional code by hand. I like it cause it really separates coding (the job) from coding (the art form). People will code by hand for aesthetic reasons just like people learn the violin instead of using a synth and we’ll have a generation of lovingly crafted stuff. But boring software will be entirely automated, if not generated on the fly based on immediate needs.
I’ll disagree on semantics here, it’s precisely because software engineering is hard (not difficult, but rigid - objective) that makes it a good fit for LLM agent execution. Soft, squishy, ill-defined fields are going to be a worse fit for LLMs because the practitioners themselves can’t create clear cut (hard) definitions of what it is they expect out of their practitioners, they just “know it when they see it.” As for relative difficulty, the “soft” fields have a very sliding scale for that with a lot of allowance given to newbies that isn’t accepted “at the highest levels” whereas, software engineering just is what it is, it doesn’t get more difficult as you progress in the field. Your job as a software architect / engineer is actually to find the easiest workable solution(s).
I think it’s more like: people will code C or Rust or Python by hand just like people still code assembly by hand - exceptionally rare stubbornness with an exceptionally small audience who could even understand what they have done to begin to care about it. Violin vs synth - most of the world can listen and appreciate and have an opinion even if a vanishingly small fraction could ever hope to have the patience, let alone skill, to compose or perform at the highest levels of either form. “Synth” is a very broad target these days, varying from direct composition to performance digital transformation, through interfaces of every description and complexity: simple contact closure keyboards through multi-dimensional velocity, attack angle, strike momentum, and many dimensions of aftertouch bends which allow more expressivity than even bow and fingers on strings do, if the performer cares to train in that popularly scorned field. Having done a little amateur composition to performance vs performance capture synth work, I’ll say: once you have trained to work with the complex input devices, capture of live performance is hundreds of times more efficient than specifying all the nuance of a real performance as notation in a composition. The main reason people hate synth performances is that most synth performances are hack level, because hack level is easier (read: possible) on synth than a minimally passable live performance on violin with strings and bow.
Similarly, most people are hating on AI slop because it’s so easy to produce and so many untalented hacks are using it to produce sub-par whatever it is they are making: code, prose, art, music… used as a tool, with a high bar of standards required before publication and release, LLMs are a powerful tool that can accelerate many creative processes, not just produce a lot of slop quickly.
Yes i think we’re actually in agreement here. I said “hard” (not difficult) as a reference to “hard problems”, a term that comes from complexity theory but is now commonly used to describe problems which can’t be reduced to an algorithm or evaluated objectively, and thus can’t readily be “solved”.
Squishy subjects like music and sociology are full of hard problems, while solid subjects like math and coding are full of easy problems. Now the change introduced by LLMs is that as long as a problem is “easy”, it can no longer be so laborious as to be impossible. Every software problem is solvable, modulo the effort/computing power you can spend on it.
You also get bonus point if your solution is average (standard, unsurprising etc…), which makes it particularly soluble in LLMs which, by definition, can only produce output that is within the distribution of their training set.
That’s not where i would put the difference. If you take a field like music, the problem is that it can’t “just work”. A nostalgic song may move the masses today but you can’t say “okay we’ve solved nostalgia let’s get to serenity next”. Soon enough you’ll need a new nostalgic song and by definition it will be out of distribution. You can’t find it in a high dimensional representation of past music, and, well, you can’t train on future data, so there is no way an LLM finds it and recognizes it for what it is.
I still believe they’ll never amount to much regarding artistic processes, and not just for the reasons i already mentioned. To make something good you need to sit with it and walk with it and spend some time in it doing all the tedious little tasks until it really feels like home and you can express yourself in it. You can’t achieve that if a machine speedruns all the little tasks for you.
The fingerprinting they’re doing now is meant to help with this. If they see a fingerprint, it was probably not human generated so don’t ingest it.
They are not only using human written data any more. Reinforcement learning through human feedback (RLHF) is a big part of it now, that’s the AI running through a problem multiple times with a human picking the best attempt and then using that best attempt as training data going forwards. The model collapse stuff from a few years ago was from an AI repeatedly ingesting its own input with no guidance over many training iterations.
I think people made some crazy magical assumptions about AI based on scifi that doesn’t apply to real life. Real things to consider and prepare for, but not likely.
Recursive exponential self improvement? Just because something knows how to code doesnt mean it can make the best ultimate next version of itself. Even physical evolution takes millions of years, and it creates mistakes and has setbacks.
We might be seeing logarithmic AI improvement today, like evolution hitting a hill that it can’t cross. We might need trillions more gigabytes of clean training data that isn’t LLM generated to hit the next level, and it might not even be worth it.
I think what the AI developers are doing by constantly promoting ai fear is linking the idea of exponential ai self improvement to it, because their biggest fear right now might be investors realizing that isn’t real, and every dollar they invest is getting less and less back.
Exponential improvement is indeed optimistic - a sigmoid curve (plateauing after a period of increase) is much more plausible, though in the computer programming case I haven’t noticed the plateau yet.
Indeed, throughout nature it's almost all sigmoids. The trick is that sigmoids look exponential before the inflection point and it's hard to predict when that inflection point is going to come.
Agreed… I’ve been dabbling in “smart” algorithms for 50 years, the recent (last 8-10 years) progress has been dramatically faster than the previous 40, but each new amazing field: voice transcription, language translation, computer vision object recognition, games mastery, have all rather obviously hit sigmoid-like plateaus. LLM agent software writing has been a slow-burn improvement over the past 18 months - from my perspective it seems like it’s still improving, though that also seems to be a combination of the models getting better, their built in instructions getting better, my local “memory” getting better, and me learning what to challenge it with and what’s unrealistic. A big sign for me is: something I challenged it with 12-14 months ago and got basically nowhere, I tried again last month and it’s made solid progress, delivering a lot of features it couldn’t last year - and those are a lot of features I “gave up on” 5-6 years ago, not because they were impossible, but because they were just too much annoying, time consuming work for the value they deliver to me (personally) - and now the barrier to entry for making those things happen in software is dramatically lower.
Looking at LLM benchmarks over time, there was still doubling of benchmark scores as of 2024 to mid-2025, then it inflected to more modest, incremental improvements, so sigmoid seems about right. Now we are seeing open models rapidly catching up now that the frontier has significantly slowed.
Yeah, improving on 2024 performance was a pretty low bar to clear, by mid-2025 I saw the continuing improvement and decided that even if it was marginal at the time, learning how to use it was probably worthwhile given the improvements that seemed to be coming. Those improvements definitely did come from my perspective. Much of it was in the harnesses - many things I used to have to tell models explicitly, repeatedly in fall of 2025 they started doing without explicit prompting by spring of 2026. I also think I learned what they could be expected to do well and what was a waste of time trying which made me more productive with them as well.
I recently heard that Kimi v3 has made significant progress in code quality - with some people calling it “on par” with Claude. I primarily use Claude Opus - when I tried Fable during their free preview it “felt” even better, but not enough better to shell out a lot of extra cash for personal playtime projects. Kimi isn’t as accessible under the fixed monthly price model so I’m unlikely to try it anytime soon.
I saw this article today summarizing a Mozilla report that open weight models are about 4 months behind the frontier: arstechnica.com/…/exclusive-open-chinese-models-c…
Progress has slowed at the frontier and open weight is right on their heels. The general advice is that Fable and the like are best for niche use cases and to make an open weight model your daily driver because they’re almost as good and are a lot cheaper. In my case for personal use, I’m perfectly happy with Qwen 3.8 27B running locally and don’t care to deal with data collection or huge bills to get a bit stronger of a model.
To switch from Claude to Kimi in service forms available to me would be a 5x price increase for how I use Claude (milking the subscription 7 day token limit dry after 5 days pretty consistently.) Decent self-hosting solutions seem to be running around $20K+ in capital equipment and more than $20 per month in electricity costs alone.
Won’t come at all. It’s mathematically impossible.
Proof?
The output of a statistical model cannot contain more information than what it already had.
So you think that AI increasing the capability of new AI just involves training a new model on the output of the old model? Or are you pretending to be a science?
[deleted]
“That” is not what evolution did.
[deleted]
I want to believe this but every biology major I talked to is like “that’s the middle-school explanation of evolution” and mention the actual process had more things going on.
All that an AI has to do to disprove that claim is pay someone to weigh one orange through fivr. So either your claim is wrong, or LLMs as they exist now are no longer statistical models. Go ahead and move the goal post if you want, my complaint was about your overly broad claim.
I think their point is the LLM didn’t contain the information on the weight of that orange, it had to outsource it.
I picked the way in which an AI model (as the exist now) could most trivially produce a piece of novel (and deliberately trivial) information.
I wasn’t worried about whether the credit could fairly be attributed to it. We’re talking about whether they can produce information with which to grow their knowledge. Moral credit for the growth of their knowledge is outside the scope of the discussion.
It’s entirely possible to improve current AI using only information available to us right now. Once that well runs dry, current AI is in theory capable of running experiments and training on their results if we give it a harness to do that. This gives it access to new information. Could it succeed doing this? Unclear, but it is capable of trying. How do you think we discover AI improvements? Divine inspiration? No, we follow a relatively simple research loop.
you’re drowning in slop my man
You assume too much. I don’t even use AI, I just keep up with the research. I haven’t been wrong so far.
A year ago they were similarly bad at writing code, often created unit tests that tested nothing, etc.
If the models are trained in what they’re doing wrong, that can accelerate their progress toward doing it right.
They would need to be trained for open ended creative tasks, which is just hard in the current reinforcement learning paradigm.
I think they’ll find infinite ways to fuck up. The guardrails will never be high enough, or strong enough.
They still don’t get it right all the time, they just stacked a few together to filter out the obviously wrong stuff.
Mmmh… They used Opus 4.8 (which is already outdated) as the model, and OpenClaw (which is utter garbage) as the harness…
heavy criticism of openclaw seems to be the consensus. recently, model makers have all been overly obssesed with toolcalling models. i expect that aspect to be improving the fastest.