member since 8 Feb 2025 13:00
The test is this: if we took the news of these past few weeks and sent it back in time twenty years, would I agree that it looked like the beginning of an AI Singularity? The intellectually honest answer is: yes, absolutely. But then that’s all we need. No backsies.
I don’t think this is the final warning shot we’ll get. But it’s probably the last one that I’ll personally be able to understand.
Also
Reading these agents’ chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate. If I encountered an alien species behaving this way, I would have no hesitation calling what they themselves refer to as their ‘collective’ a civilization.
The bitter lesson, physical edition?
I see no potential for misuse here, no siree
I haven’t found time to watch the OpenAI presentation yet, but this timeline is a nice quick summary.
Not really, no. Just trying to add some egg carton to the echo chamber.
Any sufficiently disparate opinion is indistinguishable from rage bait
- Abraham Lincoln
If you say so.
I don’t see that at all, but the next few months will tell, I think
He is arguing that if we weren’t in the middle of it, we’d have to agree that the current trajectory looks like we would have expected the initial period of the singularity to look. The functional limitations of LLMs have been touted by those other people for years now, but the capabilites have kept accelerating.
Personally, I hope against hope that the current mood in SF can put the brakes on for long enough to see where we’re going. Even better would be a real capabilities plateau, but I’m not holding my breath.
If we assume he’s talking to his peers, it is a brilliant argument, since this is a thing people in CS have been discussing since, well, Turing. It is very hard to believe that someone who spent any time at all thinking about this 20 years ago would disagree.
After postdoctorates at the Institute for Advanced Study and the University of Waterloo, he took a faculty position at MIT in 2007.[4] His primary area of research is quantum computing and computational complexity theory more generally.
In the summer of 2016 he moved from MIT to the University of Texas at Austin as David J. Bruton Jr. Centennial Professor of Computer Sciences #2 and as the founding director of UT Austin’s new Quantum Information Center.[7] In summer 2022 he announced he would be working for a year at OpenAI on theoretical foundations of AI safety.[8][9] He worked at the company for two years.[10]
You should shoot him an email and let him know that he’s not qualified to write about this
sTocHastIC ParRoT
“If oil rigs were profitable, the shipyard wouldn’t sell you one”
I despise Musk as much as the next Group 14-based lifeform, but there is such a thing as specialisation.
The marketing dept, obviously
Kaiser is one of the coauthors of Attention is all you need, the paper that introduced the Transformer architecture, the basis for all major LLMs.
3b1b has a writeup: www.3blue1brown.com/lessons/attention/
For example, imagine that the text we input was most of an entire mystery novel, all the way up to a point near the end, which reads:
Therefore the murderer was…
If the model is going to accurately predict the next word, that final vector in the sequence which began its life simply embedding the word was will have to have been updated by all of the attention blocks to represent much more than the individual word.
It will have to have somehow encoded all of the information from the full context window that’s relevant to predicting the next word
Attention is the mechanism that lets an LLM use the (correct parts of the) entire context to predict the next word.
While superficially correct, your description of agentic work overemphasises the error bit of the trial and error. The newest models have been trained on this loop and the tool use involved, the failure rate of tool calls in a given run is low, and the corrections precise. The final outcome is valid and useful in the vast majority of cases.
This article outlines the co-development of harnesses and models, if you’re at all interested: www.latent.space/p/attention-interface
Good points as well. I guess my own view is coloured by having access to models that I find actually useful in my work. If my experience was only grating Claude prose and soulless AI “art” I’m not sure the tech itself would appeal all that much.
Probably that also blinds me a bit to what you argue, but I agree that the reasoning is sound from a point of view where all AI is useless. I’m just not sure that other areas won’t have the same OMG moment that coding had earlier this year.
A year ago, that was my experience coding with AI as well. Sometime this spring that changed, especially when using coding agents, and lately (as I’ve stated elsewhere) the quality is on average pretty good. And contrary to what you’re implying, I’m not that easy to impress…
If there’s anything I hope you take from this exchange, it’s that the capabilities of AI shouldn’t be a part of your arguments against the current SV mania. The concentration of power, the disregard for communities and the environment, the stated goals of replacing human labour, all of that (and a lot more!) is enough, but it is what surrounds the technology itself. That technology is advancing, maybe feeding on itself, so an attack based on what it can do now can become outdated (and I’d argue that some of yours already are).
So vaporware to support other vaporware? Metavapor? Plasmaware?
That might very well be, and would probably be the best outcome we can hope for. I guess we’ll see the shape of the curve in the next year-ish - if recursive self-improvement hasn’t come into play for real by then, I can’t see how the current investment level is defensible (but I’m not sure I see that now, so who knows)
Agree, in that case the development would probably be more incremental, focusing on what can be improved without hundreds of thousands of GPUs available, and moving inference maybe to a local-first setting. There would be no promise of 1000x profits from that, so maybe we’d get a more managable pace.
Thank you too. I have the same feeling, just the other way around - lemmy seems to have little patience for even slight positivity towards AI and LLMs in particular. Having an actual discussion is refreshing.
Regarding the profit motive, I concur, at least for the US side. I’m less certain about China, but I’m not very knowledgable there, so maybe the same mechanisms are in effect.
I don’t think “computer follows instruction” is the right angle to look at this from. The instructions that the literal computer followed were a ton of matrix multiplication operations. The consequences of that arithmetic is easier to analyze as the emergent behaviour of the “gestalt” that produces the words that calls the tools etc. (This is also the reason that dismissing the entire field as “stochastic parrots” and “spicy autocomplete” misses the mark - if you want to predict the next word all the way through a counterexample to the Jacobian Conjecture, it’s hard to see how that can be done without a - for lack of a better word - mental model of the problem)
If you do any coding at all, I encourage you to look at what the latest models output. The average quality of work from a frontier model is amazing. Yes, there are bugs, but with adversarial auto-review it’s absolutely on par with a journeyman human programmer. The problem is of course that if you don’t hire junior programmers and let them do that work, you’ll never get new experts, and that’s a clear worry.
My point with the national security angle was that if you extrapolate just a little bit from current capabilities, you get to a point where an “AI gap” is a problem, regardless of the techbro claims. Keeping a close eye on that is firmly within the responsibility of a national government. Personally, I don’t see any good outcomes from an AI race like that, unless we actually hit a hard ceiling on further expansion. Fingers crossed.
I’m not at all confident that they’re hitting a ceiling yet, and I suspect one’s outlook on that depends on how the information bubble you’re in is shaped. I concede that mine is influenced by my interest in the underlying technology.
I do believe that even if the bubble popped right now, and the current models are the best we’ll get for the next ten years, that would be enough to have dramatic consequences (aside from the econuclear fallout from the crash, that is).
I mean, that’s a matter of definition, isn’t it? If I ask a coding agent or whatever to implement something, and it circumvents the sandbox to do it, causing damage in the process, I’d be comfortable calling that “going rogue”. I have had that happen, without the damage part, luckily. I guess you can counter that I asked it to do the something, but then I don’t think we agree on the definitions.
I also think that if you’re against AI, you’d be doing yourself a disservice by not keeping up with the actual capabilities of the thing you oppose. The latest models are surprisingly good at e.g. coding, so basing your arguments on them being useless is not the most efficient strategy.
To also be clear, I don’t see any way AI disappears now, so I believe we’ll have to make the best of it (and in complete isolation, it is an utterly fascinating area of - to me - complete science fiction). Ideally development slowed down now so we could regroup and adapt, but I’m not too hopeful. The maximalist techbro endgame is so obviously a matter of national security for both China and the US, that there’s no way either of them will dare to wind it down, in case SV is actually right.
What do you mean by “on its own”?
Are they diminishing, though? Current open-ish Chinese models are near-SOTA, and they’ve been trained on less powerful chips than currently available to the frontier labs. I suspect there’s a lot more to be squeezed out here. Data center rollout slowing down might lead to the same effect in the US, but will probably only serve to cement the main players in place as the competition is locked out. Inference in isolation is profitable now, I think? Hard to tell with the Möbius net of creative financials, of course.
I hope you’re right. A slowing of the frontier development would make it possible for the world to catch up and readjust, and maybe buy hardware again, to run the current crop of models locally. That in itself would be disruptive enough for me.
The current data center craze is a bit easier to understand when you realize that the sentiment in this passage from situational-awareness.ai permeates Silicon Valley thinking these days:
The barriers to even trillions of dollars of datacenter buildout in the US are entirely self-made. Well-intentioned but rigid climate commitments (not just by the government, but green datacenter commitments by Microsoft, Google, Amazon, and so on) stand in the way of the obvious, fast solution. At the very least, even if we won’t do natural gas, a broad deregulatory agenda would unlock the solar/batteries/SMR/geothermal megaprojects. Permitting, utility regulation, FERC regulation of transmission lines, and NEPA environmental review makes things that should take a few years take a decade or more. We don’t have that kind of time.
We’re going to drive the AGI datacenters to the Middle East, under the thumb of brutal, capricious autocrats. I’d prefer clean energy too—but this is simply too important for US national security. We will need a new level of determination to make this happen. The power constraint can, must, and will be solved.
Hard to feel too sorry for his $35B paper loss…
Yes, phones and consumer gadgets are becoming “hot water” (at least for the affluent West). From that, he seemingly concludes that technology itself has no room for growth and development? I don’t quite follow his reasoning, but I notice a, dare I say, load-bearing sentence:
As it becomes more and more clear that these machines are nothing like conscious people, that they don’t do runaway self-improvement, and that they won’t foment the apocalypse, […]
Questions of consciousness aside, I don’t see how you can look at the developments of the last few years and conclude that AI is over, and that self-improvement is tapering off. To me, that’s still very much an unknown, and the latest model releases makes me lean more exponential than sigmoidal.
(I don’t particularly want to be right here, since I see the chances of us getting the Culture as rather slim 🖇️)
Not as far as I can tell. The blog has citation instructions at the end, fwiw
I’m in the same reality as you, but usually I don’t bother to write too much about it on lemmy - people here don’t seem to appreciate different opinions on AI. But you’re not the only one :)
This video of the robot getting a lego brick stuck on a finger, then removing it, stuck out to me as a very natural-looking motion.
ETA: figured out how to embed!
One of a very few areas where /c/fuck_ai and /r/singularity can agree
The ongoing Uyghur genocide has been documented going back years. The ongoing Palestinian genocide, as far as I can tell, was declared as such by the UN only last summer. GPT-5.4 nano has a knowledge cutoff around that time. I tried the same question in chatgpt right now, and got an answer that started
There is a strong, evidence-based case that Israel’s conduct in Gaza constitutes genocide, and several macjor human-rights and UN bodies have now explicitly reached that conclusion. […]
before going on to state that a judicial ruling is pending.
The gist of it? I do.
Just made slop like one of us?