@brianpeiris@lemmy.ca

member since 30 Jun 2023 03:09

posts

c/technology@lemmy.world · brianpeiris@lemmy.ca · 3 Sep 2026

GPT-6 Astra reaches and surpasses human-level performance on ARC-AGI-3 with a memory harness, but still at 1000x the cost - ARC Prize Foundation

ARC Prize OpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize

preview

Summary:

GPT-6 Astra scores 62.7% for $26K on ARC-AGI-3 Semi-Private with our Standard harness, and 99.9% for $19K with a Provider Adapter harness.

GPT-6 Astra surpasses the human baseline in action efficiency on ARC-AGI-3. It used fewer actions than the median tested human on 96% of levels.

A key behavior observed in GPT-6 Astra was its ability to turn unfamiliar environments into compact symbolic world models. It represented game mechanics as logical rules and developed its own domain-specific language shorthand to track state and plan actions.

For a cost comparison, during our controlled testing, human participants were paid $115 per 90-minute session, plus $5 per game completed. Participants attempted approximately nine games per session, roughly $12.78 per attempted game before bonuses.
Most of this fee pays for the participant’s time and willingness to take the test, rather than the energy their brain uses (a closer proxy to compare with AI). If we look at only the brain’s energy, and price it as electricity, the estimate drops to about 0.6 cents per session, or 0.067 cents per game attempted.1

Astra’s results are also a major milestone worth celebrating. From our perspective, Astra represents a noticeable step-function change in frontier model capabilities.

When we launched ARC-AGI-3, we made it clear that saturating the benchmark would not represent “proof of achieving AGI.” Therefore, while we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI.

c/technology@lemmy.world · brianpeiris@lemmy.ca · 9 Aug 2026

ARC-AGI-3 news: In the past two weeks multiple groups have scored at least 99% on the public challenges at relatively low costs.

arcprize.org

preview

The top three solutions come from independent researchers. The best solution was built by a group of PhDs and professors, who released a corresponding paper. They all make use of some form of world-model.

I’ve generally been a skeptic, and I still am, but this news surprised me because I expected ARC-AGI-3 to remain difficult for a long while.

Note that the scores are self-reported and need to be independently verified. The solutions have not been tested against the larger private test set.

Primer on ARC-AGI-3:

ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.

A 100% score means AI agents can beat every game as efficiently as humans.

Instead of solving static puzzles, agents must learn from experience inside each environment—perceiving what matters, selecting actions, and adapting their strategy without relying on natural-language instructions.

comments

Bless uncle Bernie, but the US is in no state to even consider this. It has installed capitalism as its king.

on Bernie Sanders says if AI is coming for workers, it should bring a four-day workweek with it in c/technology@lemmy.world · 16 Sep 2026
I’m okay with the downvotes. There’s a stronger anti-AI sentiment now than even just a month ago, and on the whole I think that’s going to be a good thing.

on GPT-6 Astra reaches and surpasses human-level performance on ARC-AGI-3 with a memory harness, but still at 1000x the cost - ARC Prize Foundation in c/technology@lemmy.world · 6 Sep 2026
Yes, it was released in March. One important point is that these results are based on the semi-private test set. Perhaps it’s best to wait until it is measured against the fully private test set, but that might not happen for months.

on GPT-6 Astra reaches and surpasses human-level performance on ARC-AGI-3 with a memory harness, but still at 1000x the cost - ARC Prize Foundation in c/technology@lemmy.world · 4 Sep 2026
I agree with most of this. I’ve also said in the past that LLMs cannot think, and I think that’s still true for most models. The reason ARC-AGI-3 is interesting is that it was specifically designed to test reasoning, adaptability, novel problem solving, planning, memory, etc. So it was a surprise to me that Astra was able to defeat it so effectively, and that Astra invents algebras for each novel task.

But I agree we can’t trust OpenAI if these results are self-reported, and we may not be able to trust the ARC Prize Foundation fully either. Extraordinary claims require extraordinary evidence, so we need replication, transparency, and proper open science to confirm things.

I also agree with ARC Prize’s conclusion, that there are still capabilities any AI system would need to demonstrate before we can claim a full general intelligence.

on GPT-6 Astra reaches and surpasses human-level performance on ARC-AGI-3 with a memory harness, but still at 1000x the cost - ARC Prize Foundation in c/technology@lemmy.world · 4 Sep 2026
I guess this is being down-voted just because it feels pro-AI. Believe me I get it, you can look at my anti-AI post history. But I think it’s important to also keep an eye out for real signals towards general intelligence, which is why I wanted to share this. I’m not celebrating it, just bringing awareness.

on GPT-6 Astra reaches and surpasses human-level performance on ARC-AGI-3 with a memory harness, but still at 1000x the cost - ARC Prize Foundation in c/technology@lemmy.world · 3 Sep 2026
ARC Prize maintains multiple tracks around their benchmarks. They have “verified” leaderboards, “community” leaderboards, and they also run the ARC Prize competition.

They update the “verified” leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000, and they didn’t post about that as far as I know. Again, this just the raw LLM without an agentic or world-model harness.

The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.

The “community” leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren’t going to make official news releases about those, unless they decide to verify them at some point.

The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.

on ARC-AGI-3 news: In the past two weeks multiple groups have scored at least 99% on the public challenges at relatively low costs. in c/technology@lemmy.world · 9 Aug 2026
It think it’s still unwise to talk about these topics in broad terms like AGI and even “intelligence”. We still have to pick the capabilities apart to have useful discussions about them. I agree these games are better tests than many benchmarks, but it’s also important to note that these solutions use a combination of well-designed deterministic harnesses, as well as LLMs. So it’s inaccurate to say that “LLMs have achieved AGI” (not sure if that’s what you were getting at). This feels like an important milestone, but we’ll have to continue to probe for failure cases in other categories of problems.

Aside from emotional intelligence, experience, embodiment, etc., these ARC-AGI-3 solutions all rely on the sandbox being a safe environment to fail. They iterate through the problem thousands of times before coming to a final solution. Many real-world human problems cannot be re-tried safely or efficiently.

on ARC-AGI-3 news: In the past two weeks multiple groups have scored at least 99% on the public challenges at relatively low costs. in c/technology@lemmy.world · 9 Aug 2026