> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.
This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.
Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE
Astra is a big jump from sol and performs the same or slightly better than fable here.
anon373839 16 hours ago [-]
Hallucinations are very damaging to a model’s utility. But doesn’t the Omniscience Index focus on knowledge-based queries? To me, using LLMs for their memorized knowledge is very 2023 and suboptimal.
IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.
jascha_eng 7 hours ago [-]
Fair but I think those two behaviours are strongly correlated at least the index does represent my personal experience very well where fable is way better than opus opus is better than sol. And I haven't tried Astra yet but it having 44/43 is very interesting at least.
gizmodo59 18 hours ago [-]
I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.
testycool 12 hours ago [-]
Also ArtificialAnalysis drop older foundational models to make room for new ones.
If you customize the filter to add Gemini 3.1 Pro you'll see it ranks 5th in AA-Omniscince Index. Yet by default you wont see Gemini 3.1 Pro.
I find this to be very unhelpful and confusing.
jascha_eng 51 minutes ago [-]
But it is actually a great model it e.g. got the carwash question right from 9 months ago. While openais models all struggled.
jascha_eng 7 hours ago [-]
I dont think you read my message. Muse and sol are nowhere near fable and Astra on the omniscience index
jesuslop 18 hours ago [-]
Is there something better over there you'd recommend?
yorwba 10 hours ago [-]
The Epoch Capabilities Index uses an Elo-based aggregation method that dynamically adjusts for benchmark difficulty and they put error bars on their scores, both of which put them miles ahead of Artificial Analysis: https://epoch.ai/eci?view=graph&tab=leaderboard
re-thc 6 hours ago [-]
> A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses.
It's not useful because the benchmarks often measure the wrong thing. They're here yapping about AGI and yet the benchmarks treat it like a trained dog. Fetch this. 100 points.
Each "problem" in these benchmarks likely has more than 1 solution that can be considered correct and even should be graded in many ways. Yet we see in many benchmarks higher effort (or thinking levels) don't help because the benchmark penalizes for doing "more" than what the answers asks for. So what did you ask for?
In human school you often get marks on the process and not just the end result. Thinking tokens have been cut. All we group on is things like cost, turns and time but not the what else.
Buoylog 18 hours ago [-]
This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
redox99 19 hours ago [-]
They realized Astra having the same score as Sol was silly so they rushed to update the index so it fits what people expect.
The old index was clearly bad (Astra is way better than Sol) but it's also unscientific to tweak it like this.
paimapi 18 hours ago [-]
is any of this 'scientific'? does AA allow peer review of its processes? are these published in journals of at least medium impact? what are the sample sizes? how grounded is their mechanistic reasoning?
like they have words that are dressed in scientific language on their site like
"We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2."
but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?
there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives
3 hours ago [-]
Wheen 3 hours ago [-]
> but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?
It's not a probability. It's essentially an assertion that if the test were run 100 times, the result would be within the interval 95 times.
elvin_d 14 hours ago [-]
why keeping journals as an argument on the tech site that was always less formal with mandatory institutions but more open source. Journals discredited themselves multiple times. tech is expanding boundaries of scientific methods and AI will push it more.
tancop 11 hours ago [-]
It's not about journals. If they want to be 100 trustworthy they should release end to end reproducible pipelines for the whole process with everything but the private datasets included, and all design decisions fully documented. And let third party labs audit to confirm that the holdout questions are equal difficulty and similar task types to the public ones.
5 hours ago [-]
paimapi 4 hours ago [-]
1) open access is a rising trend in journal publishing. arxiv exists because of it but this also means that a lot of papers get 'published' before they are peer-reviewed which is often when significant issues are caught
2) there are definitely problems with journals and publishing but, like many similar absurdly reductive pronouncements, the argument that the current mode of scientific inquiry is bunk is both wrong and lacks nuance
the problem with modern publishing is that private equity is buying up publishers [0]. these publishers are then giving peer-reviewers no time and zero pay to do the necessary work of review [1] while also charging exorbitant rates for access. this is leading to worsening quality of the published research along with highly overburdened researchers who are stuck between shrinking funding [2] and their myriad other professional obligations
to just say that 'journals' are bunk is ignorant in a harmfully anti-empirical way. the process of empirical research and review is the entire reason why we see realworld results. foundations comprised of bullshit crumble fast but for some reason or another our economic system is highly driven to enshittifying everything it touches
to have defenders who claim AA is 'pushing the boundaries of the scientific method' sounds like the screeching refrain of anti-intellectual cargo cults, apeishly mimicking the features of rigor and methodology while avoiding any real accountability
Given how many private benchmarks they're using now, its likely they just tested different combos until they got the result they wanted.
Completely discredits the index if it just gets modified to match social media vibes.
kingstnap 18 hours ago [-]
In real life, there is always this feedback edge from the results to the methodology.
Theoretically it's unscientific to do tweaks like this but in reality this is what actual science is because you need to see the results understand the problems in your experimental designs.
Now the interesting thing is that you could repair the bias problem. The main issue is that you will do these tweaks after a bad result not after they look fine.
Maybe we could commit ahead of time that the experiment will be reanalyzed after results regardless of what they are. Instead of only when the results disprove the hypothesis.
So like artificial analysis committing to a fixed cadence of index updates instead of when the results start looking jank.
baq 13 hours ago [-]
> Astra is way better than Sol
This needs to be evaluated per task, jagged frontier yadda yadda. I would be not at all surprised if sol was better at some things than Astra just like people still use opus 4.6 and for good reasons.
andriy_koval 13 hours ago [-]
Maybe there is truth in it?
I asked Astra to make some changes, it built crazy overengineered code, switched back to Sol, said code is too complicated, rewrite it, and it made lot cleaner code.
I feel some models could chaise complicated benchmarks too much in expense of simpler tasks quality.
cbg0 13 hours ago [-]
Sol frequently over-engineers code.
big-chungus4 13 hours ago [-]
Well, do you have any ideas about how to make it scientific? And they clearly needed to do something based on their unnatural ranking
weird-eye-issue 15 hours ago [-]
It's a benchmark for brand new technology and you admit the old score was silly but you think updating it is unscientific?
villish 15 hours ago [-]
Hopefully this wakes people up from this addiction to benchmarks when discussing various AI models. Different model families have strengths and weaknesses in various domains, but those are never discussed.
weird-eye-issue 15 hours ago [-]
> but those are never discussed.
They are literally frequently discussed and it's why there are different benchmarks for different domains.
villish 9 hours ago [-]
> different benchmarks for different domains.
Labs know the only thing the public even discusses on model releases are benchmarks, so they devote a majority of training on just benchmaxxing. It’s marketing.
Muse Spark looks great in benchmarks. Everyone I know who has tried it (Rust & C++ projects) has determined it’s a resounding “meh”. That doesn’t mean it’s not a great tool for frontend devs, I wouldn’t know.
cma 7 hours ago [-]
> Astra is way better than Sol
For knowledge type questions is that necessarily true? In the past we've seen things like Google's models degrading on general knowledge after the preview releases while improving on code/tool use, presumably due to catastrophic forgetting from the additional training. Their preview would be free, get lots of agentic use from users, then additional training on that and probably additional automated RL.
However Astra is on an entirely new base model so I also wouldn't expect it to be worse.
throw10920 15 hours ago [-]
What evidence do you have that they tweaked the index to fit what they thought people expected?
throwaway13337 14 hours ago [-]
I have no idea how artificial analysis got to be something anyone took seriously. This is their new benchmark set?
A glance at their new index shows that whatever they're measuring, it isn't useful.
Spend an hour with gemini 3.8 and tell me that model belongs in 2026. It feels like the model has Alzheimer's. It gets confused about whether what it reads is what it did. Just crazy bad.
I haven't tried muse spark 1.3. But it must have been a miracle since 1.2 to hit that rank.
Video game journalism vibes all over this.
nojs 13 hours ago [-]
What other benchmarks do you recommend that are more accurate?
WASDx 7 hours ago [-]
Give a task you have to 3 different models and see what actually works for you. There are no good benchmarks.
Catloafdev 4 hours ago [-]
Because it's the best option currently available.
It's really easy to shit on AI benchmarks, but that noise is useless unless you're offering a solution or a better benchmark.
dist-epoch 8 hours ago [-]
As the saying goes, Artificial Analysis is the worst benchmarking company, except for all the others.
__natty__ 9 hours ago [-]
I wonder if Artificial Analysis could be influenced by certain model companies. Looking at the changelog [0], they updated a few times after new models appeared, so US models progression was much higher than that of other vendors (like when they updated the algorithm after Kimi K3 versus Opus 4.7, so Kimi dropped in the rankings). Or maybe thats just coincidence.
That would be the fastest way for them to completely torch their company.
The only thing they are selling and why people look at them is trust that they do honest evaluations.
jjcm 13 hours ago [-]
IMO one of the biggest losses of the OpenAI/Cursor breakup will be the loss of OAI models on CursorBench [1]. Their bench has always been one that most-fit my mental model of how good each of these models are. I find AA’s Intelligence index to often be out of alignment with my own subjective evals.
This is really a great achievement: "Astra dominates the output token frontier"
Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well.
Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin.
Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).
jsnell 17 hours ago [-]
The apparent advantage is exaggerated by them running Astra at six different effort levels, and almost everything else at just the maximum available effort.
I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one.
But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)
Scaevolus 18 hours ago [-]
GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens.
This also makes it much harder to monitor its reasoning.
ssivark 15 hours ago [-]
Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).
water-drummer 13 hours ago [-]
But the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.
ssivark 1 hours ago [-]
Which is why I said that cost might be a better metric than output tokens. But even that is somewhat misleading -- because there isn't really a single price -- there are a variety of offers / deals / subsidies, including subscription plans.
But if we're actually talking benchmarking intelligence -- as evidenced by "Artificial Analysis Intelligence Index" and the rest of the reporting -- then comparing systems with different amounts of recurrence on output tokens is flawed.
AnodicElegy 18 hours ago [-]
This update really gives OpenAI a boost. Not saying there's anything inaccurate or untoward about that, but the timing is unfortunate. It would have looked better had it been done prior to the Fable 5.1 and GPT 6 releases. I guess AA would say that there's no perfect time to do these updates, given the rapid fire pace of releases!
CuriouslyC 17 hours ago [-]
The timing is related to the fact that their benchmark was saying it was the same as Sol, and below Opus 5, when anecdotal reports and other benchmarks strongly disagree. It looked bad for them for their benchmark to disagree with people's lived experience so hard.
pixl97 35 minutes ago [-]
Kind of reminds me when GPU benchmarks used to game the drivers to maximize the FPS. If the benchmark slightly changes the camera view is that cheating? Or is it calling out the cheaters?
stared 10 hours ago [-]
While I like this index, calling in "Intelligence" might be confusing - it is a mix of coding and knowledge.
> Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.
dgacmu 8 hours ago [-]
I quite wish they'd move to terminal bench 4.0. 2.1 is saturated - there is no world in which Gemini 3.8 Flash is producing better code than Astra or Fable, as the 2.1 results might suggest. The 4.0 results differentiate these models much more effectively.
(2.1 is useful for knowing they can all one-shot straightforward scripts, of course.)
swingboy 7 hours ago [-]
DeepSWE has Gemini 3.8 Flash up really high, too.
dgacmu 6 hours ago [-]
It does and that one also feels kind of saturated for measuring the most advanced models - opus, Gemini, astra, sol, fable, glm, kimi all scoring within statistical noise of each other. (74 +-3% down to 69% +-5% for kimi).
It's still providing strong discrimination between weaker models.
sanxiyn 14 hours ago [-]
It is very unfortunate they upweighted SciCode from 8% to 10%. SciCode is a broken benchmark: see https://arxiv.org/abs/2608.04975.
aurareturn 14 hours ago [-]
In terms of intelligence per token per cost, OpenAI is really killing it.
nthypes 19 hours ago [-]
What version the intelligence vs cost graph is using? they didn't ran v4.2 to all models.
AnodicElegy 18 hours ago [-]
It's very different than it was earlier today, so pretty sure it's v4.2.
lousken 19 hours ago [-]
How to view the previous version to compare?
redox99 16 hours ago [-]
Images from their X account
theycallmeritik 15 hours ago [-]
how did you check the prev version to compare?
6thbit 19 hours ago [-]
Why have they not included ARC-Agi-3 on their index?
Clearly that would move things around.
thereitgoes456 14 hours ago [-]
Because playing a video game isn’t relevant to which AI people might want to use.
dannyw 9 hours ago [-]
"Video game" is the medium, the challenge and test is figuring out how to win; when given no instructions, and specifically designed to be private.
ARC-AGI takes it very seriously: they've never tested Fable, because they won't run on the eval set without ZDR.
Another possible way to look at this is that any benchmark reporting Fable scores is potentially contaminated.
https://artificialanalysis.ai/evaluations/omniscience
> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.
This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.
Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE
Astra is a big jump from sol and performs the same or slightly better than fable here.
IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.
If you customize the filter to add Gemini 3.1 Pro you'll see it ranks 5th in AA-Omniscince Index. Yet by default you wont see Gemini 3.1 Pro.
I find this to be very unhelpful and confusing.
It's not useful because the benchmarks often measure the wrong thing. They're here yapping about AGI and yet the benchmarks treat it like a trained dog. Fetch this. 100 points.
Each "problem" in these benchmarks likely has more than 1 solution that can be considered correct and even should be graded in many ways. Yet we see in many benchmarks higher effort (or thinking levels) don't help because the benchmark penalizes for doing "more" than what the answers asks for. So what did you ask for?
In human school you often get marks on the process and not just the end result. Thinking tokens have been cut. All we group on is things like cost, turns and time but not the what else.
The old index was clearly bad (Astra is way better than Sol) but it's also unscientific to tweak it like this.
like they have words that are dressed in scientific language on their site like
"We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2."
but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?
there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives
It's just high school statistics: https://en.wikipedia.org/wiki/Confidence_interval
It's not a probability. It's essentially an assertion that if the test were run 100 times, the result would be within the interval 95 times.
2) there are definitely problems with journals and publishing but, like many similar absurdly reductive pronouncements, the argument that the current mode of scientific inquiry is bunk is both wrong and lacks nuance
the problem with modern publishing is that private equity is buying up publishers [0]. these publishers are then giving peer-reviewers no time and zero pay to do the necessary work of review [1] while also charging exorbitant rates for access. this is leading to worsening quality of the published research along with highly overburdened researchers who are stuck between shrinking funding [2] and their myriad other professional obligations
to just say that 'journals' are bunk is ignorant in a harmfully anti-empirical way. the process of empirical research and review is the entire reason why we see realworld results. foundations comprised of bullshit crumble fast but for some reason or another our economic system is highly driven to enshittifying everything it touches
to have defenders who claim AA is 'pushing the boundaries of the scientific method' sounds like the screeching refrain of anti-intellectual cargo cults, apeishly mimicking the features of rigor and methodology while avoiding any real accountability
[0] https://issues.org/how-academic-science-gave-its-soul-to-the...
[1] https://www.insidehighered.com/news/faculty/books-publishing...
[2] https://www.nature.com/articles/d41586-025-00754-4
Completely discredits the index if it just gets modified to match social media vibes.
Theoretically it's unscientific to do tweaks like this but in reality this is what actual science is because you need to see the results understand the problems in your experimental designs.
Now the interesting thing is that you could repair the bias problem. The main issue is that you will do these tweaks after a bad result not after they look fine.
Maybe we could commit ahead of time that the experiment will be reanalyzed after results regardless of what they are. Instead of only when the results disprove the hypothesis.
So like artificial analysis committing to a fixed cadence of index updates instead of when the results start looking jank.
This needs to be evaluated per task, jagged frontier yadda yadda. I would be not at all surprised if sol was better at some things than Astra just like people still use opus 4.6 and for good reasons.
I asked Astra to make some changes, it built crazy overengineered code, switched back to Sol, said code is too complicated, rewrite it, and it made lot cleaner code.
I feel some models could chaise complicated benchmarks too much in expense of simpler tasks quality.
They are literally frequently discussed and it's why there are different benchmarks for different domains.
Labs know the only thing the public even discusses on model releases are benchmarks, so they devote a majority of training on just benchmaxxing. It’s marketing.
Muse Spark looks great in benchmarks. Everyone I know who has tried it (Rust & C++ projects) has determined it’s a resounding “meh”. That doesn’t mean it’s not a great tool for frontend devs, I wouldn’t know.
For knowledge type questions is that necessarily true? In the past we've seen things like Google's models degrading on general knowledge after the preview releases while improving on code/tool use, presumably due to catastrophic forgetting from the additional training. Their preview would be free, get lots of agentic use from users, then additional training on that and probably additional automated RL.
However Astra is on an entirely new base model so I also wouldn't expect it to be worse.
A glance at their new index shows that whatever they're measuring, it isn't useful.
Spend an hour with gemini 3.8 and tell me that model belongs in 2026. It feels like the model has Alzheimer's. It gets confused about whether what it reads is what it did. Just crazy bad.
I haven't tried muse spark 1.3. But it must have been a miracle since 1.2 to hit that rank.
Video game journalism vibes all over this.
It's really easy to shit on AI benchmarks, but that noise is useless unless you're offering a solution or a better benchmark.
[0] https://artificialanalysis.ai/changelog
The only thing they are selling and why people look at them is trust that they do honest evaluations.
[1] https://cursor.com/evals
Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well.
Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin.
Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).
I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one.
But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)
This also makes it much harder to monitor its reasoning.
But if we're actually talking benchmarking intelligence -- as evidenced by "Artificial Analysis Intelligence Index" and the rest of the reporting -- then comparing systems with different amounts of recurrence on output tokens is flawed.
Compare and contrast with ARC-AGI, BabaIsBench (https://quesma.com/benchmarks/babaisbench/), or MazeBench (https://mazebench.com/blog?post=introducing-mazebench).
In particular, in one Baba Is Bench post (https://quesma.com/blog/baba-is-aug-2026/), while quoting a Pareto frontier chart from AA, I noted:
> Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.
(2.1 is useful for knowing they can all one-shot straightforward scripts, of course.)
It's still providing strong discrimination between weaker models.
Clearly that would move things around.
ARC-AGI takes it very seriously: they've never tested Fable, because they won't run on the eval set without ZDR.
Another possible way to look at this is that any benchmark reporting Fable scores is potentially contaminated.