A Tail of Two Paretos

When nearly every model solves nearly every question on a benchmark, we have a word for it.

We call it saturation. The benchmark is declared solved, a harder one is commissioned, and the old one goes on the shelf. When every model fails the same question, there is no corresponding headline. The question stays in the benchmark. We pay to run it again. Its zero is folded into each model’s average, where it becomes almost impossible to see.

A leaderboard tells us how many questions each model solved. It does not tell us which questions defeated every model we tried, or which questions every model succeeded on.

We started with a question…

Are AI benchmarks assuming a Gaussian distribution of outcomes?

First, two Paretos

Most AI engineers meet Vilfredo Pareto through the “Pareto frontier”. Plot quality against cost, speed, or memory, remove the options that are worse on every axis, and inspect what remains. It is a useful way to compare systems when no single system wins on everything.

Pareto’s earlier work began with a different problem. He looked at income records from several countries, plotted the observations, and tried to determine their shape. The strong concentration he found would later be popularized as the 80/20 rule: a small share of people accounting for a large share of wealth.

The first figure puts those two contributions side by side. On the left, select a model to see whether another model dominates it, then lower the cost limit to find the best affordable choice in a Pareto frontier. On the right, move the population probe and inspect how much of the total belongs to the top and bottom shares in a Pareto distribution.

The panels ask different questions. The Pareto distribution tail in A Tail of Two Paretos is not meant to be taken literally for benchmark outcomes, which are bounded between zero and one.

The essential idea we use from Pareto is that we cannot naively assume we know the distribution. We need to be ready for the possibility that the arithmetic mean can fool us. The connection to the Pareto distribution is not about Pareto’s specific result. The connection is to his process of figuring out the shape of the observations.

Pareto was on a hunt to understand the underlying structure. So are we.

Next, look at the observations

Start with the simplest common case: each model either passes or fails each question. For each question, look across a pool of different models and record the share that succeeded. Some questions may be solved by every model. Some may be solved by none. The rest fall somewhere between those two ends.

There may be hidden structure in those observations that helps us understand the distribution of results. In Pareto’s case, the hidden structure was the concentration often summarized by the 80/20 rule. In our case, it includes point masses at the outcome ends: a “failure floor” where none of the models we observed succeed, and a “success ceiling” where all of them do.

These endpoints are relative to the pool we observed. Add another model and a question can leave the floor. Change the questions or the scoring rule and the shape can move again. A point mass at zero also says nothing by itself about the cause. The questions might be beyond the current pool, broken, ambiguous, or scored incorrectly.

We have more information about the outcomes once we can see this structure. Even though it is very different from an unbounded power-law distribution, the lesson is the same: the arithmetic mean is a very lossy compression of the results, and it can mask their underlying shape. If we look only at the average outcome while implicitly picturing a roughly Gaussian shape around it, we can draw conclusions that are wildly inaccurate, including the conclusion that two evaluations with the same averages have mostly the same structure.

The same averages hide two wildly different outcomes

The next figure makes that information loss visible. It contains constructed results for the same eight models and the same number of questions. Every model has exactly the same score in both panels. The leaderboard is unchanged. The leaderboard would declare them to be “equal”.

Start with the full maps. Move across the questions and compare how many models pass each one. One evaluation has no question that defeats every model. The other has eight. The number of questions every model solves changes too.

Now select Averages only. The question columns disappear, and the two evaluations become indistinguishable. Switch back to the full maps and the endpoint structure returns.

The mean answers how many questions each model solved, with every question counted equally. It cannot tell us where those successes overlap. It cannot distinguish a benchmark with a large shared-failure region from one with none at all.

That distinction can matter even when every number on the leaderboard is correct. It also raises the next question: what happens when a mean built from those outcomes becomes an axis on a Pareto frontier?

Back to the frontier

The first figure introduced two uses of Pareto’s name. We can now make the connection between them explicit.

A Pareto frontier compares the coordinates it is given. In AI evaluations, one of those coordinates is often a mean benchmark score. The frontier inherits every decision used to construct that mean, including the decision to count each question equally.

The final figure starts with a separate constructed matrix of 256 pass and fail outcomes. Within this figure, the cells do not change. Only the rule for how the questions count changes.

Begin with equal question weight, the familiar benchmark average. Then try the other three rules. One gives equal weight to different regions of the observed shape. Another emphasizes questions that separate the models. The last represents a made-up target population of common questions from our application’s users. Watch the model scores and the frontier as you move between them.

The outcome cells never move. Each weighting rule asks a different question of those cells, producing different scores and, most importantly, a different frontier.

What any leaderboard can already tell you

Even without the question-by-question results, the scores can tell you if a failure floor or success ceiling exists.

To look for a failure floor, add the model scores. Suppose three models each score 20%. Their scores add up to 60%, so at least 40% of the questions must have been missed by all three.

To look for a success ceiling, first subtract each score from 100% to find the percentage each model missed. Suppose three models each score 80%. Each one missed 20%, so their misses add up to 60%. At least 40% of the questions must have been solved by all three.

If the scores add up to 100% or more and the misses also add up to 100% or more, the leaderboard cannot guarantee that either endpoint exists. They may still be there, but we would need the question-by-question results to see them.

Towards Telltail

We began by asking whether benchmark statistics were making Gaussian assumptions. Following that question took us back to the observations themselves. The first thing we learned was how much structure a correct average could erase.

Then a simpler question appeared…

What shape did the outcomes have before we averaged it away?

Telltail - Outcome Geometry for AI Evaluations

Coming soon to a package manager near you.