Exploring LLM performance on the ARC training dataset

TLDR: I tagged and described 200/400 ARC training dataset tasks, merged it with evaluation data for some LLMs, did some analysis on it, and put it all on a site so you can explore it.

ARC is a hard problem, and exploring the dataset first felt like the obvious starting point. The site originally started as a way for me to just explore and take notes on the problems, but that turned into completing a taxonomy of tags and descriptions and a few other things I got carried away with.

You can explore the tagged problems here yourself, download the raw JSON file, or read on if you want some more details on the dataset, how I went about this and some basic analysis on it.

If you are interested in helping to complete the tagging and descriptions for the remaining 200 problems, please reach out to me on twitter, or you can contibute directly to it here.

Building the ARC explorer #

Understanding LLM performance #

My main objective with this project was to understand where LLMs perform well [0] and where they perform poorly. Thankfully there are notebooks for GPT-4, Claude Sonnet, as well as some of the other public leaderboard submissions such as Icecuber 2020.

Sadly I don't have O1 access, so if anyone wants to help produce a submission.json file for O1, I would be very grateful, and bonus points if you can drop it in with the other submissions here.

Computable properties #

There are a number of computable properties that I could programmatically generate such as:

• The grid size - I broke this down into small, medium, large based on the average number of input sample cells, with boundaries of 64 and 256.
• Whether they have consistent input sizes, output sizes, or whether the input size always matches the output.
• Other properties are also calculated, like the number of distinct colors used, if the input colors match the output colors, etc. These are all in the raw JSON file, but not all of them are in the site itself.

Solution descriptions #

Ultimately there were only limited things I could easily compute, and I couldn't find a way to avoid [1] doing descriptions and tags manually.

The descriptions of the problems are trying to be a human like solution to the problem. This is how my brain worked, and not necessarily how a computer might solve them.

Coming up with ways to actually describe the solution in a concrete way was much harder than I had imagined, these are not things that one usually uses words to describe, and I suspect that many of my descriptions will be completely incomprehensible to someone who isn't me.

The complexity of the solution procedure varies quite significantly across problems, even on the training set which is supposed to be easier than evaluation or test, some of the problems require a very long sequence of steps to solve, here's an example description:

Identify the several disconnected, same colored objects that have cell values only on the edge of the outline of a perfect square (which may also be 1x1) and are symmetrical in both axes and may extend beyond the edge of the input grid. The output grid has the size of the largest such outline, and its background color is the same as the background (most common) color of the input grid. Each of the identified outline shapes is copied into the output grid and centered in the middle.

Or sometimes, they are much simpler:

Draw a red outline around any horizontally or vertically connected 2 green cells.

Tagging #

Whilst there are some problem types that recur several times, some of which are almost identical, that was usually not the case. There are still problems which I don't know how to tag properly.

I ended up with 37 tags, which evolved considerably as I went through. This took a few second passes, and then some grouping.

The full list I eventually come up with is below, within groups and with a short description of each both are certainly imperfect. I drew inspiration from David Bonin's Task Tagging Notebook on Kaggle, but ended up with a slightly smaller and more focused set.

Colors #

The colors used in the images (and therefore also in some of the descriptions) are not the same as those on the ARC website. I used a different color palette for no particular reason, but now that the descriptions reference them, here is the full mapping:

Initial analysis #

There are probably limited, non-obvious insights to draw out here from the questions I've asked so far. Bigger grids, grids with more tags, and longer descriptions are all harder for the models to solve, unsurprisingly.

The tags have the potential to be an interesting signal, although I believe this would greatly benefit from some additional data and tagging the remaining 200 problems to reduce the uncertainty a little bit.

Overall LLM performance #

The performance of the 3 evaluated models here is much better than their scores on the public eval leaderboard, and that isn't a surprise as the training set is acknowledged to be much easier than the evaluation set. Otherwise the relative performance is in-line, which is a reasonable assurance that nothing went horribly wrong here.

Grid sizes #

For all 3 models, the bigger the grid, the worse the performance. gpt-4o seems particarly affected by the input grid size, with a 7.937% accuracy on large (> 256 cell) grids, compared to 39.683% on small (< 64 cell) grids. On the small grids, gpt-4o is not far off sonnet at all. icecuber-2020 is the least affected by grid size, but there is still an effect.

It's unlikely it's the grid size itself that is the problem, but rather the fact that more complex problems inevitably have to be put on larger grids, so I'm intepreting this as a proxy for problem complexity.

Tag accuracy #

The tags are quite sparse, and the margin of error on these numbers is pretty high so take it with a pinch of salt. I've only included tags with at least 5 problems. The accuracy numbers are based on whether either gpt-4o or sonnet correctly solved the problem, which has a baseline accuracy across all problems of 32.75%.

I won't draw any conclusions from this. As you would expect (and I hoped), some of these tags seem to impact performance quite a lot. But, it's hard to tease out what is that tag itself versus some other correlations with the presence of that tag.

Tag counts #

More tags generally means there is a bit more going on in the problem, requiring multiple concepts or processes to be composed in some way. This seems to turn out to be a pretty good signal of overall accuracy with the accuracy generally trending much lower for those problems with more tags, while the single tag problems are solved at a rate of 54.167% which is much higher than the baseline.

Description length #

Short descriptions are solved 52.308% of the time, compared to 21.739% for long descriptions. Interestingly, medium or long descriptions don't have much of a difference, which perhaps says something about the quality of my descriptions.

Grid properties #

Well it turns out these don't have much variance in between them at all, but as they are on the site, I'm including them for completeness.

Next steps #

The obvious question to ask is - what happens when you feed these descriptions and tags into an LLM as additional context for solving the problems. I'm not hopeful that they will change much, but it's not too hard to ask the question.

I'd like to complete the tagging process, and extend to the evaluation dataset once ARC 2025 is released when I find the time, although I'd estimate this is around 10 hours or fairly gruelling effort.

There are many more questions I'd like to answer from the data. E.g looking at compressibility of the grids, or the conditional compressibility (e.g len(zip(input, output)) / len(zip(input)) or something like that). Hopefully others would be interested in taking their own analysis further too.

Getting a few more models in here would be nice, again with the obvious caveats of it being a public dataset.

And finally, as others have tried, seeing how models fare at trying to predict the tags themselves.

And of course, trying to take a shot at ARC itself...

reply via email follow me on twitter

Notes

• [0]: There are obvious problems with scoring these models against the training (or any public dataset).
• [1]: I did initially explore a suggestion from the community to try to use an LLM to describe solutions based off of re-arc verifiers, but this wasn't very successful, and after looking at the code for the verifiers, it was not surprising that it didn't work.