Arc AGI measures progress with problems that capable humans can solve while AI systems still struggle to solve them.
2
More inference-time compute improves model performance, but the cost and delay can grow sharply, and it is not clear whether this path reaches 100% accuracy.
3
Arc Prize is developing a third benchmark based on novel games that require models to learn rules, remember hidden states, and solve unfamiliar environments efficiently.
Summary
Greg Kamradt explains why Arc Prize uses benchmarks built around tasks that are easy for capable humans but difficult for AI. Arc AGI 1 and 2 use visual input-output grids, where each task requires learning a new transformation. The benchmark uses public examples and hidden holdout data to reduce overfitting. Greg describes how OpenAI's unreleased reasoning model improved sharply on Arc AGI when given more inference-time compute, while also costing far more and taking much longer. He connects the benchmark to François Chollet's definition of intelligence as the efficiency of learning new skills. The discussion then moves to evaluation, where physics, economic outcomes, and human preference provide different kinds of feedback. Greg is clear that no single benchmark can measure every capability. Arc AGI 3 will move toward unfamiliar games and simulations, testing whether systems can discover rules and solve many novel environments rather than mastering a fixed public set.
Arc Prize looks for tasks that humans can solve while AI systems cannot
Greg says Arc AGI was created to find problems that are easy for humans but hard for AI. Existing benchmarks often keep increasing the difficulty of academic questions, yet models continue to solve them. Arc Prize instead uses the human brain as its only current example of general intelligence. Its working definition is simple: when researchers can no longer find problems that humans can do but AI cannot, they may have reached AGI. Arc AGI 2 was tested with 400 people, and every task was solved by at least two people in fewer than two attempts. Greg carefully limits the claim to capable humans rather than the average person.
Hidden holdout data tests whether models have learned the benchmark
Arc AGI includes public data that teams can study and a hidden holdout set used for evaluation. Greg says this is needed to check whether a model has overfit the visible tasks. He describes working with OpenAI to test an unreleased model on the hidden set. The model scored 75% with low inference compute and about 87% with high compute, far above the publicly available models and purpose-built systems being tested at the time. The later production model used less compute and did not match the earlier result, which shows why the exact model and evaluation setup matter.
Inference-time compute improves performance at a steep cost
Greg says the OpenAI preview model used about $20 per task in its low-compute test across roughly 500 tasks. The high-compute version cost thousands of dollars per task and used about 170 times as much compute. It also took much longer, so the job had to run overnight rather than return an answer immediately. The result confirmed that spending more at inference can improve performance. Greg does not know whether this scaling eventually reaches 100% or approaches a limit. He expects the cost to rise sharply for each additional improvement, which makes the economic tradeoff part of the benchmark story.
Intelligence can be measured by the efficiency of learning new skills
Greg draws on François Chollet's 2019 paper, "On the Measure of Intelligence." He describes intelligence as the ability to learn a new task in a new domain, with efficiency measured by the energy and training data required. A system that needs far more examples or computation to learn a skill is less efficient. Greg contrasts humans with language models, saying humans do not need internet-scale training data in their heads to produce useful language. He also points to the energy used by human brains compared with the energy required by models to solve Arc tasks.
The benchmark tasks require abstraction rather than fixed knowledge
Arc tasks show an input grid, an output grid, and examples of the transformation between them. The model must infer the rule and apply it to a new test input. Each task uses a different rule, such as adding a border around a square or filling the corners of shapes. A system cannot rely on one memorized procedure across the whole set. Greg says these operations look simple to people because humans are good at abstraction and reasoning, but they have remained difficult for AI. Arc AGI 2 makes the transformations deeper by combining rules, such as adding a border and handling the corners.
Evaluation depends on what counts as a correct answer
Greg separates verifiable tasks from evaluations based on human judgment. Arc tasks can use an equality check because there is a definite grid answer. Summaries, meeting notes, and actions taken in a business system do not have one universally correct output. Greg's preferred evaluation order starts with physics, where reality determines whether a discovery is correct, followed by capitalism, where a business operates within a set of rules and outcomes provide feedback. Human preference comes later because it is difficult to quantify at scale and requires many people to compare outputs, as in reinforcement learning from human feedback.
A benchmark portfolio is more useful than one permanent test
Greg says there is no single benchmark that can measure every model capability. Benchmarks can provide useful signal for a period and then lose value once systems become too good at them. He compares this with ImageNet, which had a major effect on computer vision but is no longer the main way models are reported. Arc AGI 1 still provides useful signal across a middle range, but scores near the extremes are less informative. Arc AGI 2 is intended to measure harder tasks, although Greg says current models are still scoring too low for it to provide much signal.
Arc AGI 3 will move beyond fixed input-output grids toward simulations and games. Greg says the benchmark will require models to work out the goal, discover the rules of an environment, remember information that remains relevant later, and solve 50 games they have never seen before. The games will be novel relative to one another, which limits the value of developer-specific knowledge. Arc Prize plans to test 400 humans on the same games and compare the actions required by people and AI. Efficiency will include actions, cost, and energy, so brute-forcing a solution will not be enough.
