The best LLM depends on the task and the metrics that matter for that task, rather than on a general benchmark ranking.
2
For code summarization, Shai Rubin compared models on cost, conciseness, latency, and similarity between their outputs.
3
When models produce similar summaries, a cheaper or faster model may be a better production choice than the most expensive model.
Summary
Shai Rubin presents a practical way to choose an LLM for code understanding. He starts by defining the task and the measures that matter: concise answers, low latency for human-in-the-loop use, accuracy, and cost. He then compares models by running a shared prompt across files from the PyTorch repository. The analysis covers token cost, answer length, response time, and the similarity of model outputs. Rubin argues that accuracy is difficult to judge when several models produce answers with the same meaning, so comparing output similarity can help identify models that behave similarly. His examples show large differences in price and latency, while many models generate broadly similar summaries. He recommends filtering models by practical constraints, then comparing their outputs and choosing the option with the best return for the task. He also says the experiment code, charts, and results are available on GitHub.
Rubin rejects the general question of which model is best. The useful question is which model is best for a specific goal. His example is code understanding, which he defines as interpreting the structure, behavior, and intent of source code so someone can reason about what it does and why. He compares answers from ChatGPT, Gemini, and Claude and says they are very similar to a human reader. That similarity makes model choice a practical question about the value gained from paying more for a different model.
For code summaries, Rubin wants answers that are concise because a long answer reduces the value of asking for a summary. He also cares about latency because his use case includes a human in the loop. Accuracy matters, although he says it is difficult to define when several answers express the same idea. Cost matters as well, since the goal is to reduce spending without giving up the needed quality. Once these measures are defined, he says they can be used in an experiment across models.
A shared prompt and a fixed file set make comparisons possible
Rubin runs models on files selected from the PyTorch repository using one common prompt. The prompt asks for a summary of up to three sentences that captures what the Python file does. Each model processes the same files, which gives him outputs and run metadata that can be compared. He shows one example containing the model's summary and information about the file, prompt, and run. The design is intended to compare models on the same task rather than rely on separate benchmark scores.
Price and answer length vary sharply between models
The cost chart shows a wide spread. Rubin says the most expensive model is 377 times more expensive than the cheapest model in his experiment. He also finds that most models produce answers of about 60 words, while two models ignore the length instruction and average 132 and 164 words. His practical rule is that a model must follow the prompt. A model that routinely produces much longer answers is harder to use in software, even when its content appears reasonable.
Latency determines whether a model fits interactive use
Rubin divides models into latency groups for code summarization. Models below roughly three seconds are suitable for interactive use in his scenario. Models between three and six seconds may work when the audience is patient. Models above six seconds are better suited to batch processing rather than an interactive human-in-the-loop workflow. Combining latency with cost narrows the field to models that are affordable and responsive enough to be useful.
Output similarity helps when accuracy is hard to score
Rubin introduces differential accuracy for cases where several models give answers with nearly the same meaning. If outputs are similar, he argues that the absolute accuracy difference may not matter for choosing among them. He measures similarity by converting text into embeddings and applying a similarity function. As a baseline, he compares outputs for different files and expects low similarity when the files have different functions. The resulting matrix is mostly near zero, which supports the method's ability to distinguish unrelated outputs.
Across the files Rubin studies, outputs for the same file are generally similar, with reported similarities ranging from 0.72 to 0.90 in the displayed samples. He identifies Nova Pro V1 as an unusual result because its outputs are less similar to those from the other models. In a comparison of the GPT family, GPT-4o is close to the other models, while Nova Pro V1 is also close to GPT-4o at about half the stated cost. He presents this as a reason to compare outputs before defaulting to the most expensive model.
The workflow should end with a task-specific ROI decision
Rubin's recommendation is to define the goal, define the metrics, and create a workflow for deciding which measures matter more. A model can be rejected because it is too slow, too expensive, too verbose, or insufficiently aligned with the desired output. The final choice should reflect the return available for the particular task. Rubin has published the experiment code, chart-generation code, and results on GitHub so others can repeat the process for different goals.
"Define your goals, define your metrics, and define a workflow that helps you identify what is important to you, what is less important to you, and what is the best ROI that you can get from the various model."20:32
Who should watch
You are choosing an LLM for a coding assistant or another tool that needs to understand source code, and you need a comparison tied to your own constraints.
Your current evaluation focuses on general benchmarks, but production use depends on response length, latency, cost, and how similar the outputs are.
You want an example of a repeatable model comparison and access to the experiment code and results on GitHub.
