Aniket Singh evaluates whether LLMs can perform tasks that humans handle, rather than relying only on benchmarks or system-level tests.
2
Confidence scores become more useful when models rate themselves on a 0-to-1 scale and receive feedback about whether their answers were correct.
3
Reliable production systems may need narrow workflows with multiple agents, separate validation steps, and models chosen for specific tasks.
Summary
Aniket Singh explains an approach to evaluating LLM capability through tasks that resemble human decision-making. His research includes simulated auction bidding, where models receive personalities and budgets, and coding assessments where hints cost part of an undisclosed budget. He compares absolute and relative confidence scores, finding that a 0-to-1 scale produces more useful variation than a 1-to-10 scale. Feedback can cause models to lower their confidence after incorrect answers, although models differ widely in how confidently they behave. The conversation then moves from research experiments to production. Aniket argues that a capable model is not enough for document processing or structured extraction. A workflow can divide the task among agents, add cross-checking, and use a final model trained or fine-tuned for the required output. He is testing how to make these workflows more reliable, cheaper, and less tied to a single model.
Aniket Singh studies what LLMs can do instead of repeating benchmark tests
Aniket Singh became interested in evaluating whether LLMs could replace parts of human decision-making. He avoided standard benchmarking and system-level evaluation because many researchers were already working there, while early LLM applications were changing too quickly. His experiments ask whether a model can perform a task when given a role, a backstory, and a situation that requires judgment. The auction example tests whether models can bid within a budget. This makes the evaluation about behavior and capability, rather than only the score on a fixed dataset.
Assigned personalities change how models behave in an auction simulation
For the auction experiment, Aniket's team gave different LLMs personalities and backstories. One model was described as an accountant with years of experience. Another, named Olivia, was an extrovert who acted spontaneously. The models behaved in ways that matched the characteristics they were given. Olivia went over budget repeatedly because she was less careful about spending. GPT-3.5 was weak at math and reasoning in the initial tests, while GPT-4 performed better. Aniket also improved GPT-3.5's results by using chain-of-thought prompting.
Stealth assessment tests budget management without telling the model
Aniket describes a second experiment called stealth assessment. The models solve LeetCode problems and can request hints or feedback, but each hint has a cost. The model knows that a budget exists, while it does not know that the researchers are testing its budget management. The design also uses the confidence attached to each answer. The experiment grew out of earlier work on the Dunning-Kruger effect, where Aniket's team examined whether models trained on human text show a similar relationship between competence and confidence.
A 0-to-1 confidence scale gives more variation than a 1-to-10 scale
Aniket's team asked models to report both absolute confidence in their own answer and relative confidence compared with other models. A 1-to-10 scale produced scores clustered around 9 or 9.5, especially from GPT-4 and GPT-3.5. After a professor suggested changing the scale, the team used 0 to 1 and saw more variation, including scores such as 0.98, 0.92, and 0.77. When researchers told a model that a previous answer was wrong, GPT-4 and Claude often lowered their confidence on the next question. The models also appeared to react when questions became harder, although the researchers did not label the difficulty levels.
Model confidence is a behavioral difference, not a reliable sign of correctness
Aniket says LLMs generally have no shortage of confidence, but their confidence does not always match the quality of their answers. Claude was sometimes hesitant even when its answers were correct. GPT-4 was much more assertive and appeared aware that people regard it as one of the best models. The experiments therefore treat confidence as something to measure and compare, rather than assuming that a confident answer is accurate. The team also used data containing misinformation to make the tasks harder and to test whether confidence changed when models encountered difficult or misleading material.
Production reliability depends on the workflow around a model
Aniket's research is moving toward applications because his startup has encountered practical problems with LLMs in production. For document extraction, one model may skim the input and produce incomplete or incorrectly formatted JSON. A multi-agent workflow can divide the document into parts, ask separate models to extract information, use another model to cross-check the results, and send the final output through a model trained or fine-tuned for the required format. This can cost more, but Aniket argues that the extra calls may be acceptable as models become cheaper.
Closed workflows are easier to control than agents that invent their own plans
Aniket is cautious about systems such as AutoGPT that prompt themselves and create plans dynamically. That trial-and-error behavior may work for some tasks, but it can be too expensive for a production application. His preferred direction is to fine-tune the creation of workflows for a particular use case, rather than asking a model to invent a new set of prompts every time. He agrees that pointed tasks, clear guardrails, and a closed system are easier to control. The team is still testing which multi-agent architectures work for complicated document-processing tasks.
A smaller model can work when the pipeline gives it a narrow job
Aniket does not expect GPT-4 to be necessary for every step of a production pipeline. A document workflow might use OpenCV for preprocessing, a TensorFlow model for extraction, and an LLM only for the part that needs language reasoning. The workflow can then be tuned to extract the specific fields the application needs, rather than asking a highly capable model to handle the entire document. This approach may reduce cost and make the system less dependent on one model, but every candidate model still needs to be tested because their capabilities differ.
"Even if it does do all this stuff that we do as humans, or if it is as aware as we are, and it has confidence scores, it knows it's going to get something wrong or right, that all doesn't really matter if at the end of the day you can't use it in production."Demetrios Brinkmann34:34
Who should watch
You are comparing LLMs and want tests that measure behavior on tasks rather than relying only on benchmark scores.
Your application uses LLMs for document extraction, structured JSON, or other work where occasional failures are expensive.
You are designing a multi-agent workflow and need to think through model choice, validation, guardrails, and call costs.
