Large language models generate plausible language without deductive reasoning, so they should not be treated as zero-shot information stores.
2
Trustworthy LLM applications need guard rails, tests, human feedback, versioned data, evaluations, and monitoring across the whole system.
3
Teams should start with the human problem, then account for regulations, harms, dependencies, and data lineage before choosing an AI solution.
Summary
Noble Ackerson argues that large language models are unreliable information stores because they are probabilistic, non-deterministic, and unaware of what they do not know. Their fluent answers can still contain factual errors, and demonstrations that work in isolation often fail when users depend on them in production. He frames the answer as a data trust problem. Teams need to explain what a system can and cannot do, add guard rails and tests, involve human experts, version prompts and data, evaluate models online, and monitor both outputs and user feedback. Fine-tuning can help with private tasks, but it brings overfitting, cost, changing data, and ethical risks. Retrieval augmented generation can provide fresher information, though it adds more system complexity. Ackerson also argues that product teams should begin with a real human problem and use systems thinking before selecting AI technology.
Data trust determines whether users will accept an AI system
Ackerson defines data trust as the responsible delivery of a machine learning model, including transparency about what it can and cannot do, checks that protect end users, and acceptance of the consequences when something goes wrong. The goal is to calibrate user trust through reliable value delivery. He connects this to public reactions to nuclear power, GMOs, crypto, Google Glass, and self-driving cars. These technologies may have useful capabilities, but public confidence suffers when people see risks without enough checks or guard rails. For enterprise AI, he says teams cannot cut corners on trust because unreliable systems will not be used.
Language models predict plausible text without knowing what they know
Ackerson describes language models as autoregressive systems trained to predict the next coherent word. They are probabilistic and non-deterministic, and they do not reason deductively without additional help or access to other systems. He says they are also epistemologically blind to what they do not know, so the same prompt can produce unpredictable answers. This makes a model dangerous as a zero-shot information store, especially inside an enterprise. The model can sound confident while producing a factual error. Ackerson's concern grew when he saw search products preparing to add generative AI features, because users might rely on those answers without understanding the limits of the underlying system.
Production failures appear when a convincing demo meets real users
Ackerson describes moving from ChatGPT-style experiments to API-based systems and then trying Google's PaLM MakerSuite playground. Adding complexity did not solve the production problem. Once users interacted with the applications, they complained that the systems felt like a PA system or a wall, because the answers were unreliable and inconsistent. He says demos can work well while the deployed product falls short. His customers needed factual correctness, repeatability, consistency, and reliability before they would use the tools. This led his team to add prompt versioning, better prompt practices, guard rails, tests, human experts in the loop, and human feedback that could improve the system over time.
Prompt design can use rules about truth, relevance, information, and clarity
Ackerson applies Paul Grice's Cooperative Principle to product outputs and prompts. He names four maxims: quality, which concerns truthfulness; quantity, which concerns being informative; relevance; and manner, which concerns clarity. He recommends using these as practical guidance for in-context learning and prompt design. A prompt should relate directly to the task instead of asking for a broad response such as an explanation of AI or a biography. He gives the contrast between a vague request and one that asks for the key principles of machine learning. The point is to make the intended task, required information, and expected communication style clearer.
Fine-tuning private data adds cost and can create overfitting
Ackerson says teams often try to make a model consistent by teaching it private data and new tasks. This can lead to overfitting, where the model fits the training data but does not generalize well to validation data. Foundation-model probabilities and knowledge gaps also carry into the resulting application. The system requires more than an initial training expense. Enterprise data changes frequently, so continued training creates an ongoing operational cost. Ethical risks also remain after fine-tuning. A model that has been adapted to private data still needs guard rails, testing, and monitoring. Ackerson also notes that a model eventually stops training, which leaves a freshness problem for new information.
A production serving framework needs versioned data and feedback loops
Ackerson presents a serving framework that prepares domain and reliable-information sources, versions the data, and feeds information into the system. It can include feedback data for a model-improvement loop and telemetry for red and purple teams. Depending on the situation, teams may fine-tune a model or use distributed transfer learning across multiple models. The framework includes online evaluations, performance tests, model selection, and post-processing that can persist chat history for longer conversations. The application receives the resulting model through a streaming path, while feedback returns to the batch pipeline. His point is that the work needed before serving is substantial and must be treated as part of the product.
Retrieval augmented generation keeps changing information outside the model
Ackerson introduces retrieval augmented generation as a response to the model's training cutoff and changing enterprise data. In his description, a batch pipeline prepares information and a streaming context uses fresher material, with feedback returning to the batch process. RAG does not remove the wider reliability problem. It adds another set of components and possible failure points, including the knowledge store and the code connecting it to the model. Ackerson says teams should monitor the output from the knowledge store as well as the model itself. The choice of retrieval or fine-tuning depends on the data, task, and operating context rather than on a single universal recipe.
Trust has to be calibrated between rejection and overreliance
Ackerson describes two bad outcomes. If a system is unreliable and users see factual errors, they may stop using it. If the system works too well without enough context or warnings, users may trust it beyond what it can support. He gives the example of a multimodal system that scans a plant and says whether it is safe to touch or eat. Such a tool should tell users when to double-check, with the required level of caution depending on the application's criticality. He calls calibrated trust difficult and says teams need to log and preserve user feedback, communicate system limits, and design the experience so people understand when the output needs verification.
AI product work should begin with the problem and its surrounding systems
In response to a question about AI product ideation, Ackerson says the model is not the product. Teams should start with the human problem and work back toward a solution. He recommends setting technology aside briefly and considering regulations, harms, socioeconomic issues, other products, and dependencies through systems thinking. From that framing, teams can establish principles for data inputs, data cards, model cards, lineage, and provenance. They should build on established machine learning operations standards rather than creating every practice from scratch. Product outputs also need protection against bias and clear explanations of what the system can and cannot do. These practices help earn the trust required for adoption.
"They are non-deterministic by nature, they're probabilistic by nature, they do not reason deductively natively without additional help or access to other things or other language models."08:27
Who should watch
You are building an LLM application and need a practical account of why a promising demo can fail with real users.
Your team is deciding between fine-tuning, retrieval augmented generation, or a simpler system and needs to weigh freshness, cost, and operating complexity.
You own product trust, evaluation, or user feedback for an AI system that must explain its limits.
