Large language models are general-purpose pretrained models whose usefulness comes from transfer learning, rather than from a strict parameter-count definition of "large."
2
Production systems usually begin with API calls and simple applications, then add data integrations, chained model calls, fine-tuning, and more engineering work as their complexity increases.
3
Teams have to choose among quality, cost, latency, privacy, and reliability trade-offs, with human review, retrieval, smaller models, and domain-specific systems helping make applications usable.
Summary
The panel discusses what it means to use large language models in production and where the difficult work begins. Rebecca Qian describes LLMs as a continuation of transfer learning, with one pretrained model applied to many downstream tasks. The group then compares API-based systems with self-hosted open-source models. Hannes Hapke explains why Digits chose to keep sensitive financial data in-house and added filters, hallucination checks, and accountant review. David Hershey describes a progression from simple text-completion products to systems that connect databases and services, then to chained agents and automated workflows. James Richards frames production decisions around a quality, cost, and latency triangle. The panel also discusses model compression, quantization, distillation, specialized models, and CPU inference. It is candid about the limits of current systems: chained calls can be too slow, model outputs can be factually wrong, and low-affordance use cases need stronger safeguards.
An LLM is defined by transfer learning more than by a parameter threshold
Rebecca Qian says the term large language model is poorly defined and describes it as a newer form of transfer learning. The field moved from smaller statistical models in the 1990s to deep learning, then toward reusable pretrained models. Instead of training separate models for sentiment classification, question answering, and ad ranking, one pretrained model can be adapted to many downstream tasks. She points to the progression from BERT to RoBERTa, T5, GPT-2, and GPT-3 as evidence that scale and data changed what these models could do. The panel rejects a simple billion-parameter cutoff. The Chinchilla paper suggests that parameter count must be considered alongside the amount of training data.
Digits uses model controls and human review around generated text
Hannes Hapke describes several uses of language models at Digits, including generating text and creating custom embeddings. The team does not put generated text directly in front of customers. It runs output through a strict not-safe-for-work filter, checks for known hallucination patterns, and discards samples that match them. An accountant reviews the output before it is sent to a client. This human step matters because accounting decisions can be subjective, even when the system is helping business owners understand their books in real time. Hapke says smaller, cleaner datasets can be enough to adapt an existing model to a specific use case after the initial pretrained model is available.
Self-hosting can protect sensitive data and create room for optimization
Hannes Hapke explains why Digits investigated models that could run inside its own infrastructure. The team wanted to avoid sending financial data to an external API and wanted control over how customer data was handled. A self-hosted model also reduces network distance and allows the team to quantize, prune, and adapt the model for its domain. The trade-off is substantial engineering work. Open-source models may require pretraining or fine-tuning, and replacing an API model means dealing with model-specific details. Hapke argues that this work can pay off when privacy requirements, latency, or usage volume make an external API a poor fit.
Production complexity grows from API wrappers to chained and automated systems
David Hershey describes production use cases as a progression. The simplest products call an API to complete text or generate sales copy, with very little infrastructure. The next level adds external information, databases, Slack, Jira, Zendesk, documents, and user history. More complex systems chain model calls, parse responses, and use one result to decide what to call next. These workflows are difficult to trust and manage. Full automation is harder still because it removes the human who validates the output. Hershey says many companies are software teams building at this middle level, while fine-tuning, reinforcement learning from human feedback, and model ownership remain a major barrier for teams without machine learning experience.
Latency is an end-to-end workflow property, not just a model statistic
James Richards distinguishes latency for an individual model call from the latency a user experiences across an entire workflow. A system with seven model calls, including fine-tuned models and embedding models, accumulates delay at every step. He describes a quality, cost, and latency triangle, where the right trade-off depends on the product. Code completion is an example where latency determines whether a user can remain in a flow state. The panel also notes that database access, input-output operations, and other application overhead add to model latency. A workflow that initially took 15 seconds was reduced to about three and a half seconds through model and system changes, but that was still unsuitable for some applications.
Architecture changes and smaller models may be needed for real-time applications
Rebecca Qian attributes much of the current inference delay to the Transformer architecture and the size of modern models. Transformers handle long-range dependencies well and are efficient during training, but their attention calculations make inference expensive. Recurrent models are sequential, yet can be faster for some inference workloads. Qian says removing the bottleneck may require a new architecture rather than incremental optimization. She mentions state space models as an area of academic research. The panel also discusses quantization, pruning, distillation, and routing from a general model into a specialized one. Hannes Hapke says domain-specific reduced models can produce large latency improvements and may allow CPU inference.
Reliability depends on the consequences of a wrong answer
Rebecca Qian defines hallucinations as outputs that are not factually accurate. The problem existed before the current wave of LLM applications, but it matters more when generated text is used in real business processes. The panel distinguishes use cases where an incorrect result is acceptable from those where it could cause serious harm. James Richards says an application that generates an outfit can tolerate a mistake more easily than a medical diagnosis system. Qian describes retrieval-augmented generation as one way to give models access to external knowledge, since model weights are not an efficient place to store every fact. The panel also discusses generating a search query, retrieving results, and summarizing them with references.
