LLMs generate useful text through next-token prediction, but they remain prone to hallucination and are slower than conventional machine-learning systems.
2
Applications should combine LLMs with fresh information from sources such as web pages, search results, enterprise data, screenshots, and OCR.
3
Teams should begin with the strongest available models, evaluate qualitative outputs with human feedback, and use traditional machine-learning evaluation for quantitative tasks.
Summary
Yangqing explains LLMs through the example of Chinese typewriters. Their layouts were optimized by counting which characters appeared together, much like next-token prediction. Modern LLMs use longer context, GPUs, and Transformer-based statistics, but they still produce hallucinations, run more slowly than conventional models, and do not behave as interchangeable services. He describes building Elmo Chat, a browser tool that summarizes web pages and videos, in two days with a small team. His practical advice is to start with a strong hosted model, prepare application-specific knowledge, and use retrieval or other forms of augmented generation. Qualitative applications need human review and prompt iteration. Quantitative tasks should use established machine-learning methods such as cross-validation. As applications become more focused, smaller models and custom data become more useful. He also describes AI infrastructure as an additional layer alongside web and data services.
LLMs are advanced next-token predictors rather than mysterious intelligence
Yangqing compares LLMs with Chinese typewriters, whose layouts were arranged by counting which characters appeared together. A better layout let operators move between frequent combinations more quickly. LLMs also predict what comes next, though they use GPUs, long context, and Transformers to make more complex predictions. This explains why they can produce reasonable responses without implying that they have general intelligence. The analogy is meant to reduce the hype around claims that LLMs are already AGI or superintelligence.
LLMs are useful, but hallucination, latency, and inconsistency shape the design
Yangqing says LLMs can generate very reasonable responses, while still remaining relatively brittle. Hallucinations are prominent, and they run more slowly than conventional machine-learning models and other systems. Even when services appear interchangeable through an API standard, their behavior differs in practice. Teams have to reduce hallucinations, fit prompts and expected runtimes to application latency, and decide how to supply information that was not present in the model's original training data.
Fresh application information can come from more than conventional retrieval
Applications often need current search results or private enterprise data in addition to the model's training knowledge. Yangqing uses the broader idea of augmented generation because the added information does not have to come from a retrieval system. Elmo Chat can parse a web page, while another application might take a screenshot, run OCR, and send the extracted information to an LLM. The source of the context should match the task and the information the application needs.
A narrow LLM application can move from idea to prototype in days
Lepton built Elmo Chat as a browser extension for summarizing long web pages and YouTube videos. It extracts the page content, sends it to an LLM, and returns summaries, opinions, keywords, and takeaways. The team built the first version in two days, with one frontend engineer using an available model through an API. They also added multilingual support by translating the core prompt from English to Japanese. Yangqing says similar prompt-based approaches have reduced development time for some applications from months to days.
Strong hosted models are a practical starting point for new applications
Yangqing recommends starting with available services such as OpenAI and Anthropic when building an application. Hosted models usually provide the best available performance, which gives a team a reference point for what the application can achieve. The team can then prepare its own knowledge and put it into a knowledge base or another format that an LLM can use. Only after the application and its requirements become clearer does it make sense to move toward more specialized or smaller models.
Evaluation depends on whether the output is qualitative or quantitative
For summarization and other qualitative applications, Yangqing recommends putting early versions in users' hands and checking the results directly. Teams can track qualitative measures and use the feedback to improve prompts and content. Quantitative tasks need a different approach. For example, a customer-support system that predicts whether a user is happy can use conventional machine-learning practices such as cross-validation. Old evaluation methods still apply when an LLM is part of a measurable prediction pipeline.
Focused applications create a place for smaller models and custom data
As teams gather application data and move into more vertical use cases, Yangqing expects them to use more fine-tuned and smaller models. A clearer task and a more specific user experience make it easier to choose a model for the actual workload rather than relying on a general-purpose system. He connects this shift to the growth of AI infrastructure, which applications need alongside ordinary web services and data systems.
"AI infra has become the new pillar that a lot of applications need in addition to the general web services and also the data services."10:37
Who should watch
You are deciding whether an LLM feature should use a hosted model, retrieval, custom data, or a smaller specialized model.
You are building a summarization or research assistant and need a practical way to evaluate early outputs.
Your application has measurable predictions inside an LLM workflow, such as customer sentiment, and you need to retain established machine-learning evaluation methods.
