Watch on YouTubeLLMs are more useful as reasoning machines than as sources of facts.
Prompt engineering can add current, application-specific information to an existing model without changing its weights.
Connecting models to data stores and APIs can produce applications that are harder to copy than a thin model wrapper.
Justin Uberti and Jon Turow argue that large language models should be treated as reasoning machines rather than fact machines. Models can generalize, summarize, and perform multistep reasoning, but they need access to the right facts for a useful application. The speakers describe three ways to provide that information: training a model from scratch, fine-tuning an existing model, and prompt engineering. They focus on prompt engineering because it can retrieve data from stores and APIs at request time, works with different LLMs, and can be implemented in software without changing model weights. Demos show a model using supplied time-zone information, reading a conference web page to identify speakers from venture firms, and combining GitHub data with a charting API. Their argument is practical: the application's surrounding data and tools can provide differentiation while the model handles interpretation and reasoning.
The speakers point to Stanford work on emerging capabilities and describe how large language models can perform complex reasoning over multiple steps. They can reason outside the domain on which they were trained and handle tasks such as generalization and summarization. This capability is the basis for treating an LLM as part of an application's reasoning system rather than simply as a database of remembered facts.
An LLM may know stable information such as the president of the United States, but it will struggle with facts that are dynamic or outside its training data. The speakers compare asking a model to work without the data it needs to leaving a person in an empty room with no tools. Supplying relevant facts gives the model material to reason over and reduces confident errors.
Training a model from scratch is presented as an option for teams that want to push the technical envelope or support structures and concepts that differ radically from other models. The speakers frame this as a choice for sophisticated use cases. It is one of three ways to put application-specific information into a model.
Fine-tuning starts with an off-the-shelf model and edits its weights so the model becomes more familiar with a particular dataset. The speakers say the effect can be more surgical or more profound depending on the amount of data. They give a range from 50 to 50,000 rows and from 2,000 to 2 million tokens.
With prompt engineering, the application does not alter the model's internals. It defines a query and response format, retrieves information from a data store or API, and adds that information to the user's input. This gives the application dynamic control over the model's output and lets it use specific facts on top of an LLM of the developer's choice.
In a demo, the speakers give the model the current time in Seattle and ask for the time in New York City. The model uses its knowledge that the cities are in different time zones to infer the answer. The point is that the application supplies the changing fact, while the model uses its world model to perform the conversion.
Manually adding facts to every prompt does not scale. The speakers show a model reading a conference web page, extracting its speaker information, and using its knowledge of venture firms to identify speakers from Madrona and Factory. The model can also decide when it needs an external tool because the requested information is absent from its existing world model.
The final demo asks Fixie to find contributors to the Transformers repository on GitHub, obtain their contribution counts, and send the results to an external chart API. The workflow combines retrieval from GitHub, model reasoning about the requested task, and a separate service for visualization. The speakers use this composition to show how external systems can add capabilities around an LLM.
"Prompt engineering is the most flexible and programmable way for us to go and extend and differentiate applications."Justin Uberti13:15