Chat interfaces give language models flexibility, but they make users put context, intent, and desired actions into a small prompt box.
2
Interfaces can make language models easier to use by combining shared workspace context with pointing, selection, contextual actions, and constrained suggestions.
3
Creative AI tools should narrow the action space enough to avoid broken or empty states while leaving room for surprising results and user exploration.
Summary
Linus Lee argues that chat is a useful starting point for working with language models, but it should not be the only interface. Chat makes users describe the surrounding context, their intent, and the desired action in text. Lee proposes interfaces where the model shares the user's workspace and can see selections, open files, cursor position, and other signals of attention. Pointing, naming, dragging objects into prompts, context menus, autocomplete, and generated interactive components can give users clearer actions without removing the model's flexibility. He also recommends using predictions and constrained happy paths, with chat available when the user needs something less expected. For creative tools, Lee wants a balance: enough constraints to avoid broken artifacts and blank-page syndrome, while preserving surprising outputs that encourage exploration. The talk is about designing the interaction around a model, rather than placing a model inside a chat box.
A conversation starts with shared context before the user types an intent
Lee breaks a language-model interaction into shared context, user intent, internal reasoning, and action. The context might be a workplace, a supermarket, or a shared activity, and it changes how an incomplete phrase or gesture is understood. A user saying "the blue brick" while assembling a Lego kit can communicate a complete request because the surrounding situation supplies the missing information. In many language-model applications, the context is close to empty, so users must put it into the prompt. Lee connects this to the difficulty of prompting. In a richer application, the model can use the surrounding workspace to interpret what the user means before it produces text or takes an action.
Workspace context should include where the user's attention is focused
Lee uses GitHub Copilot Chat inside VS Code as an example. The model can use open files, an open terminal, recent commands and their outputs, the cursor position, and selected lines. Selection is a strong signal because it works like pointing at something on a screen. A request such as writing unit tests for selected code becomes easier to interpret when the system knows exactly which code the user means. Lee makes a similar point about Microsoft Copilot for Excel. Rather than naming a column verbally in a separate chat sidebar, the user should be able to select or point to it inside the shared workspace. The agent should inhabit the same working environment as the user.
Pointing and selection turn an open-ended request into an object and an action
Lee describes pointing and selection as a noun-and-verb pattern. The noun is the object, such as a spreadsheet column, a page, a writing block, or a file. The verb is the action, such as filtering, aggregating, hiding, deleting, duplicating, or summarizing it. Web interfaces already support some of this through selectable text, copy and paste, and drag-and-drop. Chat interfaces remove many of those affordances, so users have to describe both the object and the action in words. Bringing selection back can clarify the model's context and direct its output. A user might select text, ask for a summary, and specify where the summary should go.
Interfaces can expose context through full visibility, names, or rich objects
Lee describes several ways a model can observe what the user is referring to. An omniscient model can access everything in the application, although it must infer what matters. A named-object interface lets users refer to an artboard or panel by name, which could work well in tools such as Figma or Sketch. A richer prompt could let users drag a paragraph, image, or contact into the input area as a literal reference. Lee also discusses contextual actions, such as a right-click menu that proposes actions for the selected object. Autocomplete in a code editor or terminal follows a similar pattern: the user holds an object or command context, and the system shows what actions are available.
Constraints can make language-model interfaces easier without removing their power
Lee says the interface should balance intuitive use with the flexibility of a language model. A chat box exposes almost anything the model can do, including tool use and programming instructions, but it gives users little help with discovery. This creates blank-page syndrome: the user may know they want a summary or a certain style of response, yet the interface offers no suggestions or visible affordances. Selecting an object first lets the system recommend likely actions, add structure, and provide guardrails. The user can still fall back to open-ended chat for advanced requests. Lee's proposed pattern keeps the model's broad capabilities while giving users a clearer starting point.
Generative tools should shorten the loop between trying something and judging it
Lee focuses on feedback loops in creative and productivity applications. A user has an idea, sees a result, evaluates it, and decides what to change. Instead of producing one image, a system could show several options when the results are easy to compare. Choosing among options can be easier than describing an exact result from a blank page. Lee says this approach is less suitable for outputs such as essays, where evaluating several complete results is harder. He also describes interactive components generated inside a chat response, such as a weather widget with controls for time or precipitation. These controls let users explore without writing another prompt.
A good generative interface limits failures while preserving surprising results
Lee cites the essay "Casual Creators" and its distinction between the possibility space and the action space. A creative tool should exclude broken artifacts, such as models that fall over in a 3D-printing application, but it should still allow surprising artifacts. If every result is uniform, users lose the reason to explore. His design target is therefore a light set of constraints that keeps users from getting stuck while leaving room for unexpected discoveries. Predictive interfaces can offer likely actions or multi-step trajectories, but they should keep an escape hatch to ordinary chat. The constraint should help the user start and continue, rather than fully determine the outcome.
The model should share the workspace and lead with constrained actions
Lee's closing design proposal combines the earlier ideas. An agent should co-inhabit the user's workspace, see the relevant details, and use the shared context to interpret actions without forcing every detail into a prompt. The interface can begin with constrained, likely actions that models predict from the user's focus and current state. If the user wants something more advanced or unusual, ordinary chat remains available. Lee also wants direct manipulation through a mouse or touchscreen to speed up iteration. His example is an interface that responds to selection, pointing, and controls instead of requiring the user to explain every step in text.
"The possibility space of creative tools and what you can do, the action space should be narrow enough to exclude broken artifacts like models that fall over or break when you're in a 3D printing app, but it should be broad enough to contain surprising artifacts as well."19:54
Who should watch
You are designing an AI feature inside a document, code, spreadsheet, or creative application and need interaction patterns beyond a chat sidebar.
Your users struggle to know what to type into a general-purpose language-model interface.
You are building a creative tool and need to decide how much to constrain model outputs while preserving exploration.
