Watch on YouTubeAdept AI trains an LLM to control software through natural-language requests, but the model can take actions with serious consequences when it is wrong.
Simple checks such as reversibility tests, content filters, and user confirmation can limit harmful actions without requiring another ML model.
Adept AI connects product feedback, data collection, engineering, and frequent internal model deployments in 24-hour cycles.
Jacob van Gogh describes Adept AI's natural-language software collaborator, which controls a mouse and keyboard to carry out tasks in tools such as Airbnb and Salesforce. The system starts with a foundation model and fine-tunes it to perform software actions. This creates familiar LLM problems, including hallucinations and overconfidence, with higher consequences because the model acts on the user's behalf. Jacob gives the example of an incorrect "add to cart" request being mapped to Amazon's "buy with one click" action. His proposed response is a guiding system around the model. It can check whether an action is reversible, filter content, and ask for confirmation before actions that cannot be undone. These checks can begin with simple rules rather than ML models, which lets the team test product behavior quickly. Adept also uses user feedback from deployed models to guide data collection and aims for 24-hour internal build cycles.
Adept AI's model takes a user's natural-language request and controls the mouse and keyboard to complete a task. Jacob shows examples involving a weekend search on Airbnb and work in Salesforce. The intended experience is similar to asking an assistant to use software for you while you watch what it does. The system is meant to work across software tools rather than perform only one fixed workflow.
Adept starts with a foundation model that has a general understanding of language, then fine-tunes it to perform software-tool actions. Like other large language models, it may fail to abstain, act with overconfidence, or hallucinate an action that does not fit the context. The difference is that the model is acting for the user. An incorrect answer can become an incorrect operation inside a software tool.
Jacob uses an Amazon example. On one page, a request to add a book to the cart maps naturally to an "add to cart" button. On another page, that button is absent, while a "buy with one click" button is present. The model could treat the two actions as semantically similar, even though buying the book is not what the user asked for. The error is more serious because the purchase cannot be treated like an ordinary navigation mistake.
Jacob proposes surrounding the LLM with a guiding system that checks its proposed actions. The checks can include action reversibility and content filters. They do not need to be ML-based at first. A simple word filter or rule can help the team test how the product feels, find out whether the checks move it in the right direction, and delay more expensive model changes until the behavior is better understood.
When the system identifies an action as difficult or impossible to reverse, it can ask the user to confirm before carrying it out. This creates a boundary between the model's suggestion and the user's final authorization. Jacob describes these checks as inputs that can eventually feed back into improving the large language model and its capabilities.
Adept deploys a model into its product, gathers feedback from users, and uses that feedback to shape its data collection strategy. The team wants to collect examples that have the largest effect on model quality and the user experience. This makes product behavior and data work part of the same loop rather than treating data collection as a separate, fixed phase.
Jacob says Adept strives for 24-hour internal build cycles. Each day, the team builds a model, deploys it to the internal product, gathers feedback, and may update its data collection strategy. Short cycles keep feedback current. They also require reliable internal workflows, good visibility and instrumentation, and close coordination between engineering and data collection.
Jacob argues that teams should focus on failures that are most painful for users, especially actions that are irreversible or have high consequences. He also says teams should continue using non-ML checks and older strategies for controlling automated decision systems. In the question period, he explains that the same general approach could extend to Mac Automator by mapping a user request to the actions needed to execute it, although Adept was then focused on browser tasks.
"We don't need to throw out all of the types of technologies and strategies that we've used in the past for controlling these automated decision making systems that we've been building."07:55