Reliable agents should start with narrow, repetitive tasks instead of broad autonomous goals.
2
Agent systems need constrained tools, human feedback, interpretable workflows, and safety controls.
3
Scripted workflows made from understandable nodes are easier to test, evaluate, and improve than fully self-driving programs.
Summary
Travis Fischer argues that most AI agents in 2023 are still toys, despite the attention around AutoGPT and similar projects. He defines an agent as an autonomous or partly autonomous program that uses language models for reasoning, with a task, resources, planning, scheduling, tools, and external memory. The main problems are loops, divergence, the compounding failure rate across subtasks, poor user experience, latency, cost, and safety. Fischer recommends starting with narrow, repetitive tasks and limiting the tools an agent can access. Human feedback can improve performance over time. He also recommends replacing generic model calls with deterministic code when possible. His preferred design is a scripted workflow made from interpretable nodes, rather than one fully autonomous system. Those nodes can support familiar software practices such as unit tests, evaluations, model distillation, and reusable skills. He is direct that production systems should stay closer to deterministic code for now.
An agent combines reasoning with tasks, resources, and planning
Fischer defines agents as autonomous or partly autonomous programs that use AI models to accomplish tasks. An agent receives a well-specified task and resources such as compute time, money, browser access, user accounts, APIs, or proprietary data. Its implementation usually plans or decomposes the task into subtasks, then schedules and prioritizes them. Fischer compares this to an operating system managing threads or other work. External memory can include ordinary databases and vector databases, with vector retrieval helping when exhaustive rule-based programming is too rigid.
Agents are a spectrum between deterministic code and full autonomy
Fischer places agents on a spectrum. At one end, humans write deterministic programs and remain in control. At the other, systems such as AutoGPT or BabyAGI receive a goal and attempt to complete it without further direction. He says jumping straight to fully self-driving programs is like jumping straight to self-driving cars. The practical path starts with constrained agents and gradually moves toward more autonomy as the systems become reliable.
Autonomous systems fail through loops, composition, and poor user experience
As agents become more autonomous, they can get stuck in loops or drift from the original task without recovering. Fischer describes a composability gap: an agent may solve each individual subproblem reasonably well, while the reliability of the complete task falls quickly as more subtasks are combined. Users also need to understand what the agent is doing and intervene when needed. Recurring model calls create latency and cost, while access to the outside world creates safety risks.
Narrow repetitive tasks are the practical starting point
Fischer recommends constraining the types of tasks given to agents. Broad or generic goals are likely to cause problems. He sees a better fit in repetitive, robotic-process-automation work that can run continuously in the background. Tool access should also be limited. If a task needs only a few tools, the agent should not receive access to a thousand. These restrictions reduce the number of ways the system can fail.
Human feedback is a core part of an agent workflow
Keeping a human in the feedback loop gives agent systems a way to improve over time. Fischer sees this as a major difference from traditional workflows because the underlying machine-learning models can learn from feedback. He calls human feedback a fundamental primitive for building agents. It also gives users a way to inspect or correct work before the system takes actions on their behalf.
Reliable primitives and deterministic code should replace generic model calls
Fischer recommends building an ecosystem of reliable primitives and preferring deterministic code wherever possible. A system may begin with a general model such as GPT-4, then replace parts of the workflow with code that behaves predictably. He also describes multi-agent systems as distributed systems, which makes lessons from distributed-systems engineering relevant. Reliability should come from the surrounding design rather than from assuming that a model call will always behave correctly.
Scripted, interpretable workflows make agent systems testable
Fischer prefers a scripted workflow that breaks a task into subtasks and represents them as nodes. Each node remains understandable even as the underlying language models improve. The graph can be generated statically or dynamically, but people should still be able to reason about its steps. Fischer treats this graph as a higher-level programming language where engineers can apply unit tests, evaluations, repeated input-output testing, and model distillation. He also cites Voyager, which builds and evaluates code-based skills that become reusable subroutines.
"Today I think you want to be more on the left hand of that spectrum where you're writing more deterministic code and using LLMs as little pieces instead of big pieces of fully autonomous."16:46
Who should watch
You are building an agent that needs to take real actions and want practical limits on its tools, autonomy, and access.
Your prototype works on isolated subtasks but becomes unreliable when several model calls are composed into one workflow.
You need a design that lets engineers inspect, test, evaluate, and gradually replace model calls with deterministic code.
