An agent starts with a defined environment, a goal, available tools, and measurable success criteria.
2
Early agent deployments should use human review, constrained environments, and simple off-the-shelf tools before teams build custom systems.
3
Real-time agent workflows can combine small language models, traditional machine learning, and ordinary program logic to reduce latency and inference cost.
Summary
Raj Rikhy explains agents from first principles. An agent is an entity that can act autonomously, make decisions, and use capabilities such as tools or function calls. Before choosing a model or framework, teams should write down the environment, the goal, the intended behavior, and the criteria for success. Raj argues that large language models are probabilistic by nature, so developers need to define and govern the deterministic parts of the system. He recommends keeping a human in the loop, testing in a constrained environment, and starting with simple tools rather than immediately building a complex multi-agent architecture. He also discusses how small language models, quantization, and parameter-efficient fine-tuning can help with real-time applications such as fraud detection and supply-chain optimization. The conversation ends with a practical test for tool choice: if a team cannot meet its written success criteria, it should simplify the problem or try the next tool rather than assume it needs a custom framework.
An agent is defined by autonomous action within an environment
Raj defines an agent as an entity, software or otherwise, that takes actions autonomously and has decision-making power. He traces the idea back to reinforcement learning and earlier work in psychology and dynamic programming. The useful questions are what the agent can do, why it should do those things, and how its actions can improve. The environment depends on the application. It can be a game, the pixels on a screen, Microsoft Fabric, or Microsoft Office. Defining that environment limits where the agent operates and gives the system a clear boundary.
Writing down the goal creates the first version of the agent
Before choosing a model or writing code, Raj recommends describing the universe in which the agent operates, the intended end goal, and the behavior the agent should follow. Writing this down is already a practical MVP for an agent. A game agent might need to defeat a character by using a flamethrower. That goal then leads to a list of capabilities, such as understanding the environment, reading a map, and using a tool. In implementation, those capabilities can appear as function calls or tool calls.
Success criteria determine how much freedom the agent should have
Raj says teams need explicit parameters for success. In the game example, the agent might need to reduce an opponent's health to zero or complete an action within a set time. Demetrios points out the trade-off between narrow criteria, which can improve reliability, and open interpretation, which gives the agent more freedom. Raj agrees that over-defining behavior can be a problem, but he still wants the team to understand what success means before the system runs in production. Programming is one way to reach the objective, though it is not required at the start.
Orchestration should be chosen after the problem is clear
Raj describes orchestration as the part of the system that handles planning and decides how functions are called. A service such as Azure OpenAI may handle planning behind the scenes. Other systems expose assistant or function-calling APIs, while teams that use Semantic Kernel, LangChain, LangGraph, or their own code take more responsibility for the plan. His advice is to define the environment and success criteria on paper first. The team should understand the desired behavior before it chooses how much planning to delegate to a model, an algorithm, or application code.
LLM randomness makes developers responsible for governance
Raj is direct about the behavior of large language models. They are probabilistic and stochastic, so fabrication is part of how they work. Developers can add more deterministic tools, constraints, and plans, but they must define and govern those parts. He describes the developer as the architect or person pulling the strings. That person cannot control every autonomous action, but can affect the system's environment, capabilities, and decision points. The goal is to understand which variables can be adjusted before failures reach users.
Early testing needs human review and a small state space
Raj advises teams never to give an agent raw web access out of the box. They should review outcomes themselves during the first MVP iterations and keep a human able to stop an action. An agent can produce a debug stream that explains what it plans to do, or it can simulate its activities before acting. Teams should also constrain the environment and test a small landing space. For a support agent, Raj suggests starting with clearly defined, lower-priority tickets and a limited portion of the documentation before expanding its scope.
Teams should use the simplest tool that can meet the written goal
Raj warns against moving straight from a paper design to custom code. A function-calling API, a tool API, or a no-code product may be enough for the next iteration. He compares this to using an existing database instead of writing a new one whenever a relational database does not immediately fit. AutoGPT is a useful example because its setup and documentation can create problems even for experienced developers. Raj recommends stepping back, trying a common and less cutting-edge tool, and checking whether it can meet the success criteria before taking on custom engineering.
Real-time workflows can mix small models with ordinary program logic
Raj names fraud detection, supply-chain optimization, patient monitoring, farm telemetry, and personal shopping as possible agent applications. Demetrios notes that large language models can be too slow for real-time fraud decisions. Raj points to small language models, quantization, and parameter-efficient fine-tuning as ways to reduce inference latency and cost. He also agrees that every step in an agentic workflow does not need to call a language model. A workflow can combine a small model, traditional machine learning, regular expressions, and other program logic. If speed and inference cost are the remaining problems after the workflow already delivers its intended value, Raj considers that a favorable position.
"An agent is some kind of an entity, software or otherwise, that takes actions autonomously, that has decision-making power and capabilities."Raj Rikhy02:52
Who should watch
You are deciding whether an agent problem needs a new framework, a different model, or a simpler workflow.
Your team is preparing an agent for production and needs a practical approach to testing, human review, and failure states.
You are building a real-time application and want to combine language models with traditional machine learning and ordinary application logic.
