AI systems expose users to more uncertain inputs and outputs than traditional software, so teams must plan for the trade-off between agency and control.
2
Continuous calibration and continuous development start with a high-control system, collect real usage data, and increase autonomy as the system earns trust.
3
Production AI work depends on task-specific evaluations, manual error review, fallbacks, and data-driven changes before teams add more tools or agents.
Summary
Aishwarya Naresh Reganti argues that teams should stop treating generative AI as a normal software feature. Traditional software has a defined interface and predictable user behavior. AI systems accept varied inputs, produce non-deterministic outputs, and can make several linked decisions, especially when they have agency. Her CCCD framework, continuous calibration and continuous development, starts with a high-control version of a product. For customer support, that means routing tickets before moving to a co-pilot and then an autonomous agent. Teams should create reference data, log interactions, add human fallbacks, define task-specific evaluations, deploy early, sample real traffic, and manually inspect failure patterns. Those findings drive prompt, model, retrieval, or architecture changes. She recommends deterministic evaluations where possible and says each agent needs its own evaluation process. Her answer is practical and skeptical of adding agents or tools before a team understands system behavior.
AI systems have a much wider and less predictable interaction surface
Traditional software usually gives users defined controls such as buttons, clicks, and options. AI systems accept conversations, videos, images, and other inputs, then rely on a model to interpret them. Aishwarya says this creates uncertainty on both the input and output sides. Autonomous systems add another source of complexity because each decision can affect the next one. Errors can ripple through the system, so adding features is no longer a matter of layering predictable components and assuming that integration tests are enough.
Aishwarya frames AI product design as an agency-control trade-off. When a builder gives a system more agency, the builder loses some control. The practical question is whether the system has earned that agency. Her CCCD framework, continuous calibration and continuous development, handles this by starting with tighter control and moving toward greater autonomy only as the team learns how the system behaves.
Customer support should move from routing to autonomy in stages
Her example begins with a reference data set and a high-control customer support system. Version one routes tickets to the correct department. Version two acts as a co-pilot that retrieves relevant standard operating procedures and suggests resolutions without carrying them out. Version three drafts replies and automatically resolves common tickets. Each version reduces control and increases agency, allowing the product to gain trust over time instead of starting with a fully autonomous agent.
The first system should be simple, observable, and easy for humans to override
Aishwarya recommends building the simplest architecture that can solve the reference data set. Teams should log user interactions and system responses, add fallback mechanisms, and make it easy for users or operators to overwrite AI decisions. She says companies often forget to decide what happens after a wrong AI decision. Early versions need a clear way for humans to take control, and teams should avoid overengineering before they have behavior data.
Evaluations must measure the task and account for non-determinism
The evaluation step asks what success means for the particular task. A routing system might use routing accuracy, while a retrieval pipeline needs retrieval metrics. The reference data set helps validate whether the evaluations cover important cases. Aishwarya says evaluations should be actionable and should not cost more to run than the application itself. Teams can often begin with simple checks instead of immediately relying on LLM judges or LLM juries.
Real traffic reveals failure patterns that tests miss
After deployment, teams use logs and their evaluations to establish a baseline on real user data. Aishwarya recommends sampling when interaction volume is high, then manually reviewing the weakest segments because those contain the strongest signal. She suggests pulling roughly 20 to 50 low-accuracy cases per segment in an early version. A team might find that password reset tickets go to the account team because the system latches onto the word 'account'. The fix could be a prompt change, a model change, or a retrieval change, depending on what the evaluation shows.
Architecture should evolve from evidence rather than guesswork
Once teams document repeated failure patterns, they turn those patterns into prompt edits, improved retrieval, or other architecture changes. They rerun evaluations after each change and repeat the loop. The process creates a data flywheel while gradually increasing agency and reducing cascading errors. At different stages, teams learn different things: routing reveals needed context, co-pilot work can expose conflicting standard operating procedures, and autonomous operation can show where a department is unsuitable for full autonomy.
Agents need deterministic controls and separate evaluation
In the question period, Aishwarya says live incentives for agents remain an early problem because the underlying models are stateless and do not retain everything from earlier turns. For now, teams usually address failures offline by changing prompts or adding mechanisms. She prefers permission checks, guardrails, and rule-based access controls over prompting alone. For multiple agents, she recommends building each one in isolation before adding interaction. Each agent still needs its own evaluation because performance on one data distribution does not automatically transfer to another.
