An agent harness should give models fast feedback and protect them from repeated mistakes without forcing every task through a fixed path.
2
Production AI SRE work is difficult because the answer can be anywhere in a changing environment, while operators expect much higher accuracy during stressful incidents.
3
The durable value of an AI SRE is learning company-specific decisions, constraints, and operational history so that future agents can prevent incidents and act within clear bounds.
Summary
Shahram Anver explains how Cleric changed its AI SRE architecture as models improved. The company moved from a complicated ant-colony design with many specialized workers to a simpler query agent, then relaxed rigid tool restrictions as models became better at composing tools. That change required strong sandboxing because bash and Python give an agent much more freedom. Anver argues that production reliability depends less on fast investigation than on verification and learning. An agent must understand how a particular company operates, preserve decisions made in Slack and incident work, and use current code and infrastructure state as sources of truth. He also describes a future in which coding, SRE, and security agents coordinate through explicit policies, shared knowledge, and exception handling. Humans still provide judgment, taste, and boundaries. Their role is to review important decisions rather than inspect every generated line of code.
A harness should create fast feedback while limiting repeated mistakes
Anver defines a harness as everything between the user and the model, including prompts, skills, logs, files, and tools. Its job is to give an agent fast feedback because agents can do foolish things and need to adjust quickly. Early Cleric tools were deliberately rigid. Instead of giving the agent the Kubernetes CLI, the team wrote narrow custom tools that allowed only specific actions. As models improved, those restrictions became a performance limit. Anver now prefers putting more non-determinism inside a strong sandbox, so the model can reason freely without being able to damage the surrounding system.
Harness design should follow observed failure modes
Cleric reviews investigation traces and labels what went wrong. The team looks for issues such as a log query flooding the context window or a query taking six attempts when one should have been enough. Repeated mistakes may justify a harness change, but Anver warns against fixing every imperfection with more scaffolding. An agent that is always perfectly predictable may be overfit to one case. Some mistakes are acceptable when the agent can detect them and correct course. He gives a simple CI/CD security or code-conventions check as an example of a task that can use a straightforward pass-or-fail flow instead of a complicated agent.
Better models made a simple query agent more effective than an ant colony
In late 2023, Cleric considered an architecture modeled on an ant colony. A queen-like incident commander would dispatch workers with distinct jobs. The team spent months on that design while using early GPT models that needed heavily constrained prompts. They eventually rebuilt the system as a single query agent: give it a question, let it decide where to look, and have it answer. Anver says the simpler design worked better. Rigid tools helped make behavior deterministic when models were weaker, but later models could use bash, Python, and composed tools. The tradeoff was security, which made sandboxing necessary.
The hard part of an AI SRE is verifying the answer in each environment
An agent that knows Kubernetes and general SRE practices can still fail when moved between companies. Anver compares this with his own experience moving from Gojek to another ride-hailing company: the high-level problems may match, but local systems and conventions differ. Cleric therefore spends more effort testing across varied environments and checking whether an investigation is correct. Finding a likely explanation is easier than verifying it. A severe outage can be binary and obvious, while a smaller latency increase may have many plausible causes. The agent needs company-specific context to know which parts of that search space deserve attention.
Operational memory turns incident work into future guidance
Anver describes learning as preserving the decisions and boundaries that humans usually leave implicit. Slack conversations contain useful history about what failed before, what engineers tried, and which actions worked. Cleric can turn that information into a compact map for a human to review and edit. People can explicitly tell it to remember or forget something, while ratings provide another learning signal. Anver accepts that memory will contain useless information, which he calls rot, so curation matters. Slack, code, and logs are especially useful, but current code and Kubernetes infrastructure state provide the present-day source of truth that historical conversations may lack.
Production SRE agents face a higher cost of being partly right
Anver contrasts coding agents with SRE agents. If a coding agent produces 70% of a feature, the user may still have saved much of the typing and can test the result cheaply. During an incident, an SRE agent that is 70% right can send an operator down the wrong path while customers are affected and stress is high. Production systems also have a different search shape. A coding agent works within a finite codebase while the cause of an SRE problem may be anywhere across services, logs, infrastructure, and history. This combination of technical breadth and human risk makes production reliability much harder than a convincing demo.
Humans should review decisions and exceptions rather than every line
Anver expects people to work as both managers and craftspeople. Agents can produce large amounts of code, so reviewing every line is inefficient. A better review point is the decision trace behind the code, such as whether a product should be a macOS app or a web app. Humans add value through taste, judgment, and real-world constraints. An agent can operate within stated guidelines, but it should surface an exception when it is about to leave those bounds. In SRE, this means preserving decisions about when an agent may roll back a change, restart something, or take another action at 2 a.m.
Agent teams need explicit roles, communication, and isolation
Anver expects coding, SRE, and security agents to work as separate experts with different objectives. They should exchange the information needed for a task without sharing every internal detail or filling each other's context. Cleric already experiments with an investigation auditor whose job is to challenge an investigation and demand more work. Anver sees a future with agent-team governance similar to a RACI matrix: an agent may be responsible, accountable, consulted, or merely informed. The difficult design problem is preserving isolation while allowing agents to share resources and operational knowledge when needed.
The durable product is the domain-specific learning loop, not the agent shell
Anver says Cleric avoids spending time on areas where model providers and general agent builders will improve faster. The company moved away from fine-tuning and treats the basic investigation agent as increasingly commoditized. Its focus is verification, learning, and domain-specific operational knowledge. He expects an SRE agent eventually to notice that a coding-agent change broke production, reverse it when policy permits, watch the metrics, and report the decision. If the case is outside its rules, it should ask a focused question. This requires preserved context and clear policies rather than another layer of generic prompting.
