Rohit Agarwal argues that production GenAI systems need an operations layer for observability, model routing, prompt management, and security.
2
An AI Gateway can route requests across models, apply fallbacks, evaluate responses, and keep application code independent from changing model APIs.
3
Forward compatibility means testing new models with sampled traffic and phased releases instead of waiting for the GenAI market to settle.
Summary
Rohit Agarwal describes the tooling needed to move generative AI applications from experiments into production. Drawing on large deployments, he discusses model error rates, spiky infrastructure demands, semantic caching, model routing, and the growing complexity of applications built with foundation models. He groups the emerging LLM Ops or FM Ops layer into observability, an AI Gateway, prompt management and experimentation, and security and compliance. The central design idea is forward compatibility. Applications should call a stable interface with a prompt ID and data, while a gateway chooses models, transforms prompts, evaluates responses, and handles fallbacks. This lets teams test new models with shadow traffic and phased rollouts without rewriting application code. Rohit also explains why agents remain unreliable when errors compound across multiple steps, and why smaller task-specific models may make them more practical. He is direct about unresolved problems around permissions, data leakage, prompt injection, and deciding what to open source.
Rohit Agarwal says the technology is powerful, but production-grade applications need tooling around the model APIs. He compares this with the progression from cloud servers to Kubernetes and additional DevOps tools, which let engineers spend less time managing infrastructure. In his analogy, the foundation model becomes the compute unit, with vector databases, deployment tools, orchestration libraries, and an emerging LLM Ops or FM Ops layer around it. That layer is needed to build applications beyond simple chat-with-your-document examples. The goal is to let teams focus on product behavior instead of repeatedly solving deployment and management problems.
Model reliability is difficult at both the API and infrastructure levels
Rohit says Portkey.ai handles about 3 million API requests per day and has seen an error rate of about 5% on the OpenAI base API. Production systems therefore need retries and fallbacks. Open-source models bring a different set of problems, including high compute use, traffic spikes, troughs, and uncertainty about performance and efficiency. Once basic deployment is addressed, teams still need new approaches to caching, including semantic caching, and systems for switching between models. Rohit describes production work as a sequence in which solving one operational problem reveals the next.
Most deployed applications fall into summarization, Q&A, or generation
Rohit groups the production use cases he sees into three categories. Summarization covers documents, financial reports, videos, meetings, and turning summaries into action lists. Q&A, especially retrieval-augmented generation, is the most popular category across customer support, sales, internal IT, and data inquiry. He says providing relevant context can produce accurate answers and has led companies to pay for these systems. Generative features include drafting, brainstorming, changing email tone, and fixing grammar. Demetrios Brinkmann adds that grammar and tone tools can help people who do not speak English as their first language.
Rohit describes agents as a framework whose accuracy is currently around the level of GPT-3 in January 2021, or roughly 70% in his comparison. Each additional step propagates the chance of failure, which is why he has seen few agents in production. He expects agents to work first in recurring tasks where perfect accuracy is less important, such as writing a unit test and trying again when it fails. Code Interpreter is one example he gives. He expects systems to combine factual data APIs with probabilistic model steps, with retrieval, question answering, and actions forming a path toward multi-step agents.
Small task-specific models may make agents practical
Rohit is excited by a model architecture in which a larger model orchestrates smaller, highly accurate workers. These smaller models may still be large by older standards, but they are cheaper and faster than the largest available models. He cites natural-language-to-SQL as an example where companies fine-tune Llama 2 on their own data and can outperform larger general models for that task. A reliable sequence could translate a user query into SQL, retrieve data, and turn it into an insight. Rohit says individual research efforts are improving each step and estimates that this could make agents useful within about six months.
Rohit says teams moving an application into production need observability, an AI Gateway, prompt management and experimentation, and security and compliance. Observability covers latency, cost, and accuracy, since a successful API response does not prove that the returned content is correct. The AI Gateway connects applications to multiple models and supports testing, rollbacks, load balancing, and fallbacks. Prompt management gives teams a prompt library and a way to run experiments. Security and compliance cover issues such as data leakage, data residency, GDPR, and access controls. He describes these parts as an evolving full-stack LLM Ops layer.
Forward compatibility requires a stable boundary around model calls
Rohit defines forward compatibility as protection against investments becoming obsolete as models, databases, and frameworks change. Instead of hard-coding an OpenAI call into application code, the application can request an LLM call with a prompt ID and its data. The AI Gateway chooses a model according to configured rules, transforms the prompt and response, and returns a consistent format to the application. Teams can send a sample of traffic to a new model in dark mode, evaluate its responses against the model users see, then release it to 5% of users before moving toward 100%. This makes model changes resemble controlled CI/CD releases.
Security controls must cover both inputs and outputs
Rohit says the gateway can sit between the application and every model call, so it can check requests before they leave the company's boundary and inspect responses before returning them to users. He discusses personally identifiable information anonymization, data residency, denial-of-service protection, and virtual keys stored in a secret vault with spending limits. He is also concerned about vector-database permissions and fine-tuning confidential company data, since a model could expose information to users who should not have access. He says prompt injection and leakage remain open problems, and he does not claim that Portkey.ai has solved the whole security and compliance area.
"You probably need to have a system in between your application and generative AI that allows you to keep up with the new models, the new databases, and the new frameworks that come into play."Rohit Agarwal46:24
Who should watch
You are building an LLM application and need to decide what belongs in an operations layer before production.
Your model provider or model choice changes often, and rewriting application code for every switch is slowing down testing.
You are evaluating agents, fine-tuning, or retrieval systems and need a grounded discussion of error compounding, permissions, and data leakage.
