High-agency agents generate many hidden tokens, access sensitive systems, and can become part of mission-critical workflows.
2
Self-hosting fits agents when teams need control over data location, latency, throughput, model versions, and jurisdiction.
3
Enterprise inference should be centralized through a platform team and shared by application teams through a hub-and-spoke model.
Summary
Meryem explains why AI agents create different infrastructure demands from simple chatbot inference. A low-agency system sends a prompt and returns an answer, while a high-agency system may query databases, write SQL, search the web, execute code, call other models, and generate many hidden tokens. These systems often access sensitive data and can run business-critical workflows such as KYC checks. She compares public APIs, cloud APIs, and self-hosted deployments, arguing that self-hosting gives teams more control over data location, performance, model versions, and regulatory boundaries. A basic self-hosted stack needs hardware, an inference engine, and an open-source model, but production also requires gateways, authorization, orchestration, autoscaling, monitoring, and chargeback. Meryem recommends a centralized inference team that provides shared models and infrastructure while application teams retain control over domain-specific customization and fine-tuning.
Agents vary from simple API calls to systems that coordinate tools and other agents
Meryem describes agency as a range. At the simplest level, an application calls a model and receives a response. More advanced systems let a model choose actions from available tools, while the highest-agency systems can start other agentic workflows. In enterprise applications, she says teams usually work somewhere in the middle of this range. A low-agency system has a simple input-output flow. A high-agency system may query databases, perform retrieval-augmented generation, search the web, execute code, or call another model. This distinction matters because the infrastructure and security requirements change as the model gains more control over actions and data.
High-agency agents generate far more work than their visible answer suggests
For a low-agency system, most generated tokens appear in the answer returned to the user. A high-agency system generates tokens during many hidden steps. It may write SQL queries, search the web, look up records, call specialized models, or perform other actions before producing the final response. Meryem compares this with reasoning models, where intermediate reasoning can also involve substantial generation. The visible response is therefore only a small part of the inference workload. This makes agentic systems more computation-intensive than simple chatbots and changes how teams need to plan capacity, throughput, and cost.
Agents create higher data risk because they can reach internal systems
A simple chatbot only sees the information placed in its prompt. A high-agency agent may have access to internal databases, tools, and services, including systems hosted in a different cloud or outside the cloud entirely. Meryem uses a KYC agent as an example. It could check a prospective client against internal bank records and produce a report that would otherwise require manual work. Because such an agent can access proprietary data and participate in an important workflow, a failure or inappropriate disclosure has more serious consequences than a basic chatbot outage.
Self-hosting is most useful when agents need predictable control over data and performance
Meryem compares public AI APIs, cloud APIs, and self-hosted deployments. Public APIs are useful for rapid experiments and access to frontier models. Cloud APIs offer a broader set of models but remain multi-tenant services. With self-hosting, a team deploys an open or custom model on its own VM, on-premises GPU, or private environment. Meryem says this is useful when an agent handles sensitive data or systems, when latency and throughput matter, or when jurisdiction limits access to models in a public service. She also describes changing service quality and noisy-neighbor effects in shared systems as reasons teams may want more control.
A production inference platform needs much more than a model and a GPU
The basic self-hosted setup contains hardware, an inference engine, and an open-source model. Meryem calls this a cucumber sandwich: it can expose an API and run inference, but it is not enough for production or scale. Day-two requirements include an API gateway, authorization, routing across multiple models, orchestration for batch and real-time requests, autoscaling, GPU monitoring, usage tracking, and chargeback. Teams can build these pieces themselves, but Meryem says the work becomes complicated quickly. The production platform must handle operational concerns after the initial model deployment.
Enterprise inference should be centralized because shared models and GPUs change the economics
Meryem argues that each application team should not build its own complete inference stack as enterprises deploy more agents. GPU resources are expensive, and many models share a common backbone. For example, several fine-tuned variants may use the same Llama architecture. A central team can deploy the shared backbone once and let multiple use cases call it, rather than running separate copies. A centralized inference team can also make the platform fault tolerant and resilient in one place. She compares this with traditional machine learning, where models were often specific to one use case and lighter to deploy.
Central platform teams still need to leave room for application-specific customization
Centralization does not mean that every decision moves away from the teams building business applications. Meryem says use-case teams still need to fine-tune models and choose or add tools for their domain. Her proposed structure puts inference infrastructure and shared operational knowledge in a central platform or MLOps team. The application teams consume inference as a service and retain the business-specific knowledge. This creates a hub-and-spoke model: the hub provides shared models and reliable infrastructure, while the spokes build agents for particular workflows.
Self-hosting does not remove security risk, so teams must control the degree of exposure
In the question period, Meryem says a self-hosted agent can still create data security risk when it accesses internal and external information. The risk can be reduced when the model runs in the same environment as the data and when teams control who can see the model output. She describes security as a matter of degree rather than an all-or-nothing property. She also supports hybrid designs, such as using a large model as a controller and smaller specialized models for particular tasks, while warning that privacy concerns may remain if sensitive data is sent back to the larger model.
"A high agency system might be a situation where I have a model that's a controller model but maybe that model has the ability to use a range of different tools based on what the situation and the use case requires."Meryem02:57
Who should watch
You are deciding whether an agent should run through a public API, a cloud service, or infrastructure your organization controls.
Your team can run a model on a GPU but has not yet planned gateways, authorization, autoscaling, monitoring, or chargeback.
You are building several agents and need a platform model that gives application teams customization without duplicating expensive inference infrastructure.
