Production AI systems need careful evaluation, user feedback, and manual review after the first prototype is working.
2
Teams are moving toward open models when they need lower cost, local data control, stable versions, or more control over model behavior.
3
Agent platforms need governance, observability, constrained actions, and telemetry that records tool calls and other steps.
Summary
The panel discusses how AI platforms are changing as teams adopt agents, open models, and more distributed systems. The hackathon example grounds the discussion: more than 100 teams built mental-health conversational systems, while the organisers ran evaluation infrastructure on Kubernetes with hosted models, GPUs, CPUs, and a leaderboard. The speakers describe why teams may move from API-based models to locally hosted open models. Cost, data residency, model stability, and control over inference all matter once a product reaches real usage. They also describe the hard work between an 80% prototype and a production system, including evaluation, user feedback, software engineering practices, and human review. The panel treats agent governance as an operational requirement. Agents need limited permissions, observable tool use, and safeguards against irreversible actions. Allen Roush also argues that constrained generation can enforce tool order and output formats, although software sandboxes remain an imperfect defence against capable coding agents.
A self-service evaluation platform helped the hackathon handle large submission volume
Frédéric Bénard describes a mental-health hackathon organised with Bell Canada and Kids Help Phone. More than 100 teams built models or prompts that decided when a conversation should move to a human, including cases involving suicide ideation. His team built an evaluation pipeline on Kubernetes in the BuzzHPC environment, using hosted language models, GPUs, and CPUs. Teams could trigger jobs themselves, inspect their scores, and compare results on a leaderboard. The system received 1,000 submissions during the week. Shuo Wang adds that a hidden dataset introduced midway through the event, forcing teams to reassess systems that had performed well on earlier data.
The platform boundary depends on reliability, scale, support, and governance
Shuo Wang says an AI platform should give teams reliable service, rapid scaling, and fast support when something fails. Built-in guardrails can also reduce the governance work that developers otherwise have to handle themselves. Allen Roush describes BuzzHPC as a sovereign AI cloud with infrastructure for training, fine-tuning, inference, and production workloads. He argues that a platform can provide the compute, model-serving infrastructure, notebooks, and user interfaces while teams focus on their applications. The speakers leave room for developers to control more of the model and agent stack when they need specific behaviour or stronger data control.
Open models become attractive when API cost and model control start to matter
Frédéric Bénard says many teams begin with an API key because it makes experimentation easy. Later, they may need a smaller model for a particular use case, want to keep data in their own country, or need to reduce token costs. Hosting a model brings new work, including GPU management and model serving, so a local provider can help teams make that move. Shuo Wang describes a practical example from the hackathon: frontier-model usage limited synthetic-data generation and evaluation, while open models on BuzzHPC could be used without the same token limits. A frozen open model can also avoid unexpected provider upgrades that break prompts and tests.
GPU and model choices depend on workload, generation, memory, and changing prices
Allen Roush separates GPU size from GPU generation. Newer hardware can improve inference for highly quantized models, while older GPUs may still have enough memory for models that cannot fit on smaller new cards. He says model selection also depends on the task. A highly capable model is useful when success is uncertain, while smaller models can handle routine work. He recommends caution with small local models for complex agent communication, since their limitations become visible on longer contexts. BuzzHPC supports scaling inference down to zero and back up, as well as multiple inference instances. Roush also gives current example prices near the end: H100 at $2.50 per hour, H200 at $3.50, and A40 at $0.50.
The difficult part of an AI product starts after the first prototype
Frédéric Bénard says generative AI prototypes can reach about 80% of the desired result quickly. Moving from 80% to the 95% needed for production requires evaluation, iteration, user feedback, and attention to specific failure cases. He warns against relying only on one language model to judge another, since that simply moves the evaluation problem elsewhere. Human experts and users still need to inspect outputs. The panel also connects this work to ordinary software practice. Teams should define requirements, choose an architecture, write tests, review generated code, and build incrementally. Agents can help with these activities, but they do not remove the need for them.
Agent stacks are changing too quickly for teams to depend on one fixed framework
Frédéric Bénard describes a stack that has grown from a language model and prompts to retrieval, tools, memory, context, MCP servers, and agent harnesses. At Mila, his team builds early applications with research and industry partners, and he expects the implementation to change within months as new practices appear. His advice is to choose what a particular product needs without becoming attached to the current stack. The application still has to deliver value today, but the team should expect to replace parts of it later. Shuo Wang describes three enterprise paths: a common platform for simple agents, AI features embedded in existing software, and custom production systems using frontier or open models.
Governance and observability must constrain what agents can do
Shuo Wang says enterprise agents need governance and observability alongside useful capabilities. The controls should cover access to databases, resource creation, and company processes, not just obvious failures such as deleting a database. He gives an example of an agent making a large number of unintended Jira payments. Allen Roush recommends recording tool calls and MCP-server calls so teams can see where an agent entered a loop or invented an action. AgentOps and Arize Phoenix are mentioned as tools that use OpenTelemetry-style traces. The panel also recommends limiting irreversible actions with wrappers or permissions. Prevention is preferable to discovering the problem after the agent has acted.
Constrained generation can make agent workflows more predictable, while sandboxes remain limited
Allen Roush describes constrained or structured generation as a way to restrict the model's possible outputs. A schema can require a numeric range, a tool call, or a particular order of operations instead of relying only on instructions in a prompt. He connects this approach with Pydantic and frameworks such as XGrammar, Outlines, and Guidance. It can support repeatable workflows such as research or domain-specific document generation. The same control raises safety concerns because prefix editing and output constraints can weaken the safeguards of closed models. The panel also warns that software sandboxes are only one layer of protection. A capable coding agent may still escape a software-only sandbox, so permissions and hardware boundaries matter.
