# What if AI Doesn't Need Structure, It Needs Connection?

David Vargas Fuertes, Vellum | MLOps Community | 28:19

Source: https://www.youtube.com/watch?v=wZ2icNcFX2A
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/what-if-ai-doesnt-need-structure-it-needs-connection
Published: 2026-01-16
Tags: prompt-engineering, structured-outputs, tool-use

## TL;DR
- LLMs produce better results when they can communicate in formats that resemble the language in their training data.
- AI applications should keep deterministic parsing and workflow changes outside the model, so the model can focus on interpreting user intent.
- Vellum uses Markdown, XML, Python, and diff syntax to let its agent builder produce structured output without asking the model to generate large JSON objects.

## Summary
David Vargas Fuertes argues that AI systems often force language models to communicate in JSON even though models are trained on much more natural language, code, Markdown, and other formats. He compares this with an intern who struggled to summarize a customer call as a spreadsheet or PDF but succeeded when asked to make a TikTok. Vellum saw a similar problem in its visual agent builder, where users had to learn the product's internal language. Its newer agent builder lets users describe what they want in English and generates the workflow. Under the hood, Vellum uses Markdown and XML for rendered chat interfaces, Python code for graph structures, and universal diff syntax for workflow edits. David argues that models should handle interpretation and inference, while parsers, tools, evals, and other system layers handle deterministic constraints. He accepts that this leaves room for hallucinations and says the surrounding system must account for them.

## Key ideas
### People work better when they can produce output in a familiar language
[02:18](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=138s)
David opens with an intern who attended a customer call and was asked to summarize the findings. The spreadsheet was a confusing set of rows and columns, including a column called "vibes". A PDF report also failed. The intern finally succeeded when asked to record a TikTok, producing a short, useful summary in a format that felt natural to them. David connects this to research on language and concepts, then applies the same idea to language models. If a model is asked to produce output in formats that resemble the material it was trained on, he says the result can improve.

### JSON became a workaround for making model output usable by software
[06:02](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=362s)
David traces the move from asking a model to classify a tweet in a sentence to requesting a JSON object with a sentiment field. Natural-language answers required regular expressions and substring parsing, which became brittle as use cases grew. Few-shot examples improved the format, but models still added backticks, changed key casing, or returned invalid JSON. Function calling, which David dates to June 2023, made it easier to connect models to existing software and helped drive the rise of AI agents. He says the remaining problem is that providers still cannot guarantee format and schema compliance.

### Structured output still asks models to perform deterministic work poorly
[17:17](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=1037s)
David says JSON places deterministic tasks inside the model. The model must produce matching braces, quotes, colons, keys, and schema details, and applications often need retry loops when it fails. He also points out that these formatting tokens add output that does not express the user's intent. The design question is how to keep output structured enough for software while using a language that fits the model's training. His proposed split puts interpretation and inference in the model, while parsers and other application components handle predictable transformations.

### Vellum changed its product after users struggled with its internal language
[12:03](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=723s)
Vellum had spent more than two years building a drag-and-drop sandbox editor for AI systems. Customers repeatedly became confused about how to build agents and needed substantial support. David compares this with his intern story: the product asked users to speak "Vellum" instead of responding to the language users already knew. The new agent builder lets users describe what they want and generates the workflow beside their request. The change is an interface decision, but it also required finding model-friendly representations for the generated artifacts.

### Markdown and XML give chat interfaces structure without making the model emit JSON
[14:24](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=864s)
For its chat renderer, Vellum needs to place interactive UI inside generated text. Examples include asking for an API token, requesting confirmation, or collecting another user input. David says rich-text systems often represent these elements with abstract syntax trees, which become large JSON structures. Vellum instead asks the model to produce Markdown with small XML components, then translates that output into the rendered interface. He argues that models have encountered Markdown and XML-like text more often in forums and other training material than they have encountered the application's private JSON representation.

### Python code lets the model describe workflow graphs through a familiar code-generation task
[16:10](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=970s)
Vellum's sandbox contains nodes and edges, and the original product represented each node as JSON with types, attributes, inputs, and data. The data model also had legacy and modern forms that were difficult for both the company and the model to manage. Vellum now represents the graph through a Python SDK. Each node can be a Python class, and the model can generate code that references upstream nodes. David chooses this approach because code generation is a common model use case and gives the graph a representation that the surrounding system can parse.

### Workflow edits can be expressed as diffs instead of tool calls or JSON instructions
[17:17](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=1037s)
The agent builder edits workflows through a series of changes. Since the workflow is represented as Python code, Vellum asks the model to produce a universal diff, similar to the output of running git diff between files. Parsers then interpret and apply the changes. David says this is easier for the model than generating a JSON description of every change. He presents Markdown, Python, and diff syntax as three formats Vellum is actively using, while leaving room to explore other grammars and domain-specific languages.

### The model should handle ambiguity while the rest of the system handles predictable work
[20:59](https://www.youtube.com/watch?v=wZ2icNcFX2A&t=1259s)
David's main design claim is that AI agent engineers should remove as much deterministic work as possible from the model layer. Models are trained for non-deterministic tasks such as inferring what a user meant or intended. Parsers, processes, tools, and evals can handle syntax, fixed transformations, and system constraints. In the questions, he clarifies that he is not arguing for unstructured systems. He wants the smallest structure that still lets downstream systems process the result. He also says hallucinations may be treated as a model-layer behavior that the surrounding agent system must accommodate and evaluate.

## Notable quotes
- "People in general perform a lot better when they can produce output in the language that they are trained on." (04:53)
- "The title of the talk here is we don't speak JSON and neither should our LM." (15:09)
- "How can we teach Vellum to respond to English?" (13:18)
- "We want to take as much determinism out from the LLM layer." (23:03)
- "I'd rather it's hard to imagine like what the two sides of that argument is but I like to try to if there is other side I'd like to take the other side and say, let the model hallucinate." (23:44)

## Tools & references mentioned
- Vellum
- MIT
- JSON
- function calling
- Markdown
- XML
- Python SDK
- Agent Builder
- CFG grammars
- Anthropic
- OpenAI
- GPT-5
- git diff
- Reddit

## Who should watch
- You are building an agent that repeatedly fails to produce valid JSON or needs retry loops around structured output.
- Your product asks users to learn an internal workflow language instead of letting them describe the result in ordinary language.
- You want to decide which parts of an agent belong in the model and which belong in parsers, tools, evals, or other system components.

## Related talks

- [The Future of User Interfaces: AI is Changing Everything](https://mlopstalks.com/talks/the-future-of-user-interfaces-ai-is-changing-everything) (, 27:33)
- [A New Way of Building with AI](https://mlopstalks.com/talks/a-new-way-of-building-with-ai) (Jiquan Ngiam, Lutra AI, 1:04:46)
- [DevTools for Language Models: Unlocking the Future of AI-Driven Applications](https://mlopstalks.com/talks/devtools-for-language-models-unlocking-the-future-of-ai-driven-applications) (Diego Oppenheimer, Factory, 29:55)
- [The Truth About AI Agents](https://mlopstalks.com/talks/the-truth-about-ai-agents) (Silen Naihin, AutoGPT, 31:40)
- [Controlled and Compliant AI Applications](https://mlopstalks.com/talks/controlled-and-compliant-ai-applications) (Daniel Whitenack, Prediction Guard, 25:13)