# We Can All Be AI Engineers and We Can Do It with Open Source Models

Luke Marsden, HelixML | MLOps Podcast | Episode 273 | 54:31
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=6BBm9DlE78g
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/we-can-all-be-ai-engineers-and-we-can-do-it-with-open-source-models
Published: 2024-11-20
Tags: evals, open-source, testing

## TL;DR
- Luke Marsden argues that production GenAI applications need the same CI/CD discipline as other software, with evals replacing traditional software tests.
- AI Spec describes a generative AI application in version-controlled YAML, including its model, prompt, knowledge, integrations, tests, and deployment details.
- HelixML tests integrations as complete workflows, using an LLM judge to check outputs and prevent prompt changes from breaking existing use cases.

## Summary
Luke Marsden explains how software engineering practices can apply to generative AI applications. He compares evals with software tests and argues that AI applications should have version-controlled specifications, similar to Kubernetes manifests. His AI Spec format describes the model, system prompt, knowledge source, API integrations, and tests. A simple knowledge setup can cover common RAG needs, while advanced users can connect custom HTTP endpoints and tools such as LlamaIndex. Luke also describes a path from browser-based prototyping to production. A non-technical user can create an application through a GUI, then hand its YAML definition to a technical team for review, testing, and deployment. His most concrete example is a Jira integration. HelixML uses separate prompts to classify requests, build API calls, and summarize responses. Tests exposed that open-source models needed examples of Jira's JQL syntax. Adding those examples made the tests pass and protected the integration against later prompt changes.

## Key ideas
### GenAI applications need tests and controlled deployments
[04:34](https://www.youtube.com/watch?v=6BBm9DlE78g&t=274s)
Luke Marsden applies familiar software engineering practices to generative AI. He says production-ready software has two basic requirements: tests and deployment. Tests run before code is merged, while deployment records which version should run and makes rollback possible. For GenAI, evals take the place of ordinary software tests. Deployment still requires a controlled, repeatable definition of what should run. Luke's claim is that the workflow is fundamentally the same, even though the meaning of testing changes from checking deterministic code to evaluating an AI application's behavior.

### AI Spec puts an AI application's configuration in version-controlled YAML
[06:10](https://www.youtube.com/watch?v=6BBm9DlE78g&t=370s)
Luke describes AI Spec as a way to define a generative AI application in YAML, much as Kubernetes manifests describe software components. The proposed specification includes the model, system prompt, knowledge, API integrations, tests, and deployment information. Keeping these pieces together allows teams to review changes, reproduce an application, and see whether a prompt or integration change affects existing behavior. Luke says this approach is intended to make GenAI accessible to people with DevOps and software engineering backgrounds, while giving product or business users a path to create prototypes first.

### A simple knowledge definition can cover common RAG needs
[14:07](https://www.youtube.com/watch?v=6BBm9DlE78g&t=847s)
Luke says AI Spec should handle ordinary knowledge use cases without requiring every user to build a sophisticated retrieval system. A knowledge definition might specify a website to scrape, remove headers and footers, follow links to a chosen depth, and refresh on a schedule such as hourly or nightly. Users with more demanding requirements can move beyond that default. They can reference an HTTP server with custom ingestion and query endpoints, allowing an agent to choose between databases or use other advanced retrieval logic. Luke says AI Spec is not intended to compete with LlamaIndex.

### AI development can move from a browser prototype to a technical handoff
[20:21](https://www.youtube.com/watch?v=6BBm9DlE78g&t=1221s)
Luke describes three levels of work. A non-technical person needs only a web browser to create a customized AI application with a prompt, knowledge, and integrations. A person with technical skills can take the resulting YAML, put it under version control, and write tests. A production team can then continue the work with pull requests and deployment practices. Luke was influenced by ChatGPT's GPTs editor, including business examples where people assembled chains of customized GPTs for tasks such as film risk assessments. His open-source approach adds local execution for organizations that cannot send data to an external service.

### The same approach can expose AI features inside existing products
[30:36](https://www.youtube.com/watch?v=6BBm9DlE78g&t=1836s)
Luke says AI Spec can support both internal productivity tools and AI features embedded in a company's existing application. The specification can expose an application through an OpenAI-compatible chat completions API. That endpoint can include built-in knowledge and API actions, so the calling application interacts with an AI service that already has context and can perform business-system requests. Luke says this can shorten the path to adding AI capabilities because OpenAI-compatible APIs are supported across many languages, frameworks, and plugins.

### The Jira integration uses a classifier, request builder, and response summarizer
[42:38](https://www.youtube.com/watch?v=6BBm9DlE78g&t=2558s)
Luke explains that HelixML does not send every user request directly to Jira. One model call classifies whether an API call is needed and chooses the relevant API. A second prompt uses the API's Swagger definition to construct the request. HelixML then makes the API call. A third prompt turns the returned JSON into a natural-language answer. For a Jira request, the system may retrieve issues, construct a JQL query, send that query to Jira, and summarize the response. The user sees a conversational answer while the intermediate calls remain inside the application.

### Integration tests revealed that models needed explicit JQL examples
[39:48](https://www.youtube.com/watch?v=6BBm9DlE78g&t=2388s)
Luke built the Jira integration after a prospect asked for functions such as finding issues, finding issues assigned to the current user, finding overdue issues, and writing code for a specific issue. The open API specification alone did not make the models reliably produce valid JQL. Luke tested several open-source models, including Llama 3.1 and Qwen, and saw garbage queries. He then added examples to the request-building prompt, such as an empty query for all issues, 'assignee = currentUser()' for assigned issues, and 'due date < startOfDay()' for overdue issues. The examples fixed specific failing cases.

### End-to-end evals protect the whole integration from prompt regressions
[49:43](https://www.youtube.com/watch?v=6BBm9DlE78g&t=2983s)
Luke says the tests cover the complete integration rather than each intermediate prompt in isolation. A natural-language test can say that the user asks what issues exist and expect a list of issues with brief summaries or details. An LLM judge examines the final response and decides whether it meets the expected behavior. When a test fails, developers can inspect the internal calls, change a prompt, and run the suite again. Once the Jira examples made the failing cases pass, future prompt changes could be checked before merging, just as failing software tests block a pull request.

## Notable quotes
- Luke Marsden: "My argument is basically to do production-ready GenAI with CI/CD for GenAI, and then we can break that apart." (06:55)
- Luke Marsden: "The AI spec is model, prompt, knowledge, integration and tests." (13:25)
- Luke Marsden: "We need to get to that point as a community, I think, test-driven development for GenAI." (17:07)
- Luke Marsden: "The answer is you do test-driven development just like you do with software, but you're testing the output." (50:11)

## Tools & references mentioned
- HelixML
- AI Spec
- Kubernetes
- GitOps
- LlamaIndex
- ChatGPT
- GPTs
- Cursor
- Claude
- OpenAI API
- Jira
- JQL
- Llama 3.1
- Qwen
- Swagger

## Who should watch
- You are building GenAI prototypes and need a practical route from a browser-based experiment to a version-controlled application.
- Your AI feature calls business APIs and prompt changes keep breaking previously working requests.
- You work in DevOps or software engineering and want to understand how evals fit into CI/CD for LLM applications.

## Related talks

- [Building Effective Products with GenAI](https://mlopstalks.com/talks/building-effective-products-with-genai) (Faizaan Charania, LinkedIn, 53:14)
- [Holistic Evaluation of Generative AI Systems](https://mlopstalks.com/talks/holistic-evaluation-of-generative-ai-systems) (Jineet Doshi, Intuit, 56:04)
- [Extending AI: From Industry to Innovation](https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation) (Sophia Rowland & David Weik, SAS, 1:01:37)
- [GenAI in Production - Challenges and Trends](https://mlopstalks.com/talks/genai-in-production-challenges-and-trends) (Verena Weber, Verena Weber, 48:43)
- [Architecting Modern AI Systems: Platforms, Agents, and Integration](https://mlopstalks.com/talks/architecting-modern-ai-systems-platforms-agents-and-integration) (Allen Roush, BuzzHPC & Frédéric Bénard, Mila & Shuo Wang, Bell Canada, 57:00)