# AI Data Engineers: Data Engineering After AI

Vikram Chennai, Ardent AI | MLOps Podcast | Episode 309 | 48:07
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Ne8yzzWKJGM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ai-data-engineers-data-engineering-after-ai
Published: 2025-04-25
Tags: agents, coding-agents, context-engineering, data-pipelines, human-in-the-loop

## TL;DR
- An AI data engineer connects to a company's existing pipelines, code repositories, and databases, then builds or changes data pipelines through code.
- Vikram Chennai uses context reduction, task planning, staging environments, and deterministic checks to limit agent mistakes before changes reach production.
- Ardent AI keeps a human approval step because current agents are useful for data engineering work but are still far from perfect.

## Summary
Vikram Chennai describes Ardent AI's AI data engineer as a coding agent connected to an organisation's existing data stack. It can inspect repositories, APIs, schemas, and databases, then create pipelines or schema changes and submit them as pull requests. The product focuses on data engineers because they understand the details of their company's tools and can review the agent's plan. Vikram explains that large enterprise environments create context problems, so the agent narrows what it retrieves while retaining a path to more information. It plans work in small steps, checks whether a task is possible, runs code in a staging environment, reads test results and error logs, and loops until the result works. He is cautious about full autonomy and wants users to approve changes. Ardent AI prices access through subscriptions with credits covering model and compute use. Vikram's strongest example is an agent coordinating Airflow and Databricks while writing Spark code that worked on its first attempt.

## Key ideas
### Data engineers are the first users because they understand the stack
[01:59](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=119s)
Vikram Chennai says data engineers are Ardent AI's primary users. They know which pipeline tools their organisation uses and understand the small details that can make a data task work or fail. That knowledge lets them direct an agent effectively and review what it produces. He says users have already used the system to do more work faster, while still relying on their understanding of the existing environment. The product is aimed at people who know how their pipelines, databases, and repositories fit together, rather than at replacing that knowledge with a generic assistant.

### The AI data engineer writes into existing systems instead of replacing them
[02:50](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=170s)
Vikram defines an AI data engineer as an agent connected to a company's stack that can build pipelines or perform schema migrations. In his example, a user asks for a pipeline, and the agent inspects an API, checks how its output is structured, looks at the organisation's repository, and writes the pipeline code. It can push the result to GitHub as a pull request. The agent also has database context, so it can see existing schemas and choose an appropriate table instead of inventing one. Ardent AI supports the tools and databases that companies already use, including Airflow, Dagster, Prefect, PostgreSQL, Snowflake, Databricks SQL, and MongoDB.

### Specialised context is needed when enterprise environments become large
[06:58](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=418s)
Vikram describes a company with 15,000 pipelines as an example of why an agent cannot simply receive the whole environment in its context window. Even if the window could hold everything, he says the agent would become confused about which information mattered. Ardent AI therefore tries to simplify the problem by retrieving relevant material and removing unnecessary context. Pipeline code is indexed so the agent can find a particular pipeline and its location in GitHub. Other information can be discovered later when needed. Vikram's approach is to provide enough context for achievable work while keeping irrelevant material out of the immediate task.

### Staging environments give the agent room to correct its own work
[08:36](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=516s)
Ardent AI is exploring lightweight, temporary staging environments for connected systems. If an agent proposes a database change, the change can be tested away from the actual database. The agent can inspect what happened, compare the result with the intended outcome, and try again without committing a mistake to production code or data. Vikram says this gives the system several attempts to reflect and correct itself. The validation can include existing data quality checks and tests, along with error logs. The agent can use those signals to identify bad syntax, hallucinated details, or output that does not match expectations.

### Planning checks whether a requested task is possible before execution
[12:25](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=745s)
Vikram gives a simple failure case: asking the agent to create a PostgreSQL table without providing PostgreSQL credentials. Early versions tried to continue anyway. Ardent AI added a check in the planning phase that asks whether the task is possible. The agent gathers context, searches the web when needed, and presents a step-by-step plan for approval. Users can correct a wrong table or an incorrect step before execution begins. If required information is missing, the agent should say that the task cannot be completed and ask the user to revise the request rather than fabricate access or context.

### Small planned steps give data engineers more control
[14:32](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=872s)
Vikram prefers breaking a large request into smaller operations. Instead of saying only that it will build a pipeline, the agent lays out the individual steps it plans to execute. This improves accuracy and lets data engineers review the details. A user can identify that one step is wrong, that a table was inferred incorrectly, or that the intended transformation is different. The design keeps some responsibility with the user, who can apply domain knowledge before the agent runs. The same plan also helps the agent work through complex tasks without having to solve the entire request as one opaque action.

### Evaluation works best when success can be checked in the data
[28:15](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=1695s)
Vikram agrees that data engineering provides unusually clear evaluation signals. Ardent AI's benchmark includes simple checks such as inserting a record into a PostgreSQL database or creating a table, then verifying that the expected object or data exists. The evaluation can add execution time and query speed. This gives the training process more information than asking a language model whether generated code looks correct. The system can check whether the data arrived where it should, whether the query ran too slowly, and whether the pipeline completed. Vikram says these deterministic outcomes help the agent improve across many tasks.

### Human approval remains part of the product's safety boundary
[25:05](https://www.youtube.com/watch?v=Ne8yzzWKJGM&t=1505s)
Ardent AI can run through an SDK or API when a pipeline fails, including an error handler that triggers an agent at 3 a.m. The agent can investigate, prepare a fix, and offer a pull request or a restart. Vikram says the user still gives the final approval. He compares this with hiring a junior developer and allowing them to push every change directly to production. Agents are useful, but he considers them far from perfect, so retaining control matters. He does not know whether full autonomy will ever be appropriate for every data engineering change.

## Notable quotes
- Vikram Chennai: "The fundamental way it works is it is a coding agent and what our thesis is that seems to be working is that coding is a language that we've already created to interact with every service out there." (43:33)
- Vikram Chennai: "What you want to do, at least for data engineering agents or I think coding agents in general, is give it as much as possible because we have this policy of achievable outcomes." (17:23)
- Vikram Chennai: "I honestly don't know if we'll ever get to full autonomy." (25:25)
- Vikram Chennai: "We saw the data end up where it needs to go. We saw how long it took. We saw how fast the actual query ran." (29:43)

## Tools & references mentioned
- Ardent AI
- Airflow
- GitHub
- Dagster
- Prefect
- PostgreSQL
- Snowflake
- Databricks SQL
- MongoDB
- Spark
- Databricks
- Slack
- Jira
- Uber prompt engineering toolkit
- Notion AI

## Who should watch
- You are a data engineer considering an agent that can work inside an existing pipeline and database stack without a migration.
- Your team is building an agent for data or coding tasks and needs practical ideas for context selection, planning, staging, and evaluation.
- You are assessing how much autonomy to allow an AI system when a bad change could affect production data or pipeline operations.

## Editor's note

Vikram Chennai describes an AI data engineer that writes pipelines into a company's existing systems, including Airflow and Databricks. ZenML lets teams write workflows as Python pipelines and choose the orchestrator and other infrastructure through configuration, so the same pipeline code can run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Hundreds of Users Love Our Data Analyst AI Agent](https://mlopstalks.com/talks/hundreds-of-users-love-our-data-analyst-ai-agent) (Ioannis Zempekakis & Donné Stevenson, 29:00)
- [Building Artificial Engineering Intelligence for Engineering Teams](https://mlopstalks.com/talks/building-artificial-engineering-intelligence-for-engineering-teams) (Aishwarya Shankar, Intelligence AI, 10:10)
- [Co-Engineering: The New Era of Human-AI Collaboration](https://mlopstalks.com/talks/co-engineering-the-new-era-of-human-ai-collaboration) (Kiriti Badam, OpenAI, 29:25)
- [How AI Agents Are Transforming Data Analytics](https://mlopstalks.com/talks/how-ai-agents-are-transforming-data-analytics) (Ines Chami, Numbers Station, 24:26)
- [Before Building AI Agents Watch These Hard Earned Lessons](https://mlopstalks.com/talks/before-building-ai-agents-watch-these-hard-earned-lessons) (Nishikant Dhanuka, Prosus Group, 52:37)