# What is the role of ML Engineers in the time of GPT4 and BARD?

Hannes Hapke, Digits | LLMs in Production 2023 | 22:31

Source: https://www.youtube.com/watch?v=l59AS0lK5rA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/what-is-the-role-of-ml-engineers-in-the-time-of-gpt4-and-bard
Published: 2023-05-05
Tags: build-vs-buy, careers, privacy, prompt-engineering

## TL;DR
- Hannes Hapke argues that public APIs will commoditize machine learning projects built on public data, with no special security, latency, or deployment requirements.
- Machine learning engineers will spend more time advising stakeholders about privacy, bias, safety, cost, and the trade-offs between internal models and third-party APIs.
- In-house work will focus on proprietary and subjective data, strict requirements, low latency, model operations, and the parts of a business that contain valuable intellectual property.

## Summary
Hannes Hapke compares the arrival of large language model APIs with the printing press. APIs make machine learning available to domain experts, but they also reduce the need to build some standard models internally. Projects based on public data, with no special security or latency needs, are most exposed. Hapke argues that proprietary data, subjective tasks, privacy requirements, device deployment, and low-latency systems remain strong reasons to build in-house. He also describes the limits of APIs: outputs can change between identical requests, hallucinated values can look convincing, and users usually cannot inspect the training data or add corrections to the provider's training loop. The ML engineer's work therefore shifts toward stakeholder advice, API evaluation, safety, bias, privacy, model deployment, and integration. Prompt design becomes one part of the job, alongside large-model serving, tokenizer behavior, distribution across GPUs, and latency reduction.

## Key ideas
### Large language model APIs can remove the implementation time from simple ML projects
[00:03](https://www.youtube.com/watch?v=l59AS0lK5rA&t=3s)
Hannes Hapke contrasts building an address parser with deep learning against sending the same request to an API such as OpenAI. An internal model might take a week or two, depending on the project and the available data. An API can return a useful answer with almost no ML implementation work, although rate limits, monthly costs, and availability create practical problems. Early demos also made it possible to generate HTML and CSS from a website description, which raised questions about what would happen to the wider ML engineering ecosystem.

### The printing press is a useful analogy for the democratization of machine learning
[02:31](https://www.youtube.com/watch?v=l59AS0lK5rA&t=151s)
Hapke calls the current shift a Gutenberg moment. Before the printing press, monks copied books by hand. Printing made books available more quickly and contributed to wider literacy across Europe. In his analogy, the ML engineer is similar to the person who previously produced books by hand, while model APIs are the modern printing press. Domain experts can now apply data to an API without first becoming machine learning specialists. The change also brings public discussion of AI and machine learning to people who previously struggled to understand what ML engineers did.

### API results can look useful while remaining inconsistent or wrong
[06:06](https://www.youtube.com/watch?v=l59AS0lK5rA&t=366s)
Hapke warns that impressive first results can be misleading. In an address-parsing example, repeated requests with the same prompt returned the same values but changed the keys in the JSON structure. Other tasks produced well-formed data with hallucinated values. Biographies and descriptions of people could sound convincing while being entirely incorrect. These failures make it unsafe to judge an API only by its first demonstration or by whether its output has a plausible structure.

### Public data and ordinary requirements make a project vulnerable to commoditization
[08:38](https://www.youtube.com/watch?v=l59AS0lK5rA&t=518s)
Projects built on public data have a high chance of being absorbed into the data used to build general APIs. Projects without special requirements are also exposed. If stakeholders do not need on-device execution, strong security, or another specific environment, a public API may replace a custom model. Hapke says this does not end machine learning research, since many problems remain unresolved, but it does change which applied projects are worth building internally.

### Proprietary data and strict system requirements support in-house ML
[09:46](https://www.youtube.com/watch?v=l59AS0lK5rA&t=586s)
Hapke gives several reasons to keep a project inside the company. Digits treats data security as a requirement and does not send its data to third parties, so it cannot use OpenAI's APIs for that work. On-device systems, data that cannot be shared, low-latency applications, and unstable third-party availability also favor internal models. Highly specific domain models can outperform generalized APIs because they focus on the task rather than producing unrelated outputs such as cocktail recipes.

### General APIs weaken several established ML engineering goals
[12:29](https://www.youtube.com/watch?v=l59AS0lK5rA&t=749s)
Traditional ML projects aimed to reduce bias in the data, document model and data limits, collect misclassifications for later training, protect user privacy, and run smaller models on devices. Hapke says API users often lack information about the training data and cannot feed an incorrect GPT-4 response back into their own training process. A smaller earlier model such as T5 could sometimes be fine-tuned for a focused task, which restores more control over feedback and behavior.

### The ML engineer becomes an adviser between technical and business stakeholders
[15:53](https://www.youtube.com/watch?v=l59AS0lK5rA&t=953s)
Hapke says the role shifts from mainly developing models and infrastructure to moderating decisions between stakeholders. CEOs, CTOs, security teams, and other business groups need help understanding the benefits and costs of internal development versus third-party APIs. After GPT-4 launched, his team had extended discussions with security colleagues about what data would be shared, whether sharing could be avoided, and what would be gained by keeping the system in-house. ML engineers also advise on privacy, safety, and bias.

### ML engineering remains necessary around prompts, serving, integration, and safety
[17:25](https://www.youtube.com/watch?v=l59AS0lK5rA&t=1045s)
Hapke prefers the term prompt design to prompt engineering because the work is still iterative and lacks a fixed process. He expects it to become one part of ML work rather than replace the profession. Serving models with billions of parameters, distributing them across instances or GPUs, and reducing latency for real-time systems remain difficult. Engineers also need to understand tokenizers, model sensitivity, carbon costs, bias, and safety before handing API keys to other teams.

### Future in-house projects should use proprietary or subjective data
[19:58](https://www.youtube.com/watch?v=l59AS0lK5rA&t=1198s)
Hapke recommends focusing on proprietary datasets, subjective machine learning, and specific system requirements. His team works on accounting and bookkeeping questions where the right classification can differ between users, so a single global model is not sufficient. Recommendation systems are another example where sending a full shopping history to an external API could be too expensive. Plain sentiment classification or basic cat-and-dog detection is more likely to be handled by an API.

## Notable quotes
- "Machine learning got democratized." (04:05)
- "The first result can be misleading." (07:01)
- "Our role has drastically shifted from developing machine learning models or creating the infrastructure of those machine learning systems to being the moderator between stakeholders." (15:53)
- "Prompt design, I'm not calling this prompt engineering because right now there's too much guessing in this game." (17:25)
- "Focus on projects with proprietary data." (19:58)

## Tools & references mentioned
- GPT-4
- Bard
- OpenAI
- Claude
- Digits
- T5
- Google
- New York Times
- Gutenberg
- model cards
- data cards
- federated learning
- encrypted machine learning
- Stable Diffusion

## Who should watch
- You are deciding whether to build an ML system internally or call a general-purpose model API.
- Your team needs to assess API privacy, bias, latency, availability, or data-sharing risks before production use.
- You work on ML infrastructure or stakeholder planning and want to understand how general APIs change the work that remains valuable.

## Related talks

- [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg, 50:12)
- [Extending AI: From Industry to Innovation](https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation) (Sophia Rowland & David Weik, SAS, 1:01:37)
- [MLOps at the Crossroads](https://mlopstalks.com/talks/mlops-at-the-crossroads) (Patrick Barker, Kentauros AI & Farhood Etaati, AIMedic, 49:02)
- [MLOps at the Age of Generative AI](https://mlopstalks.com/talks/mlops-at-the-age-of-generative-ai) (Barak Turovsky, Scale Venture Partners, 56:56)
- [A Playground for AI Engineers](https://mlopstalks.com/talks/a-playground-for-ai-engineers) (Paulo Vasconcellos, Hotmart, 54:42)