# No Rose Without a Thorn - Obstacles to Successful LLM Deployments

Tanmay Chopra, Neeva | LLMs in Production 2023 | 10:24

Source: https://www.youtube.com/watch?v=lnbx4dOW220
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/no-rose-without-a-thorn-obstacles-to-successful-llm-deployments
Published: 2023-05-24
Tags: build-vs-buy, deployment, latency, reliability

## TL;DR
- LLM production failures come from infrastructure problems such as latency, cost, API downtime, evaluation, and integration decisions.
- Generative output varies in format and content, so production systems need validation, fallbacks, reproducibility controls, and protection against prompt hijacking.
- The safest first deployments add value in non-critical, relatively high-latency workflows while teams build toward lower costs and stronger serving infrastructure.

## Summary
Tanmay Chopra argues that LLM demos often avoid the conditions that make production difficult. The main infrastructure problems are slow responses, the buy-versus-build decision, rising costs, unreliable provider APIs, and weak evaluation methods. The model output creates another set of problems because formats vary, identical inputs can produce different results, and users can manipulate prompts into producing harmful content. Chopra suggests streaming and interface design to make latency feel lower, buying access while collecting data for later in-house models, and using fallbacks across providers. Teams should fail safely around trust and safety, while using user feedback and product metrics to improve less risky features in production. For an initial deployment, he recommends a non-critical workflow with relatively high latency tolerance. He also advises planning to build while buying so API costs can remain manageable as adoption grows.

## Key ideas
### LLM production problems fall into infrastructure and output categories
[00:46](https://www.youtube.com/watch?v=lnbx4dOW220&t=46s)
Tanmay Chopra divides deployment obstacles into infrastructure problems and output problems. Infrastructure includes technical and integration issues such as slow responses, the choice between buying API access and building a model, provider downtime, and evaluation. Output problems come from the generated text itself. These include variable formats, different answers for the same input, and adversarial behavior such as prompt hijacking. The split matters because a team can solve a model-quality issue and still have a system that is too slow or unreliable to put in front of users.

### Latency changes what users will accept from an LLM feature
[01:24](https://www.youtube.com/watch?v=lnbx4dOW220&t=84s)
Chopra uses search as an example of a product with a fast status quo. Generating an LLM response takes longer, so a direct replacement can feel broken even when the answer is useful. Teams can make the model faster with distillation, pruning, or smaller models, especially when they build their own system. They can also make the product feel faster through interface choices, including loading animations, streamed output, and parallel work on tasks that do not block the main result.

### Buying first and building later can reduce early risk
[01:44](https://www.youtube.com/watch?v=lnbx4dOW220&t=104s)
Buying access to a foundational model has a smaller upfront cost and can get an MVP to market quickly, while building a model requires more initial investment and leaves uncertainty about whether quality will reach the required level. Chopra recommends buying while building. A team can validate the use case with an API, collect data, and fine-tune an in-house model over time. That approach addresses the risk that API costs become infeasible as adoption grows without forcing the team to solve every modeling and serving problem before launch.

### Provider fallbacks and graceful failure protect user trust
[05:33](https://www.youtube.com/watch?v=lnbx4dOW220&t=333s)
When a team depends on a foundation-model API, Chopra recommends a multi-cloud style of fallback across providers. He says he has not seen two major foundation providers fail together, so having alternatives can reduce the impact of a single outage. The product also needs a last-resort behavior for cases where no model returns an answer. Users may accept that the technology is still developing, but they need the application to fail in a controlled way rather than silently breaking or presenting a bad result.

### Evaluation should be strict around safety and fast around product learning
[06:15](https://www.youtube.com/watch?v=lnbx4dOW220&t=375s)
Chopra separates safety checks from ordinary product iteration. Trust and safety failures need fail-safes before launch, while teams can move faster on the core product and use strong user feedback loops to improve it. He recommends connecting an LLM integration to a top-line product measure, such as stay duration or session length, then checking how the feature changes that measure. This gives the team a way to judge whether the output is helping the product rather than relying only on manual quality review.

### Prompt examples, validation, and temperature control address output instability
[06:58](https://www.youtube.com/watch?v=lnbx4dOW220&t=418s)
Output format variability is one of the largest integration problems because downstream pipelines expect predictable structures. Chopra recommends giving output examples in the prompt and using guardrail libraries to validate the response, with another model call when needed. A simple fallback can handle invalid output. For reproducibility, he gives a direct fix: set the temperature to zero. Prompt hijacking belongs with trust and safety because an attacker can push the model toward outputs that violate the application's safety requirements.

### The first production use case should add value without becoming a dependency
[08:03](https://www.youtube.com/watch?v=lnbx4dOW220&t=483s)
Chopra recommends starting with a non-critical workflow where the LLM adds value but a failure does not stop the product. Relatively high-latency use cases are also better starting points because users already expect to wait longer. As serving infrastructure becomes more reliable, the team can take on more critical workflows. He says the first deployment should be planned so the system can remain in production, which means controlling future costs while using an API and considering how the model will scale later.

## Notable quotes
- "Why do such great MVPs and demos never make it to production?" (00:46)
- "So you can make models faster or you can make models seem faster." (04:13)
- "The first and probably most vital aspect to think about is project positioning." (08:03)
- "Do not underestimate the HCI component." (09:23)

## Tools & references mentioned
- Neeva
- TikTok
- Twitter
- guardrails

## Who should watch
- You are deciding whether an LLM feature belongs in a user-facing production workflow and need a practical way to choose the first use case.
- Your application depends on a model API and you need ideas for handling latency, provider downtime, cost growth, or invalid output.
- You are connecting generated text to a downstream pipeline and need simple controls for format validation, reproducibility, and prompt attacks.

## Related talks

- [The Confidence Checklist for LLMs in Production](https://mlopstalks.com/talks/the-confidence-checklist-for-llms-in-production) (Rohit Agarwal, portkey.ai, 32:34)
- [Building LLM Applications for Production](https://mlopstalks.com/talks/building-llm-applications-for-production) (Chip Huyen, Claypot AI, 35:23)
- [Pitfalls and Best Practices: 5 Lessons from LLMs in Production](https://mlopstalks.com/talks/pitfalls-and-best-practices-5-lessons-from-llms-in-production) (Raza Habib, Humanloop, 30:27)
- [LLMs in Production Conference - Part II](https://mlopstalks.com/talks/llms-in-production-conference-part-ii) (Matei Zaharia, Databricks and Stanford & Chip Huyen, 37:21)
- [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01)