# All the Hard Stuff with LLMs in Product Development

Phillip Carter, Honeycomb | MLOps Podcast | Episode 170 | 1:01:04
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=DZgXln3v85s
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/all-the-hard-stuff-with-llms-in-product-development
Published: 2023-08-11
Tags: evals, product-strategy, prompt-engineering, security

## TL;DR
- Phillip Carter's team shipped a natural-language query assistant for Honeycomb in one month, then improved it by studying real user inputs and outputs in production.
- The team treated security as a product constraint by parsing model outputs, sandboxing access, limiting requests, restricting input size, and testing prompt injection attacks.
- The assistant's return on investment was assessed through leading indicators such as complex queries and saved queries on boards, alongside feedback from the sales team.

## Summary
Phillip Carter describes how Honeycomb built and shipped a natural-language query assistant for its observability product. The assistant turns user requests into a JSON query specification, using dataset definitions and schema context to choose the right fields. Honeycomb released an initial version quickly, then improved it by examining real inputs, model outputs, parsing errors, execution results, and user feedback. A prompt change raised the query success rate from roughly 76 or 77 percent to about 94 percent, with later results approaching 95 percent. Carter also explains why the team rejected a chatbot interface for the first release. A focused UI made it easier for users to take over and edit queries, while reducing the attack surface. He gives practical security measures for prompt injection and data access. For ROI, the team tracked whether users created complex queries or saved queries to boards, because those actions were linked to later activation and payment. Carter argues that ML engineers should talk directly with users and work with product and sales teams to define measurable outcomes.

## Key ideas
### Honeycomb used an LLM to lower the barrier to querying observability data
[05:43](https://www.youtube.com/watch?v=DZgXln3v85s&t=343s)
Honeycomb had strong product-market fit with SREs and platform engineers, while ordinary software developers and product managers often struggled with its active querying model. Users could describe what they wanted in natural language, so Phillip Carter's team built a system that converted those requests into Honeycomb's JSON query specification. The team moved quickly after ChatGPT arrived and API prices dropped by two orders of magnitude. They gave themselves one month to ship a live version to everyone, accepting that it would have missing features. Carter says the point was to release a real product rather than a carefully controlled early-access demo.

### Real production inputs were more useful than speculative prompt experiments
[12:00](https://www.youtube.com/watch?v=DZgXln3v85s&t=720s)
After the first release, the team spent another month iterating on production behavior. They reviewed user inputs, LLM outputs, parsing and validation results, query execution outcomes, and feedback marked yes, no, or unsure. This exposed patterns that were hard to predict in advance. Users entered short terms such as "errors" and "traces," pasted trace identifiers, and even inserted large expressions from Honeycomb's derived-column language. Carter's team grouped failures by frequency, found recurring mistakes such as incorrect answers to requests about error rates, and adjusted prompts or applied static corrections. This approach raised the assistant's success rate by about six percent in one iteration.

### Structured output controls improve consistency without solving every ambiguity
[07:25](https://www.youtube.com/watch?v=DZgXln3v85s&t=445s)
Honeycomb queries are JSON objects with specific rules, rather than SQL or PromQL. The assistant receives context such as dataset definitions, including the canonical field used to represent errors. That context helps it choose between, for example, a Boolean error field and a string error-message field. Carter says function calling can help models conform to a requested shape, but it created a problem for ambiguous inputs. The system often produced something usable for a pasted identifier or unusual request without function calling, while function calling more often produced nothing because the input did not fit the required schema. Prompt engineering remained dependent on the particular task.

### A focused query interface was safer and more useful than a general chatbot
[24:09](https://www.youtube.com/watch?v=DZgXln3v85s&t=1449s)
Honeycomb's early prototype was a chatbot with broader ambitions. It could answer general observability questions, create queries, and describe hypothetical queries for data that did not yet exist. The team cut that scope after seeing the complexity and hearing from sales that users mainly wanted a Honeycomb query quickly. The shipped assistant filled the existing query UI, where users could inspect the filters and group-bys and continue editing directly. Carter also saw chat as an end-user-reprogrammable system. Textual back-and-forth gave attackers more chances to manipulate the system, request customer data, or attempt data exfiltration.

### Security required several layers because prompt injection was easy to attempt
[28:53](https://www.youtube.com/watch?v=DZgXln3v85s&t=1733s)
The team asked a former Honeycomb employee to attack the feature, and he immediately tried requesting data associated with known customers. Honeycomb also saw suspicious values such as script tags in incoming attributes. Carter describes several protections: parse every model output, apply protections against SQL injection and other unsafe query behavior, keep the model away from the main database and systems that handle user and team information, rate-limit users, limit input size, and parameterize the prompt. The initial rate limit was 25 uses per day, although it later changed. These controls did not make attacks impossible, but they made repeated experimentation difficult enough to reduce the appeal.

### The team estimated API cost before deciding whether free usage was viable
[35:06](https://www.youtube.com/watch?v=DZgXln3v85s&t=2106s)
The assistant was free, including in Honeycomb's unauthenticated sandbox. Carter explains that a successful feature creates both token costs and rate-limit pressure, so the team worked to reduce prompt and input size. For a rough forecast, he used the monthly volume of manual queries and assumed a similar volume of natural-language queries. That estimate put the OpenAI bill at about $100,000 per year with GPT-3.5. He considered that affordable compared with the cost of an engineer or a conference sponsorship. GPT-4 would have pushed the cost into the tens of millions per year, which changed the model choice. The model had to fit the product's cost and accuracy constraints.

### Leading indicators gave Honeycomb a way to judge the assistant before revenue arrived
[39:52](https://www.youtube.com/watch?v=DZgXln3v85s&t=2392s)
Carter's team did not have direct revenue attribution after only two months, so it tracked actions associated with later activation. A complex query included at least one group-by and one filter. A board let users save a query in a collection of executable queries. Among new teams, the share that created a complex query was more than twice as high for teams that used the assistant than for those that did not, rising from about 15 percent to 36 percent. Adding a query to a board rose from about 6 percent to almost 17 percent. Feedback from sales also indicated that prospects needed less help getting started. These signals supported making the assistant more visible.

### ML engineers can contribute to product decisions by speaking with users
[47:36](https://www.youtube.com/watch?v=DZgXln3v85s&t=2856s)
Carter says teams should begin with a user problem and then decide whether an LLM can help, rather than search for a problem for the technology. ML engineers can interview free-tier users, ask why they signed up, and learn what they want from the product. After five or six short interviews, recurring needs often appear. Product managers with growth experience can help identify what is already measurable and what needs instrumentation. Carter also wants product managers to understand concepts such as embeddings and model failure modes. ML engineers need to understand user problems and business measures, while product managers need enough data knowledge to work with the available systems.

## Notable quotes
- Phillip Carter: "You should be looking for a problem to solve and seeing if the technology can be used to solve it." (47:36)
- Phillip Carter: "The fastest thing you can do is just literally look at the inputs and outputs." (12:00)
- Phillip Carter: "Chat is an end user reprogrammable system where they can use that rapid feedback of textual input and output to manipulate the system behind the scenes." (27:07)
- Phillip Carter: "If your feature is successful you're going to hit rate limiting, if it's successful it's also going to cost a lot." (36:10)
- Phillip Carter: "Every product person knows the first thing you release is going to kind of suck in some ways." (52:12)

## Tools & references mentioned
- Honeycomb
- OpenTelemetry
- ChatGPT
- OpenAI
- GPT-3.5
- GPT-4
- LLMs
- SREs
- platform engineers
- product managers
- sales team
- service level objectives
- derived columns
- embeddings
- prompt injection
- SQL injection

## Who should watch
- You are building an LLM feature and need a practical way to inspect failures after launch instead of tuning prompts only from theory.
- Your product sends user data or customer data through an LLM and you need concrete ideas for limiting prompt injection and unintended access.
- You work in ML engineering and want to connect model work with user interviews, product measures, sales feedback, and activation signals.

## Related talks

- [Shipping LLMs: Buckle Up & Enjoy the Ride](https://mlopstalks.com/talks/shipping-llms-buckle-up-enjoy-the-ride) (Rex Harris, 12:51)
- [Using LLMs to Punch Above Your Weight!](https://mlopstalks.com/talks/using-llms-to-punch-above-your-weight) (Cameron Feenstra, Anzen, 35:49)
- [Product Engineering for LLMs](https://mlopstalks.com/talks/product-engineering-for-llms) (Charles Frye, Full Stack Deep Learning & Sahar Mor, Stripe & Sarah Guo, Conviction & Shyamala Prayaga, NVIDIA & Willem Pienaar, 31:46)
- [Building Products](https://mlopstalks.com/talks/building-products) (Sam Charrington, TWIML AI Podcast & George Mathew, Insight Partners & Asmitha Rathis, PromptOps & Natalia Burina, Meta & Sahar Mor, Stripe, 45:18)
- [Building Defensible Products with LLMs](https://mlopstalks.com/talks/building-defensible-products-with-llms) (Raza Habib, Humanloop, 24:10)