LLM features can reach users quickly, but prompt changes, retrieval changes, and unique request context make them difficult to maintain reliably.
2
Honeycomb captured the full path of its natural-language query feature, including user context, prompts, model responses, retrieval decisions, validation failures, and feedback.
3
Daily deployment and analysis of end-to-end traces creates a feedback loop for improving prompts, fine-tuning models, and evaluating user experience.
Summary
Phillip Carter argues that observability has to be part of the engineering toolkit for reliable LLM features. A team can ship an initial product quickly, but traditional unit tests, debugging, and regression tests do not cover the changing behavior caused by prompts, retrieval pipelines, and request-specific context. Carter describes Honeycomb's natural-language query feature, which initially handled about 80% of use cases. The remaining cases were difficult because fixing one could break another. Honeycomb instrumented the whole feature, including user and team IDs, complete prompts and model responses, roughly 40 retrieval decisions, parsing and validation steps, failures, and user feedback. Distributed traces connected the application, OpenAI, and the query engine across 48 spans. The team used those traces to isolate failures, change prompts, deploy daily, and compare success rates over the previous 24 hours. Carter also points to OpenTelemetry semantic conventions and automatic instrumentation as work intended to make this easier.
LLM features are easy to ship before they are easy to maintain
Carter says a product engineering team can often put an LLM feature in front of users in about a month with OpenAI API keys. The first release may solve real problems and create a strong marketing moment, but users soon expect more. Teams then need to change the system without breaking behavior that already works. That is difficult because the usual software tools, including unit tests, IDE debugging, and regression tests, are hard or impossible to apply to many LLM tasks.
Small changes can produce large and hidden regressions
Very small prompt changes can greatly alter an LLM's behavior, according to Carter. The same applies to changes in a retrieval-augmented generation pipeline or to the other context attached to each request. A team may introduce a regression in behavior it did not realize was working. This makes reliability harder as more components are added around the model, since each component can affect the final response.
Honeycomb's first natural-language query release left a difficult long tail
Honeycomb released a natural-language interface for querying its observability product around May 3. Carter says the first version handled about 80% of use cases. The remaining 20% contained a long tail of cases that paying customers cared about. Fixing one case could break another, and the team had few established tools or practices for managing that work. The feature still improved product retention, conversion to paid tiers, and the sales process because users could type a request and receive a query that filled out the interface.
The team instrumented the entire feature rather than only the model call
Honeycomb captured user IDs, team IDs, the full text sent to OpenAI, every decision in a retrieval pipeline of about 40 steps, complete model responses, parsed data structures, validation results, every failure path, and user feedback. This covered the path from clicking 'get query' to executing the query and displaying the result. Carter's point is that useful observability needs the application context around the model, not only latency, token usage, and errors.
Distributed traces connect the user experience across systems
The feature crossed Honeycomb, OpenAI, and a separate querying engine, so Carter's team used distributed tracing to connect those systems. The relevant trace contained 48 spans, with the retrieval pipeline making up much of the trace. More than 20 spans could occur before the application sent a request to OpenAI. Seeing the full sequence let the team understand what happened before a failed response and use that information in prompt engineering or model fine-tuning.
Queries over failure dimensions isolate specific problems
Carter describes monitoring the complete end-to-end experience instead of only model latency and errors. The team could split the data by fields and values, find a specific failure such as a model response that did not contain valid JSON, and inspect the request that produced it. Grouping failures by user input and model response also showed where the system failed to produce a query. This turned a broad reliability problem into cases the team could investigate.
Daily deployment creates a measurable improvement loop
The traces gave Honeycomb a feedback loop. After changing prompt engineering, the team deployed the fix and examined the previous 24 hours of behavior. They asked whether the success or failure rate for the complete user experience went up or down. Carter presents repeating this process, with daily deployment and review, as the way to improve LLM reliability over time.
Telemetry can feed prompting, fine-tuning, and evaluations
Carter says teams can start with OpenTelemetry, instrument their applications, and use an observability product to track real-world behavior. The resulting data can influence prompt engineering, provide training data for fine-tuning, and supply data for evaluations. He also says the OpenTelemetry community is developing semantic conventions for AI and vector database components, automatic instrumentation, and guidance for capturing application-specific context.
