LLM API uptime, latency, deprecations, and rising prompt costs make production applications hard to operate.
2
Short prompts with useful context, versioned prompt management, caching, and vector databases improved Speaker Scribe's outputs and costs.
3
Useful LLM products need easier access, domain-specific context, and services that extend what ChatGPT can do.
Summary
Soham Chatterjee describes six months of building Speaker Scribe and the problems that appeared when an LLM product moved beyond a proof of concept. External APIs had inconsistent latency, weak commitments around uptime, sudden endpoint deprecations, and costs that grew with prompt size and product complexity. Prompt maintenance was also fragile. Small changes in punctuation could break a prompt, while hallucinations, bias, incorrect data, and repetitive outputs made evaluation and delivery difficult. His practical responses were to add context with few-shot prompting, save and version prompts, cache previous outputs, and use vector databases to retrieve relevant past talks and workshops. He recommends keeping chains short and avoiding agents because they were unreliable and hard to reproduce in his experience. For product design, he argues that applications need to reduce the effort of using an LLM, add domain knowledge, and provide services beyond the raw model output.
External LLM APIs create operational and cost problems
Soham says API providers did not give reliable commitments around uptime and latency. Delays of only a few seconds could interrupt the flow of a creative application. His team also had to adapt when OpenAI deprecated the DaVinci 002 endpoint, because prompts that worked well on the old endpoint needed substantial adjustment. As applications became more complex, longer prompts drove API costs up quickly. He says this can push teams toward fine-tuning or training their own models.
Prompt behavior is fragile and difficult to evaluate
Soham describes prompt engineering as an inexact process. In his application, apostrophes, commas, and other punctuation could break prompts. Hallucinations were difficult to spot, and there were few useful evaluation metrics for checking outputs. That made it hard to trust results enough to serve them to clients. He also had to manage outputs that were insufficiently creative, along with bias and incorrect data.
The approach that worked best for Soham was giving the LLM more context. He uses few-shot prompting and passes the previous output when a user asks Speaker Scribe to regenerate a proposal. This gives the model information about what came before and can make the next result more creative and useful. He cautions that more elaborate prompting techniques can make prompts and outputs much larger, which increases API costs.
Soham treats a good prompt as intellectual property. He recommends saving and protecting prompts, then versioning them so a team can see how changes affect output. He suggests that this practice could lead to a form of prompt operations. The point is to manage prompts as production assets rather than leaving them as untracked text inside application code.
Vector databases help with retrieval, caching, and cost
Soham describes vector databases as a way to provide context while reducing cost. Caching previous outputs can avoid repeated API calls, which saves money and improves latency. Retrieval can also find relevant material to include in a new prompt and reduce hallucinations. For Speaker Scribe, his team stored past talks and workshop submissions, then retrieved related examples when generating a new proposal.
As LLM products become more complex, Soham says long chains become a problem. Chains can fail, and longer chains add cost, latency, and application complexity. His team's solution was to keep chains short. He also advises against agents based on his experience, because he found them unreliable and difficult to reproduce.
LLM products need a reason to exist beyond ChatGPT
Soham says a product must make it easier for people to access an LLM, add domain knowledge, or provide services around the model's output. Grammarly Go is an example of reducing the effort required to use an LLM. Speaker Scribe adds knowledge from previous talks, while Sleek can use customer data as context. He also points to deck creation as an opportunity for additional services, such as producing images that ChatGPT did not provide at the time.
