# Impact of LLMs on the Tech Stack and Product Development

Anand Das, Bito | MLOps Podcast | Episode 188 | 55:31
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=IafvCmYZ0eA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/impact-of-llms-on-the-tech-stack-and-product-development
Published: 2023-11-07
Tags: evals, guardrails, inference, prompt-engineering

## TL;DR
- Bito moved from a collaboration tool to an AI coding assistant after seeing strong adoption of a simple Chrome extension that explained code.
- Bito routes requests across multiple model providers and accounts because rate limits, context sizes, latency, availability, and answer quality vary by model.
- Useful guardrails start with better context and prompts, then add deterministic tools, human feedback, and pre-verified tests where possible.

## Summary
Anand Das describes how Bito moved from an IDE collaboration product to an AI coding assistant. The change began with a simple Chrome extension that explained code, then expanded to Visual Studio Code and JetBrains with code generation, explanations, and test writing. At scale, Bito cannot rely on one model provider. It uses several APIs and accounts, chooses models based on context size and task requirements, and fails over when a provider reaches its limits or becomes unavailable. Anand is candid about the operational work involved, including separate prompts for different models, cost and latency management, and graceful degradation. Bito keeps code indexes locally in the IDE and currently uses a homegrown in-memory vector layer. For hallucinations, the team reviews user feedback, updates prompts and rules, and runs tests against pre-crafted answers. Anand recommends giving coding assistants detailed context, especially the relevant code, exception, and runtime conditions.

## Key ideas
### Bito found product demand by launching a very small coding assistant
[08:42](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=522s)
Bito began as a collaboration tool that put Slack and Google Docs ideas into the IDE, but Anand Das says users did not adopt it. Developers already copied code into Slack even though it was hard to find later. The team then heard repeated requests for help onboarding engineers and explaining unfamiliar code. When GPT-3.5 became available, Bito pivoted toward generative AI and launched a Chrome extension with one option, "explain code." Rather than asking people whether they might use it, the team released it and watched downloads and usage. The response showed there was demand. Bito later added Visual Studio Code and JetBrains extensions for code generation, explanations, and test cases.

### Smaller or local models could reduce cost and protect enterprise code
[11:00](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=660s)
Anand is experimenting with open-source models and ways to provide them with context instead of immediately training them. He sees two reasons to pursue this. A smaller model may answer questions about a specific codebase at lower cost, and enterprises are concerned about sending intellectual property to external LLM providers. Local hosting is possible with some models, but Anand says the response time is often not good enough for users. He is also learning about fine-tuning, although Bito was not using it at the time of the conversation. The goal is to understand whether a locally hosted model can become useful without the cost and complexity of operating a large GPU fleet.

### Bito routes requests across providers and accounts to avoid rate limits
[12:59](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=779s)
Bito uses several commercial APIs because a single provider cannot support its usage reliably. Anand names OpenAI, Anthropic, Azure, Amazon Bedrock, and Google as available options. The team began hitting tokens-per-minute and requests-per-minute limits as usage grew, so it built a request multiplexer. For each request, the system checks available limits across accounts, chooses a suitable account, manages the resulting costs, and fails over when a model or provider is unavailable. The fallback path can move between direct and Azure-hosted APIs, then to Anthropic models. This work is necessary for a generally available product, even though a basic LLM integration may look simple.

### Model choice depends on context size and the kind of answer required
[19:50](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=1190s)
Anand says cost is a lower priority than giving the user a useful answer. The first decision is whether the input and chat history fit within a model's context window. The second is whether the model can handle the operation. GPT-4 may be better for reasoning, while GPT-3.5 can produce unwanted explanatory text when Bito needs a command that can be parsed and run. Code indexing can create a large context, which may rule out a model with a smaller context window. When the preferred model is unavailable, Bito degrades functionality, such as reducing the amount of context, rather than abandoning the request. Latency also varies widely with provider load.

### Bito keeps code indexes on the user's machine
[25:52](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=1552s)
For its feature that understands a user's code, Bito currently uses a homegrown in-memory vector layer rather than a hosted vector database. The IDE extension indexes code locally, and the index file remains on the user's machine. Anand says the layer has operations similar to a system such as Pinecone, including insert and upsert. The design avoids moving source code to Bito's servers, but it has limits. Web views can impose memory restrictions, and the index may be several times larger than the source code on disk. Bito does not support repositories above the size limit it had chosen at that point. The team expected to move toward a locally installable vector database library.

### Guardrails combine prompt changes with deterministic tools
[29:04](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=1744s)
Anand says Bito still hallucinates and was not using a dedicated guardrail tool. The team studies thumbs-up and thumbs-down feedback, reviews the indexed context and answer, then changes prompts or adds rules. He gives security analysis as an example of where an LLM alone is insufficient. A deterministic open-source vulnerability scanner can first identify vulnerabilities. Its output can then be passed to the LLM with a request to locate and fix the issues. This gives the model verified information to work from instead of asking it to invent a security assessment from its training data. Anand calls hallucination a necessary downside when users want help generating new ideas, but existing code needs stronger verification.

### Prompt evaluation needs pre-crafted answers because AI cannot reliably grade itself
[36:25](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=2185s)
Bito has a homegrown prompt-testing process. The team collects user feedback, creates test scenarios, and builds a set of possible answers that people have checked before. It then runs prompts against those cases. Anand says the hard part is verification, since a human reviewer may lack the required programming experience and an LLM may hallucinate while judging another LLM's output. Bito therefore avoids depending on an AI-generated answer as the only test oracle. Anand expects open-source and commercial tools to cover more of these common problems over time, but says the available solutions were not yet ready for daily production use.

### Each model needs its own prompt repository
[40:46](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=2446s)
Anand says Bito maintains separate prompts for different LLMs. GPT-4 handles structured prompts well, including JSON or YAML-style rules, while GPT-3.5 may need explicit markers and application logic to remove unwanted text. A prompt that works with GPT-4 may need changes for Anthropic, and provider model updates can change expected outputs. The team tests prompts across versions and adds rules when behavior changes. Anand's advice for smaller systems is to choose one model and become comfortable with it. Adding models brings extra decisions about prompts, cost, latency, context, and fallback behavior, so the operational benefit should justify that complexity.

### A coding assistant works better when the developer supplies concrete context
[51:56](https://www.youtube.com/watch?v=IafvCmYZ0eA&t=3116s)
For debugging, Anand recommends treating an LLM as a source of useful general knowledge that does not know the user's exact situation. A developer should provide the exception, the relevant file and code, when the error occurs, and any other conditions that narrow the problem. An assistant can often explain a common exception from the error text alone, but application-specific failures need the code and runtime context. Anand also says experienced developers tend to get better results because they can describe the situation and ask a precise question. A person still needs enough understanding to judge whether the answer fits a complicated system.

## Notable quotes
- Anand Das: "We started with a very simple thing to test out, a simple Chrome extension with a single menu option, explain code." (09:03)
- Anand Das: "If your scale is not huge, pick one model and get comfortable with it." (43:59)
- Anand Das: "As much context as you can provide, I think context is very important." (52:18)
- Anand Das: "We don't want to have the same thing happen with our testing process." (37:49)

## Tools & references mentioned
- Bito
- OpenAI
- GPT-3.5
- GPT-4
- Anthropic
- Azure
- Amazon Bedrock
- Google
- Visual Studio Code
- JetBrains
- Pinecone
- Weights & Biases
- Chrome
- Slack
- Google Docs
- GPT-2
- open-source vulnerability tools
- MLOps Community

## Who should watch
- You are building an LLM product and keep running into provider rate limits, context-window constraints, or unpredictable latency.
- Your team is deciding whether to use hosted APIs or operate its own models and GPUs.
- You want practical ways to improve coding-assistant answers, test prompts, and reduce hallucinations.

## Related talks

- [DevTools for Language Models: Unlocking the Future of AI-Driven Applications](https://mlopstalks.com/talks/devtools-for-language-models-unlocking-the-future-of-ai-driven-applications) (Diego Oppenheimer, Factory, 29:55)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)
- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [From Building Self-driving Cars to Building LLM Applications](https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications) (Effy Zhang, Baserun, 10:45)
- [Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security](https://mlopstalks.com/talks/guarding-llm-and-nlp-apis-a-trailblazing-odyssey-for-enhanced-security) (Ads Dawson, Cohere, 59:41)