# Using LLMs to Power Consumer Search at Scale

Aravind Srinivas, Perplexity AI | LLMs in Production 2023 | 37:04

Source: https://www.youtube.com/watch?v=HzGiVzYbf2I
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/using-llms-to-power-consumer-search-at-scale
Published: 2023-08-16
Tags: evals, prompt-engineering, search, tool-use

## TL;DR
- Perplexity combines language models with search, databases, and other tools so answers can use current web information while retaining conversational context.
- Human control is central to Perplexity's search experience, through clarifying questions, follow-up prompts, personalization, and shared search threads.
- Building a reliable answer engine requires orchestration, evaluation, prompt versioning, query reformulation, model selection, and latency work across the entire stack.

## Summary
Aravind Srinivas introduces Perplexity as a conversational search engine built to answer questions with citations and current web information. He explains how Copilot asks clarifying questions, reformulates an ambiguous request into several searches, gathers a wider set of pages, and lets people continue with contextual follow-ups. Personal profiles provide global instructions for language, interests, and other preferences. Srinivas argues that language models become much more useful when connected to search indexes, databases, calculators, and other tools. He describes the engineering behind this system, including several specialized model calls, orchestration, parallel work, prompt updates, evaluations, and latency optimization. Perplexity initially took five to seven seconds per query, so the team controlled more of the stack to make it faster. He is direct about the limits of medical answers and recommends consulting a practitioner rather than acting on the system blindly.

## Key ideas
### Perplexity is designed as a conversational answer engine with citations
[00:42](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=42s)
Srinivas describes Perplexity as a conversational search engine for answering questions and browsing the web. Its aim is to give people access to useful knowledge without waiting for a specialist to respond. The product covers questions ranging from relativity and economics to trip planning and learning a new skill. Answers include citations because Srinivas wants truth to be visible in the response. The company describes its mission as being obsessed with knowledge and productivity, with the goal of helping people recover time and learn more.

### Copilot keeps people involved by asking clarifying questions
[04:52](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=292s)
Perplexity's Copilot uses language models to ask clarifying questions after examining a request. This matters because an initial search can be ambiguous. The user can guide the system toward the right part of the web instead of accepting an unexplained result. Copilot also supports follow-up questions and shared threads. Srinivas compares these threads to a public collection of natural-language programs: people can learn from another person's prompt, fork the thread, ask more questions, and share the result.

### Personal instructions reduce repeated prompting
[08:05](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=485s)
Srinivas presents AI profiles as a way to personalize search through global instructions. A profile can contain interests, hobbies, location, and language preferences, while the system asks dynamic questions to understand the user better. He says this differs from older web personalization, where people mainly selected topics or domains. A profile can tell the system how to respond across future searches. His example is a user who wants answers in Dutch, avoiding the need to add that instruction to every query.

### Tool use gives language models access to current information and calculations
[12:14](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=734s)
Srinivas says language models work well as reasoning engines, while search indexes, databases, knowledge graphs, and tools provide information and computation. Perplexity connects the model to live search so it can use updated web content rather than relying only on information stored in model parameters. He also describes combining models with Wolfram for mathematical explanations and calculations. In his view, the useful system comes from combining these capabilities, since a model alone cannot provide the same access to current sources or specialized tools.

### Orchestration creates latency and reliability problems
[15:03](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=903s)
Connecting several tools and model calls increases latency and creates more places where a request can fail. Srinivas says Perplexity initially took five to seven seconds per query. The team worked across the stack to make it faster, rather than relying on high-level tool libraries such as LangChain. Chained calls can fail when one component produces a bad result. The team addresses this with human feedback, prompt updates, prompt version control, quality benchmarks, model migration tests, and careful integration of search, rendering, databases, and other components.

### A search answer involves several specialized model workers
[18:37](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=1117s)
Srinivas explains that a Perplexity response can involve separate models for query classification, query reformulation, follow-up question generation, and summarization. A question about a research paper may be rewritten into three or four searches. Copilot then examines 16 results, giving the final model a broader set of pages than one search would provide. Another orchestration layer coordinates these workers. This architecture is why he describes the work as building an entire software system around language models.

### Evaluation happens whenever prompts or search indexes change
[26:45](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=1605s)
In the question period, Srinivas says the team runs evaluations whenever it changes prompts or the search index. The results are benchmarked before the change is accepted. He also uses the product himself and investigates complaints from users, since individual reports can reveal problems that affect many other searches. His practical description is that the system is tested in production while the team watches how it behaves and follows failures back to their causes.

### Model choice should follow evaluations and task requirements
[28:21](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=1701s)
Srinivas recommends choosing foundational models according to evaluation results and the type of work. At the time of the talk, he considered GPT-3.5 Turbo and GPT-4 especially strong for reasoning and logic. He preferred GPT-3.5 Turbo for many summarization and language-processing tasks, and said Claude Instant was competitive for some work but weaker for coding. Perplexity also tested smaller and open models, including MPT and Falcon, while he judged them less capable or slower than the proprietary models then available.

### Medical answers can help during panic, but they do not replace a practitioner
[32:16](https://www.youtube.com/watch?v=HzGiVzYbf2I&t=1936s)
Asked whether someone could use Perplexity for medical needs, Srinivas describes using it after injuring himself during a cave hike. The answer helped him understand possible meanings of his symptoms while he was waiting to reach a medical center. He says the system can help during the first moments of confusion and can point out when action may be needed. He also says users should consult a practitioner and should not act with blind faith in the answer. Perplexity supports questions in multiple languages, which he sees as useful for people asking in their native language.

## Notable quotes
- "The real end game is to connect LLMs with a lot of other tools like search engines, Python interpreters, and leverage the tool's power and robustness." (13:15)
- "The major computational cost was actually the LLM more than the search index." (36:01)
- "I would ideally consult a practitioner and make sure it's perfect. Just don't go blind faith of what Perplexity is saying and just act on it." (33:27)
- "We don't use anything else like, I guess, we use Kubernetes and all that." (30:53)

## Tools & references mentioned
- Perplexity AI
- Copilot
- OpenAI
- GPT-3.5 Turbo
- GPT-4
- Claude Instant
- Wolfram
- LangChain
- Kubernetes
- Elastic
- Qdrant
- Pinecone
- Weaviate
- MPT
- Falcon
- Chrome extension
- iOS
- Android

## Who should watch
- You are building a search or question-answering product and need to understand where retrieval, query reformulation, and tool calls fit around an LLM.
- Your system uses several model calls and you need practical ideas for latency reduction, evaluation, prompt version control, and model selection.
- You want a candid product discussion about the limits of AI-generated medical information and the role of human judgment.

## Related talks

- [Agents as Search Engineers](https://mlopstalks.com/talks/agents-as-search-engineers) (Santoshkalyan Rayadhurgam, Meta, 29:38)
- [Embeddings and Retrieval for LLMs: Techniques and Challenges](https://mlopstalks.com/talks/embeddings-and-retrieval-for-llms-techniques-and-challenges) (Anton Troynikov, Chroma, 35:19)
- [Unlocking Real-World LLM Use Cases](https://mlopstalks.com/talks/unlocking-real-world-llm-use-cases) (Hamsa Buvaraghan, Google Cloud, 12:16)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)
- [DevTools for Language Models: Unlocking the Future of AI-Driven Applications](https://mlopstalks.com/talks/devtools-for-language-models-unlocking-the-future-of-ai-driven-applications) (Diego Oppenheimer, Factory, 29:55)