Search is harder to monitor than conventional machine learning because queries can have no correct answer, many correct answers, or ambiguous feedback.
2
Human-labeled data is the starting point for useful synthetic data, and its quality matters more than simply increasing the dataset size.
3
RAG systems should begin with simple architectures and metrics, then add complexity only when measured improvements justify it.
Summary
Sam Bean discusses the operational problems behind search, LLM applications, and production RAG. Search has many failure modes because a query may have no answer, many valid answers, or an answer whose quality is difficult to judge without user behavior. Sam recommends starting with a small set of human-labeled examples, then using those examples to guide synthetic data generation and smaller models. He applies the same discipline to system design: begin with simple retrieval, evaluation, and business logic, then add neural models, behavioral signals, agents, or more elaborate RAG patterns only when metrics show a need. At Rewind.ai, he works on systems that capture screen activity, audio, and text so an LLM can retrieve a person's past context. He also explains why he prefers DSPy for self-improving and distillation workflows, while noting that distributed training and infrastructure remain practical barriers.
Search needs behavioral feedback because its answers are inherently ambiguous
Sam Bean contrasts traditional forecasting work with search. In forecasting, each point has a feature set and the model output can often be evaluated against a known target. Search is different. Some queries describe things that do not exist, some have millions of valid answers, and most sit somewhere between those cases. Without behavioral data, it is difficult to know whether a result was useful. A/B tests, surveys, and other forms of user feedback are expensive. Adding an LLM makes the system harder to deploy in some ways, but it also creates a clearer end-to-end question-answering target. Datasets such as SQuAD v2 and HotpotQA provide examples where an answer can be judged as right or wrong.
An end-to-end answer metric can expose problems in the search stack
A search system may contain intent detection, query rewriting, retrieval, reranking, and filtering models. Sam describes this as five to ten models working together. Instead of trying to assess every component independently from the start, he recommends putting a question-answering system at the end of the pipeline and measuring whether it answers questions correctly. That gives the team a human-readable view of progress. Engineers can then work backward through the subsystems and ask whether a change helps the final answer. He says this creates a useful grounding for search work because information that helps the LLM understand a question will often help a person understand it too.
Sam says there is no shortcut around collecting the first batch of labeled examples. He describes labeling 100 or 200 queries by hand, often in a spreadsheet, and using that set to align a smaller model or prompt a larger model to produce similar examples. Synthetic data can then expand the dataset, but its usefulness depends on whether it stays close to the real distribution. If synthetic examples hurt evaluation scores, the initial human-labeled set may be too small or poor in quality. Sam's point is that 100 good examples can be more useful than a much larger set of weak examples. Human judgment remains the source of the initial definition of a correct answer.
More data can allow a team to replace an oracle model with a smaller model
Sam describes a progression from a large oracle model and a small amount of hand-labeled data toward smaller models trained on more examples. At You.com, a few-shot prompt with about ten lines of Python beat an intent model that had taken six months to build. That result then made it possible to use the larger model to label data and train a smaller replacement. The process can eventually converge toward a model similar to the original distilled BERT system, but with a much larger body of synthetic data. Sam connects this to the REST and less-is-more alignment ideas, stressing that data quality and distribution matter more than collecting examples indiscriminately.
Rewind.ai treats a person's captured activity as searchable memory
At Rewind.ai, Sam works on software that captures and indexes a user's activity. The system records microphone audio, applies OCR to screen captures, converts the material to text, and makes it available to an LLM. The intended use is to recover things a person saw or heard but later forgot, such as a paper or an earlier discussion. Sam imagines the system creating links across events in time, so a conversation about a forgotten paper could lead back to the original page and related earlier context. He says the company is considering multimodal approaches because operating directly on speech and images could avoid the extra translation layer created by converting everything into text first.
Text should be mastered before behavioral signals are added
Sam pushes back on adding metadata and behavior signals too early. A document's content should first be shown to be relevant to the current question. Features such as how long a browser tab has been open may become useful later, but they can make a system harder to explain if the text model is not understood first. At web scale, popularity and personalization signals can matter because the same word may refer to different subjects. For a smaller document collection, Sam says those signals may not be needed, and ordinary business logic could be enough. His preferred sequence is to understand and deploy the text models before building an ensemble that combines them with behavioral features.
Sam recommends starting search with established counting methods such as word frequencies, TF-IDF, BM25, and a single OpenSearch or Elasticsearch instance. He applies the same principle to evaluation. A bag-of-words comparison between a predicted answer and a ground-truth answer may be imperfect, but it can provide a lower bound that the team understands. If that score improves, the team has evidence of progress. A complicated evaluator can produce scores whose meaning changes across examples, making it difficult to decide what engineering work to do next. Sam says a simple baseline should be easy to beat, then replaced only when its limits become clear.
RAG complexity should be justified by representative metric improvements
Sam considers naive RAG a reasonable starting point, even though the label can make teams avoid it. Teams can add knowledge graphs, tree structures, agentic search, or multihop reasoning later, but each addition needs a measurable reason. The comparison should use representative queries rather than selected examples. A new system should improve weak cases without degrading other cases. When a change moves the metrics, it may be worth keeping. When it adds code without changing results, Sam would remove it. He argues that the final architecture may look unlike systems discussed online because it will be shaped by the company's own data and business problems.
"If you are creating more work for yourself and you are saving using an LLM, then you're probably using it wrong."Sam Bean42:23
Who should watch
You are building a search or RAG system and are tempted to add agents, graphs, rerankers, or an LLM judge before you have a dependable baseline.
Your team needs a practical way to collect evaluation data and use human labels to expand a dataset with synthetic examples.
You are deciding whether to add behavioral signals or a self-improving LLM workflow and want to hear what infrastructure and measurement should come first.
