Arjun Kannan's team uses unstructured property documents to answer 95% to 99% of resident questions and support far more apartments per associate.
2
Jason Liu argues that a small technical capability can have large value when it is attached to a specific customer and expensive workflow.
3
Agnieszka Mikołajczyk-Bareła describes a sales-call analysis system that found effective responses to objections and helped increase sales by 30%.
Summary
This panel focuses on the difference between an LLM capability and the value created around it. Arjun Kannan describes using property documents, including posters, PDFs, and informal replies, to answer resident questions. The system increased answer coverage from about half of questions to 95% to 99% and allowed each associate to support many more apartments. Jason Liu explains that transcript extraction or summarization becomes much more valuable when it serves a narrow audience with an expensive workflow, such as executive coaches preparing board presentations. Agnieszka Mikołajczyk-Bareła discusses analyzing sales-call transcripts to identify arguments that worked for particular objections, which her customer used in training. The speakers also describe early mistakes: over-engineering prompts, planning too far ahead, and treating product work as an engineering problem. They recommend fast experiments, continuous evaluation, customer-led priorities, and a focus on human outcomes.
Unstructured property knowledge can answer most resident questions
Arjun Kannan describes a customer-service system for rental properties. Residents ask about gyms, emergency hotlines, amenities, and other property-specific details, but the information may be in PDFs, old posters, notes, or someone's memory. His team puts these materials into a vector database. Before this, they answered about half of resident questions in a month. The current system answers about 95% to 99%. Training customers also became much faster because they no longer need to complete long questionnaires. Arjun says the system reduced expected support headcount by about half. One associate previously supported 7,000 to 8,000 apartments and now supports closer to 20,000.
A narrow customer problem can make a simple capability valuable
Jason Liu separates an LLM capability from the value attached to it. Summarization by itself may be cheap and easy to copy. Applied to a sales call, it can help capture pain points or identify opportunities that improve conversion. He gives an example involving transcript extraction for executive coaches. The API call cost about 50 cents, while the coach's work of finding quotes and preparing a presentation could take two hours at an hourly rate of $800 to $1,000. Because the output addressed a specific workflow, the service could support a much higher price. Jason's advice is to narrow the audience and understand exactly what problem that group is trying to solve.
Sales-call analysis can turn objections into training material
Agnieszka Mikołajczyk-Bareła describes analyzing thousands of hours of sales-call transcripts. The system detected objections and compared the arguments used in response to them. For example, saying that a company is the market leader might work less well than explaining that its price is the best. The customer used the findings to train sales workers and reported a 30% increase in sales. Agnieszka also points out that some callers may already have been using better arguments, while the wider team had no way to share that knowledge. The analysis made those patterns available for training.
Shipping early is better than spending months perfecting an LLM workflow
Arjun says his team initially treated LLM work as an engineering optimization problem. They spent too much time searching for the best prompt and latency, which made shipping take weeks instead of hours or days. They changed their process by putting a basic version into production and evaluating it continuously. For conversation scoring, they now start by asking the model to identify anything worth noting. They collect about a thousand examples, use those examples to define scoring buckets, and then apply the buckets to another set. The process helps them learn what the product should do before investing in detailed design.
Jason says teams often plan too far ahead by defining deterministic results, breaking work into tickets, and predicting edge cases. LLM optimization has a scientific element, so teams cannot guess every outcome in advance. He recommends building a test pipeline around metrics, hypotheses, and experiments. A test that runs in a minute changes the work compared with one that takes an hour. Evaluations also let engineers compare strong and weak examples and build practical intuition about the system. He advises junior engineers to learn the available evaluation methods and make experiments quick to run.
Customer demand should determine which LLM projects get attention
Arjun says Resides first used LLMs as a discovery tool, looking for capabilities that might become valuable features. As customer needs became clearer, the company began working from a more validated problem pipeline. The team weighs whether a project should automate human work or make the human team more effective. Arjun currently favors improving the team's understanding of documentation, customers, and useful conversations. Traditional impact-versus-effort scoring became less useful because many LLM projects required similar effort. The company instead tests what customers might pay for, since projects can differ greatly in commercial value even when their implementation effort is similar.
The desired outcome matters more than the AI feature
Jason uses a value equation based on the desired outcome, the likelihood of success, the time required, and the sacrifice involved. A product may save two hours by completing a task in ten minutes, improve accuracy from 95% to 99%, or reduce the effort of reading difficult legal material. He is more interested in outcomes such as becoming a better salesperson or having better memory in relationships than in selling summarization or search by themselves. His example of Rewind involves remembering details that help a salesperson follow up with someone or help a person remember what another person likes.
Production exposes a long tail of problems that prototypes hide
Jason says many people can achieve an 80% result and post about it, but launching a product exposes problems that are invisible in a demonstration. A product can gain attention and then fail to deliver on its promises. Production work also reveals industry-specific issues, such as unusual questions in a retrieval application or difficulty parsing a particular kind of data. Agnieszka's lesson is to focus on user value instead of getting lost in exciting research. Arjun adds that his customers often prefer a product that feels human rather than one that makes its AI obvious.
"The biggest shift for us was just changing that to be like, no, just ship what you can and then evaluate it continuously in production."Arjun Kannan12:21
Who should watch
You are building an LLM feature and need to connect its technical capability to a customer's willingness to pay.
Your team is spending weeks designing prompts, workflows, or edge cases before testing a basic version with real data.
You work with sales calls, support conversations, or messy documents and want examples of how analysis can improve human work.
