An AI product should be designed from the user's desired experience, then traced back through data, model, and infrastructure decisions.
2
Good model recommendations cover only part of the product requirement because latency, input quality, evaluation, cost, and API limits also shape the user's experience.
3
Engineers can gain production experience by experimenting with services such as SageMaker, Hugging Face, and AWS Bedrock, even before doing this work at a company.
Summary
Annie Condon argues that teams should start with the experience they want users to have and work backward into the AI system. She illustrates this with a recommendation product where a business wants users to find a suitable match within their first five recommendations. That requirement raises questions about recommendation quality, the information new users must provide, response speed, and how much guidance users need when entering descriptions. Condon explains that LLMs can appear throughout the data science lifecycle, including text cleaning, feature engineering, and similarity search, but producing good recommendations is only one part of the work. Teams also need to consider API rate limits, cost, compute, evaluation data, reproducibility, and deployment. She describes using AWS while testing these questions and points to SageMaker integrations with Hugging Face and AWS Bedrock as ways to experiment. Her conclusion is honest: deploying LLMs is not as quick and simple as marketing often suggests, but people can start learning by building and testing small applications themselves.
Condon begins with a business requirement: users should find a match within their first five recommendations. That immediately shifts attention toward the front end. The team has to consider how good the recommendations are, what a new user must enter, and how quickly the application can respond. She notes that some users may be Boomers who are less accustomed to applications, so the product has a short window to provide a good experience. The business request therefore contains product and system requirements before anyone discusses model performance.
A recommendation model is only one part of the product
Condon contrasts this end-first view with the more linear data science lifecycle she knows: obtain data, improve features, and seek better model performance. Her team was using inference rather than fine-tuning or retrieval-augmented generation, yet LLMs appeared across much of the lifecycle because their input was mostly unprocessed text and they lacked labels. They used LLMs for cleaning scraped text, engineering features, and creating similarity searches. Those steps can help produce recommendations, but they do not by themselves satisfy the whole business need.
Using an OpenAI API for data cleaning introduces practical constraints that differ from traditional preprocessing. Condon asks how long it will take to process users' data at once when the API has rate limits and costs money. Tokenization and text embeddings also raise compute questions, as does running a pretrained model such as BERT. These concerns belong in the design from the start because they affect whether the intended product can operate within its required time and cost.
Condon asks how teams will evaluate LLM predictions and similarity searches against what they want users to experience. The team needs to collect the right data points, rather than stopping after observing a technically interesting result. Input quality is part of evaluation too. A user may provide only a one-sentence description of a person or business, which may not contain enough information for a good recommendation. The product therefore has to decide how much structure and guidance to provide during data collection.
A successful experiment still has to be reproducible in the application
Condon describes data scientists becoming discouraged when they realize that an impressive LLM result must be reproduced and automated inside an application. Her own response was to deploy models in AWS and test the limits of the questions she had raised. The point of this work is not a particular cloud provider. It is to expose the operational details that are easy to overlook when a model is tested in isolation.
Cloud services can lower the barrier to first experiments
Condon points to AWS integrations as a practical way to get experience. SageMaker can work with Hugging Face to deploy LLMs without downloading model artifacts. AWS Bedrock can invoke an LLM through the Bedrock runtime. She also mentions a DeepLearning.AI course on creating serverless LLM applications with Bedrock. Her advice is aimed at people who have not yet deployed an LLM professionally: create an account, try the tools, and learn by testing rather than waiting for the perfect job or project.
Condon closes by revising her initial promise to give a quick and dirty way to deploy an LLM. After working through the details, she concludes that deployment is not simple. The host agrees that marketing teams often present it as easy, while the real system becomes much more complicated. Condon's practical conclusion is to start anyway, accepting that hands-on experimentation is how people begin to understand the work.
"Once we have those good recommendations and we've sort of been through that data science life cycle, if we go back to that business need, that only covers just a small portion of that business need."05:47
Who should watch
You are building an application with LLMs and need to connect model choices to response time, input collection, and the user's actual experience.
Your team has good experiments but has not yet worked through rate limits, cost, compute, evaluation data, or reproducibility in production.
You want a practical starting point for experimenting with LLM deployment before you have a dedicated production project.
