LLM use cases should be assessed for both business impact and technical feasibility before a team commits to building them.
2
The impact checklist covers the customer, the job to be done, whether an LLM is needed, and whether the proposed solution is better, faster, or cheaper than alternatives.
3
The feasibility checklist covers the solution level, deployment mode, automation, user type, task shape, model choice, and data sensitivity.
Summary
Rens Dimmendaal and Eva Bosma present a checklist for discussing LLM use cases with business and technical stakeholders. They start with an impact-feasibility matrix: teams should prefer use cases with high impact and high feasibility, while avoiding ideas that score low on both. The impact checklist asks who the user is, what job they need done, whether an LLM is actually necessary, and whether it is better, faster, or cheaper than another approach. The feasibility checklist then examines the level of complexity, deployment mode, automation and human review, internal or external users, specific or open-ended tasks, model choice, and data sensitivity. Examples include latency for online applications, different accuracy requirements for movie recommendations and patient questions, and a natural-language query assistant at Honeycomb. The framework is simple, but it gives mixed-experience teams a shared way to challenge attractive ideas before investing in them.
The speakers use an impact-feasibility matrix to connect business and technical discussions. The preferred area is the upper-right corner, where a use case has high impact and is feasible to execute. Ideas in the lower-left corner have low impact and low feasibility, so the team should avoid them. Use cases in the yellow areas require either capability building or acceptance that they are small, quick gains. The matrix gives stakeholders a common starting point before they debate a specific LLM proposal.
An LLM should not be the default answer to every problem
Eva and Rens describe customers arriving with a specific generative AI idea because LLMs are the current hot topic. They compare this to the golden hammer syndrome, where a popular tool gets applied to every problem. The checklist asks whether a more traditional AI or data science solution would work instead. Their test for value is whether the LLM solution is better, faster, or cheaper than the available alternative. This keeps the discussion focused on the use case rather than the novelty of the technology.
The impact checklist starts with the user and their job
The impact assessment has four parts. First, identify the customer or user who will use the solution. Next, define the job to be done. Then examine the generative solution proposed by the business and challenge the assumption that it needs an LLM. Finally, compare the value drivers. A proposed LLM application has value when it improves on another solution by being better, faster, or cheaper. These questions help business stakeholders explain the intended outcome in concrete terms.
The feasibility checklist begins with the solution level: a prompt, retrieval-augmented generation, or agents that chain multiple prompts. The speakers warn that accuracy can fall as steps are compounded. Deployment is another factor. Batch and on-demand applications have different latency requirements. They cite examples of website slowdowns affecting viewing and revenue to make response time understandable to business stakeholders. If a use case needs a fast response, latency can make the idea less feasible.
Teams should decide whether a person remains in the loop or whether the system will be fully automated. The relevant concerns are reliability and risk tolerance. Ninety percent accuracy may be acceptable for automated movie recommendations, while answering patient questions may require human review. The right design depends on the consequence of an error, not on an accuracy number in isolation. This gives stakeholders a practical way to discuss how much oversight the use case needs.
Specific tasks are easier to validate than open-ended chat
The speakers distinguish between internal and external users, then between specific and open-ended tasks. Internal users are easier to reach and support, so they can be a simpler starting point. Turning an existing process into an open-ended chat application is harder when the team lacks clear success measures. They cite Honeycomb's natural-language query assistant as an example of a more grounded approach. Honeycomb already had a query tool, knew what users were trying to produce, and could check whether the generated query was valid.
Start with an existing model before training your own
Model choice is another feasibility question. Teams can use a vendor model through an API, an open-source model, or a tuned model. The speakers recommend starting with an off-the-shelf model and only considering tuning if the initial solution shows that it is needed. This gives a team a way to test the use case before taking on the extra work associated with training or adapting its own model.
Data sensitivity can make an application much harder
The final checklist item is data. The speakers separate public data, company-level sensitive data, and individual-level sensitive data. Teams need to check the terms and conditions that apply to the data, then consider how retrieval or tuning changes the application. More sensitive data can require rules about who is allowed to see which information. Those access boundaries add complexity to the design and affect feasibility.
"It's not rocket science, but you have to do it together to actually show whether the use case that you or your business counterpart has in mind is actually bringing value to the organization."Eva Bosma09:36
Who should watch
You are collecting LLM ideas from business teams and need a consistent way to decide which ones deserve further investigation.
Your stakeholders have different levels of technical experience and discussions about value, latency, risk, or data access keep going in circles.
You are considering a generative AI application and want to test whether a simpler AI or data science approach would meet the need.
