Small companies can use LLMs to build useful production features quickly, even when they lack the money, people, and data of larger competitors.
2
Anzen automated insurance application classification and extraction, reducing a process that required manual work to a system built in under a week.
3
Production LLM systems still need infrastructure knowledge, evaluation metrics, and careful cost planning, especially when business logic changes frequently.
Summary
Cameron Feenstra explains how Anzen, a small insurance company, uses language models to move quickly against larger incumbents. The first example is insurance underwriting. An automated classifier identifies applications arriving in the inbox, while AWS Textract extracts fields from documents. The second example is a compliance product that extracts facts from employment documents, then applies rules based on legal knowledge gathered from attorneys. Feenstra describes prototypes built in about a week with small datasets and existing models. He is also direct about the limits. LLMs need production infrastructure, evaluation, and cost controls. API-based systems become expensive when prompts or business rules change. In the discussion, he says model outputs are nondeterministic and can change when a provider updates a model. He recommends treating evaluation as an ongoing engineering task. Generative models may later help with underwriting and compliance search, but those ideas were still exploratory at the time of the talk.
Small teams can use language models to close some gaps with larger companies
Cameron Feenstra frames LLMs as a way for a small company to do work faster and accomplish more with fewer people. Large incumbents have more money, larger teams, valuable datasets, and network effects. They can also be slower to adopt new technology because of bureaucracy or a preference for existing systems. Anzen had fewer than 20 people when Feenstra gave the talk. He uses the company as an example of how a small team can build useful systems quickly instead of trying to match a large competitor resource for resource.
Anzen automated the first steps of insurance underwriting
Insurance underwriting starts with a company application, financial information, prior lawsuits, and other details. An underwriter then decides whether to write a policy and what to charge. Anzen receives applications through brokers, who may work with many carriers and clients, so reducing friction matters. Its system classifies incoming email attachments, extracts information, checks for duplicates, and alerts the underwriting team when a new application needs review. The underwriter still makes the decision. The LLM-related system removes repetitive intake work from the process.
A small labelled dataset was enough to build a useful application classifier
Anzen began with a classifier based on Google's BERT model. The initial performance was close to random, so the team pulled about 300 attachments from the inbox and labelled them manually in an afternoon. The resulting classifier reached about 95 percent accuracy, 90 percent precision, and 100 percent recall in their testing. The team accepted some noise because missing an application was worse for this use case. They also tried several open-source models and training configurations on Hugging Face. Feenstra says the end-to-end system worked in under a week, with much less effort and data than training a model from scratch.
Document compliance checks combine semantic retrieval, question answering, and rules
Anzen explored a product where someone could upload an offer letter or separation agreement and receive immediate feedback about possible compliance issues. Attorneys supplied domain knowledge about problems found in these documents. For each document type, the system extracts features such as salary and exempt or non-exempt status. Sentence embeddings first find the relevant parts of the document. A question-answering model then extracts the exact value. Business logic turns those values into positive and negative insights. The prototype used a BERT-based question-answering model and Microsoft's MiniLM for sentence embeddings, with a few dozen examples for each document type.
Production LLM systems require ordinary infrastructure and ongoing evaluation
Feenstra says teams hosting models themselves need strong infrastructure knowledge because the models consume substantial resources and can fail under load. Existing production engineering practices still apply. Evaluation metrics should show whether performance matches testing results and whether it changes over time. For prototypes, Anzen ran fairly large models on ordinary CPU instances instead of immediately buying specialised hardware. He also warns that API costs include the business logic around model outputs. When prompts, inputs, or rules change, repeated testing can become expensive or even cost prohibitive.
Generative models could assist underwriting and compliance search, but those ideas were still early
Anzen had not put its more ambitious generative-model ideas into production. Feenstra thought a model could gather company information for an underwriter and make an initial recommendation. He also saw value in asking natural-language questions about compliance and receiving an answer that points to the right source. The information needed for compliance is scattered across many places, which is one reason companies rely on lawyers. These uses remained ideas and experiments rather than finished products. The systems Anzen had already deployed used narrower classifiers, extraction models, and explicit business logic.
LLM debugging depends on measurements because model behaviour is hard to inspect
In the discussion, Feenstra says debugging unfamiliar LLM behaviour often comes down to trial and error. Teams can vary parameters and compare results against an evaluation metric, which feels similar to testing model configurations. He says there are no tools that let engineers see the full flow of information through an LLM and use that to explain a failure. Traditional software engineering methods still help, but the models are nondeterministic. An API provider can also change the underlying model without the customer knowing, so identical inputs may not always produce identical outputs.
LLMs should be used where they solve an unsolved problem, not where ordinary code is enough
Feenstra agrees that a language model can make software worse when it replaces a simple coded solution. A model may be slow and only work correctly part of the time, while the original code could have been faster and more reliable. He says Anzen's production use cases avoided simply sending a large amount of information to ChatGPT. LLMs can help with unsolved problems such as difficult underwriting decisions, but teams should still ask whether the model is necessary. He describes LLMs as a new software primitive that also creates new user-experience problems.
"We were able to put this together and well the training data we were able to put together in an afternoon and basically end to end we got the whole thing working in under a week."12:02
Who should watch
You are building an LLM feature at a small company and need examples of narrow systems that can reach production quickly.
You are deciding whether to use a language model for document extraction, classification, or compliance workflows.
You need a candid account of evaluation, infrastructure, API cost, nondeterminism, and the risk of using an LLM where ordinary software would be simpler.
