Enterprise AI needs data and labels that reflect each organisation's own tasks, documents, and standards.
2
Programmatic labeling combines expert rules, prompts, embeddings, custom code, and human review into labels that can scale across a dataset.
3
Focused error analysis helps teams review uncertain examples, improve labeling functions, and train simpler models for production.
Summary
Elena Boiarskaia argues that enterprise AI teams should treat data development as a software development process. Off-the-shelf models and chatbots often lack the accuracy and organisation-specific behaviour required in areas such as healthcare and finance. Labels remain necessary across the pipeline, including document curation for retrieval-augmented generation, fine-tuning, alignment, prompt evaluation, and production monitoring. Snorkel's approach captures subject-matter expertise in labeling functions. These can use keyword rules, embeddings, prompts, custom code, and human annotations. The functions are combined into one label per data point, then reviewed through disagreement, confidence, bias, coverage, and slice analysis. Elena explains that this process turns a small set of expert judgments into labels for a larger dataset. A model can then learn from those labels, often with a simpler architecture than a hand-built rule system or a heavily tuned model. The method also applies when a data point is an entire document or a prompt-context pair in a RAG system.
Enterprise models need organisation-specific data and labels
Elena frames the talk as a change in how teams think about data in AI development. Enterprise data is often a company's differentiator, but an off-the-shelf LLM or chatbot does not automatically understand the organisation's tasks or standards. It may lack both accuracy and differentiation. That matters more in specialised settings such as oncology, where an incorrect prediction can have serious consequences. Her first takeaway is that teams need to specialise out-of-the-box models using their own data.
Labels remain necessary throughout generative AI systems
Elena rejects the idea that generative AI removes the need for labeled data. Teams still need labels to judge whether a chatbot response is acceptable, whether documents were used correctly, and whether outputs match the intended tone. In a RAG system, curation, chunking, embeddings, and indexing determine what context reaches the model. Teams also need labels for fine-tuning, alignment, prompt engineering, and evaluation. These labels help expose bias, hallucinations, and other errors before a system reaches production.
Subject-matter expertise should be captured as programmatic data operations
Snorkel's approach starts with collaboration between machine learning teams and subject-matter experts. Experts understand whether a model output is useful, but they cannot review an entire dataset. Their decisions can be distilled into programmatic data operations, including weak supervision labels. The process should also be adaptable when experts change their minds and auditable enough to explain how a label was created. Elena describes Snorkel AI as an IDE for programmatic data development.
Multiple labeling functions provide signal for one data point
A labeling function can capture a small piece of evidence rather than a complete definition. Elena gives the example that an email mentioning free cash may be a signal for spam, while an email asking about money may be another. Teams can combine expert-written rules, prompts to open-source LLMs, embeddings, custom code, and available human annotations. Snorkel combines these signals into one label for each data point, allowing a small amount of expert knowledge to spread across a larger dataset.
Programmatic labels also help curate and evaluate RAG data
The same approach applies to retrieval-augmented generation. Teams can label prompt and context-chunk pairs, then assess whether a response is good for a given prompt. This gives them a way to test the context presented to the LLM and the response that follows. Elena presents this as a more regular way to evaluate generative systems than relying on large amounts of manual annotation. The labels guide improvements to retrieval and model quality.
Error analysis directs humans to the examples that need review
After labeling functions are combined, disagreement and low confidence identify examples that deserve human attention. Rather than asking people to inspect the whole dataset, the system can direct them to a smaller group of uncertain or error-prone documents. Elena also describes checking data slices for label coverage, bias, and precision. The workflow is iterative. Teams add labeling functions and domain knowledge, inspect the resulting errors, and repeat the process.
Better labels can make the production model simpler
Elena's proposed shift is away from treating feature extraction, feature importance, and model tuning as the centre of the work. If the labels are good, the downstream model can be simple and lighter to deploy. In the discussion, she explains that a set of rules is not meant to become the inference system itself. A complex rule set can turn into an unwieldy SQL WHERE clause. Instead, the rules create labels, and a model such as logistic regression can learn how to generalise those labels across raw data.
The platform supports documents, different data types, and production handoff
Elena shows a no-code interface alongside a code-based SDK. Users can write fuzzy keyword matches, custom labeling functions, prompt-based functions, embedding-based functions, and regular expressions. A data point can be an entire document, not only a structured row or a small text field. Snorkel supports use cases such as document classification, information extraction, computer vision, and entity linking. Data can come from systems including Databricks, the platform can run in the cloud or on premises, and it outputs an MLflow object for deployment.
