A production data product needs software checks, testing, monitoring, maintenance, and MLOps support alongside machine learning.
2
Teams need to collect and govern data with its quality, integrity, reliability, and lineage in mind before they build predictive products.
3
Companies should define data roles clearly and use machine learning only when it adds value beyond a simpler rule-based solution.
Summary
Elizabeth Chabot draws on startup and consulting work to explain why data products often fail before the model is the real problem. Companies frequently hire data scientists when they need data engineering, application development, data governance, monitoring, or MLOps. They also assume that collected data is useful without checking how it was gathered, whether fields changed over time, or whether the data is still reliable. Chabot argues that leadership must own the wider data collection effort, while data scientists should have room to test algorithms and build predictive tools. Early products may need hard-coded rules or human review before machine learning is justified. Tool choice depends on the team, the use case, cost, and risk. A tool can fill a skills gap when the risk is low, but security and other high-risk areas need qualified people. Role definitions should change as a startup moves from an MVP to a product with real customers.
At a startup building Bookkeeping AI for small and medium-sized businesses, Elizabeth Chabot found that different financial structures forced the team to create many variations of its algorithms. The team had statisticians and people who understood machine learning, but they lacked the software engineering experience to maintain pipelines and fit them into the application. Leadership kept suggesting more data scientists as the answer. Chabot argued that the real need included software checks, testing, monitoring, maintenance, and someone who could connect the model to the product. She told startups that they would eventually need an MLOps person, or someone who could combine MLOps and data engineering skills.
Bad data collection blocks predictive products before modeling starts
Chabot describes companies that wanted reporting, insights, or predictive analytics without having collected the required data. One company had not kept timestamps or useful attributes, and some identifiers stopped being unique. Another startup believed it had the right data for an algorithm, but its collection processes were not correcting the data properly. Her first questions are simple: what do you want to do, and do you have the data? Supervised machine learning needs historical information that was collected properly, with sufficient quality, integrity, reliability, and lineage. Startups change their processes quickly, so a field collected two years earlier may not mean the same thing today.
Data collection and governance belong across the organisation
Chabot says data collection cannot usually be left to the first data scientist, especially when that person was hired to develop algorithms. Collection, governance, integrity, and reliability need leadership support because they affect the whole company. She recommends setting a shared baseline of terms, then checking whether the organisation has a data dictionary, business glossary, and active reporting that people actually review. In older companies, leadership may believe a field is still being collected while engineers know that a bug has caused it to output zeros for years. Chabot prefers addressing the full gap directly rather than revealing problems one at a time, because gradual disclosures can damage trust.
A full-stack data scientist becomes a bottleneck when models need care
Chabot warns against centring a company on one person who is expected to do everything. A data scientist who creates algorithms, monitors them, maintains them, and handles application integration soon stops being available for new work. She says the team may need to divide responsibilities among data science, data engineering, software engineering, DevOps, security, and MLOps as the company grows. Early employees can combine roles, but real customers bring service expectations that require dedicated time. She also describes a case where a data engineer and MLOps person maintained the database, created reporting, and built a machine learning pipeline, even though the first algorithm needed more iteration.
A pipeline is worthwhile when the product already provides user value
Chabot contrasts two startup hiring mistakes. One company devoted half its 20-person team to an algorithmic product that had not been tested well enough with users and was not clearly producing value. It had to cut half the team and reconsider the product. In another case, the team had a viable use case, positive user response to a semi-automated output, and a pipeline that could support further algorithms. The model was not yet as effective as hoped, but the surrounding system made iteration possible. Her distinction is that a pipeline is not the problem when the underlying product has value. Hiring a whole team before testing that value is the problem.
Education helps leaders and practitioners agree on what data work requires
Chabot says the gap between executives and people collecting data is mainly an education problem. Leaders may be uncomfortable with engineering, analytics, or code, while practitioners may not understand the business context. She recommends defining basic terms without assuming prior knowledge, then using the product roadmap to work backward from the outputs the company wants. Teams can identify the data points needed for a future result and discuss them with product owners and engineers. This exercise may take several sessions, especially when the company is changing direction, but it gives people a shared way to discuss collection, metrics, timing, and outputs.
Machine learning should follow a demonstrated product need
Chabot argues that investors and product teams often ask for machine learning when they really need an automated output. An initial MVP may be better served by a human review step or a simple hard-coded rule. If users confirm that the output saves them work or provides value, the company can then bring in a data scientist to test and improve the solution. She describes a data scientist who produced three algorithms in a Jupyter notebook, after which executives assumed the work was finished. The algorithms still needed engineering and production work. A data scientist hired for advanced modeling should not quietly become the organisation's data governance caretaker.
Tool and hiring decisions depend on skills, cost, and risk
Chabot used SageMaker for an MVP, but it did not fit the team's longer-term iteration needs. Her rule is to use a tool when the team lacks a skill, the tool can solve the problem, and support is available from a service representative or engineer. A tool may cost less than hiring a person, but the organisation still needs enough people who understand how it works. She makes a stricter distinction for high-risk areas such as security. A company should hire qualified people rather than assume a security product closes every gap. Teams should also post an MLOps role before customers use the system and the team can no longer track who is testing what.
"If it's your first product and you know that it's an if else solution and you can put it in place and you don't know anything after that is hiring at a data scientist at that point a good idea?"Elizabeth Chabot53:42
Who should watch
You are hiring your first data or machine learning employees and need to separate data science, data engineering, software, DevOps, and MLOps work.
Your organisation wants predictive features but has not checked whether the required data is collected consistently and can be trusted.
You are deciding between a managed tool, a new hire, and a simpler rule-based MVP.
