Data engineering lets data scientists work with larger datasets, build pipelines, and apply feature engineering at scale.
2
A single person can understand the whole machine learning stack, but usually needs to go deep in one area and work with complementary specialists.
3
Security, cost, reliability, and user needs should shape a machine learning project from the beginning, rather than being addressed after the model is built.
Summary
Dan Sullivan explains why data scientists benefit from learning data engineering. Data engineering adds the skills needed to move, clean, scale, and process data when spreadsheets or small pandas data frames stop being enough. It also helps data scientists understand cloud costs, managed services, streaming systems, reliability, and security. Sullivan argues that no one can know every part of data science, data engineering, software engineering, and operations in depth. A T-shaped approach is more realistic, with people specializing while collaborating across the system. He describes a failed natural language processing project where a strong F1 score did not help the researchers who were meant to use the tool. The project lacked early involvement from its users. Sullivan recommends starting with a good-enough model, getting feedback early, and improving in small increments. He is also direct that saying "I don't know" and forgetting tools that are not currently in use are normal parts of technical work.
Modern machine learning grew from better tools, data, and computing
Sullivan contrasts the symbolic AI systems of the 1980s and 1990s with current deep learning. Earlier systems used expert systems and hand-built language parsers, and they were often brittle. There were no widely shared platforms comparable to Spark or scikit-learn. He says progress came from larger datasets, more computing power, and methods for training deep networks. He also expects ideas from symbolic AI to combine with deep learning, pointing to work that used a neural network for an integration solver as an example of this direction.
Production machine learning requires software engineering and operations
Sullivan describes research in a Jupyter notebook as a different process from running a system that people depend on. Production work includes version control, automated testing, incremental releases, security, patching, monitoring, and responsibility for dependencies. Data scientists who come from mathematics and statistics may not have this software engineering background, while software engineers may not have the same depth in math. Sullivan therefore advocates collaborative development because the whole system contains too much knowledge for one person to master.
Data engineering gives data scientists power at larger scales
Sullivan compares learning data engineering to moving from a handsaw to power tools. A data scientist may know which algorithm fits a streaming analytics problem, yet still struggle to clean data, engineer features, and move files when the dataset becomes large. Data engineering introduces ways to process data at scale, from Linux command-line tools to managed pipelines such as Cloud Dataflow. It also includes data exploration tools that can show value distributions, null counts, and other signs of data quality.
Domain knowledge matters because real data is rarely ready for an algorithm
Sullivan says data scientists and machine learning practitioners often have more contact with what is inside the data than data engineers who focus on moving it. They need to understand the problem domain as well as the mathematics. A measurement can have the wrong unit, such as centimeters instead of inches, or a field can mean something different in finance than in life sciences. These details affect how data should be cleaned and transformed before it reaches a model.
A T-shaped skill set is more realistic than full-stack mastery
Sullivan says one person can understand the pieces across data science, data engineering, software engineering, and operations, but usually cannot work at depth across all of them. Some people are better at visualization and working with domain experts, while others specialize in databases and moving large data volumes. He says teams should combine these strengths. He also treats saying "I don't know" as normal, and notes that he would need to reread documentation before working with Spark again because he cannot remember everything.
A failed NLP project taught Sullivan to involve users early
While working at a genetics research institute, Sullivan built natural language processing tools to help molecular biologists read and annotate research papers. He focused on algorithms and celebrated a strong F1 score, but the finished tools did not fit the curators' workflow. The project was useful for publishing papers and discussing technical results, yet it did not solve the users' problem. Sullivan says he should have involved the people who would use the system earlier and now places more value on user experience and human factors.
Data engineering knowledge helps control cloud cost and reliability
Sullivan says data engineering helps people understand the cost of processing large amounts of data and avoid practical mistakes, such as leaving a cluster running for days. Knowledge of cloud features can also reduce spending. He mentions AWS spot instances and preemptable virtual machines on Google Cloud Platform for workloads that can tolerate interruption. Data engineering also covers scaling and resilience, such as using messaging systems to decouple services and designing systems that continue operating through small failures.
Security and compliance should shape the project from the start
For data covered by privacy or health regulations, Sullivan recommends contacting information security staff early and reading the organization's policies before building too far. Sensitive data should be removed from working datasets when it is not needed. Fewer copies mean fewer places that must be locked down, monitored, protected, and audited. He treats this as a practical design choice that can prevent expensive fixes later.
Sullivan prefers incremental improvement over waiting for a fully optimized system. A model that is only somewhat better than a baseline linear regression can still be useful for gathering feedback. Putting early results in front of the people who will make decisions with them can reveal that the model answers the wrong question, predicts the wrong variable, or misses an important domain detail. His approach is to work early, often, and in small increments.
"The fewer copies that are of that data that are around the fewer that need to be locked down and monitored and protected and audited."Dan Sullivan44:54
Who should watch
You are a data scientist who can build models but struggles when datasets become too large for local tools or when pipelines need to run repeatedly.
Your model works technically, yet production teams still need answers about cost, security, monitoring, scaling, or reliability.
You are deciding how broadly to learn across machine learning and data engineering and want a realistic view of specialization and teamwork.
