Matei Zaharia created Spark after seeing that Hadoop-style systems handled web indexing well but were poorly suited to iterative machine learning and interactive queries.
2
Spark and MLflow grew through early user feedback, composable open-source ecosystems, and company support that gave other organisations confidence to adopt and contribute.
3
Smaller language models connected to search or databases could make applications easier to update and control than relying only on large models or retraining.
Summary
Matei Zaharia describes how Spark began as a Berkeley PhD project focused on large-scale data processing. Existing systems such as Hadoop worked for web indexing, but users needed iterative machine learning, interactive queries, and reusable libraries. Spark gained users quickly, then Databricks provided full-time contributors and a company that could help organisations adopt it. He draws a similar lesson from MLflow, which defined shared concepts and APIs around experiments, metrics, models, and model registries while validating them with companies at different stages of ML adoption. Zaharia also explains Databricks' lakehouse ML approach, where model inputs, predictions, delayed feedback, monitoring, and analytics can be handled together. His Stanford group is studying applications that combine language models with search and retrieval systems. He expects this kind of pipeline to give engineers more control than training a model from scratch, while allowing knowledge to be updated without retraining the model.
Spark started because existing data systems could not support the applications people wanted
Spark began during Matei Zaharia's PhD at Berkeley, when large-scale computing was spreading from web companies to more organisations. Hadoop and related systems made it possible to collect and process large datasets, but they were mainly suited to tasks such as building a web index. Zaharia saw that users wanted to run machine learning and other algorithms over the same data. Spark initially focused on iterative machine learning algorithms such as stochastic gradient descent and on interactive ad hoc queries. Its connection to ML also came from Berkeley's environment, where early users were machine learning researchers who wanted to try large-scale methods.
An open-source community can grow before a company exists around it
Spark was open-sourced in 2010, and Databricks began in 2013. Zaharia says the user community formed within about two or three years. At first, he knew nearly every user and answered many mailing-list questions himself. Later, users began answering one another. Spark benefited from arriving when interest in large-scale machine learning and data analytics was rising. Its programming model was easier to use than competing systems, and it supported libraries and an ecosystem of packages. The project also let users compose functions into an execution plan that the system could optimise.
Company backing helped Spark move beyond research users
Zaharia says Databricks was needed to take Spark beyond technology companies and early research users. Some established vendors felt threatened because Spark competed with existing products such as Hadoop and made future demand difficult to predict. A company could hire people to contribute to Spark full time, help make the project stable, and give other companies confidence to adopt it. Once more organisations joined and contributed, the community became stronger. Zaharia presents this as a practical part of open-source growth, since adoption depends on support and trust as well as on the code itself.
MLflow was designed around shared concepts and an extensible ecosystem
Zaharia describes MLflow as a later project that benefited from Databricks' backing from the start. The team saw that ML users needed an open system that different tools could extend, rather than a platform controlled by one vendor. They worked through concepts such as experiments, metrics, models, and model registries, then tested those ideas with users at startups and large companies. A model registry could hold model versions and support review before approval. Zaharia says the team kept the number of concepts low and tried to find ideas that worked across organisations with very different ML teams and production experience.
Lakehouse ML connects model operations with ordinary data analysis
Zaharia explains lakehouse ML as a way to treat data and ML as connected processes that produce information for later analysis. A model may generate predictions first, while feedback arrives later, such as whether a user clicked a recommended product. Databricks' model monitoring approach can place model events into tables, where teams can analyse them with data warehousing tools. The same setup can check shifts in input or output distributions, calculate delayed feedback, connect operators across a pipeline, and create dashboards or alerts. This approach avoids treating model monitoring as a separate activity from the data work around it.
ML applications need more control than a single language model call
At Stanford, Zaharia's group is working on DSP, or Demonstrate Search Predict, which combines language models with systems that return text, especially search and retrieval systems. A question-answering application could search Wikipedia several times, use the results to guide later searches, and then produce an answer. Zaharia says language models can follow instructions and generate useful text, but their quality is difficult to control when they are used alone. DSP puts the model inside a larger pipeline with a specific task. The application can call models multiple times, use several models, find mistakes, and improve reliability without making one model responsible for everything.
Smaller models with external memory could be easier to update
Zaharia argues that large language models contain both language-processing ability and a large amount of memorised world knowledge. That knowledge contributes to the cost and scale of training. He thinks those functions could be separated, with a smaller neural network querying a search index or database for information. The model's knowledge could then be updated directly. His example is a change in the US presidency: a system could upload a current Wikipedia article instead of waiting to retrain the whole model. He still considers training your own model useful, but he sees it as a coarse tool with fewer ways to adjust an application after training.
Good ML infrastructure becomes more useful when it becomes simpler
Zaharia sees room for new ML application and infrastructure companies because algorithms and application patterns are changing quickly. Current systems often require teams to connect many separate components. A company that simplifies a specific application area could make it available to more users and expand the market. He also points to OpenAI, Labelbox, and Scale AI as organisations that helped more teams attempt technically or operationally difficult work. His broader advice is to understand users directly. He tells students and product builders to find someone who does the task as part of their job, then use that experience to uncover problems that researchers or developers may not see.
"You might be able to produce equally good answers and end up with a small model if you take a smaller neural network plus the ability to query just a search index."Matei Zaharia53:14
Who should watch
You are deciding whether an open-source project should become a company-backed product, and you want practical lessons from Spark and MLflow.
You are building ML systems that need monitoring, delayed feedback, and analytics alongside model serving.
You are evaluating language-model applications and want alternatives to repeatedly retraining a large model.
