Podcast

MLOps as Tool to Shape Team and Culture

Ciro Greco, CoveoEpisode 95 · 43:02 · Apr 2022 · 439 viewsHosted by Demetrios Brinkmann
Thumbnail for MLOps as Tool to Shape Team and Culture Watch on YouTube
TL;DR
  1. 1

    Ciro Greco argues that data quality and data management create more value for Coveo's machine learning systems than small gains in model quality.

  2. 2

    Machine learning engineers need to own the final data preparation because they understand what the data is for and how it supports a business use case.

  3. 3

    High-performing ML engineers build an end-to-end slice of a business problem, then work with people outside engineering to test and improve it.

Summary

Ciro Greco describes his move from cognitive science research into NLP, entrepreneurship, and his role as VP of AI at Coveo. Coveo supports search and recommendations across customer service and e-commerce, where similar-looking products have different data, personalization, and engineering needs. Ciro says the company treats data management as the shared foundation for its machine learning applications. Raw, immutable data gives teams a historical record that they can revisit when an abstraction proves wrong. Tools such as Snowflake help, but they do not create the discipline needed to preserve data integrity across partners and developers. Ciro also explains why ML teams should become more connected to business problems. He values engineers who can build a complete working slice, even if it is limited, and then learn from real use. The organizational side is harder than selecting tools because teams need room to try ideas, learn from failure, and work across different skills and managers.

Key ideas
03:43

Ciro Greco moved from theoretical language research into applied machine learning through Yahoo and a startup

Ciro Greco began in cognitive sciences, studying language and collecting data from deafblind patients who had developed a new sign language. His work asked how children acquire language despite languages being different. In 2017, he started working with Yahoo to connect this theoretical research with natural language processing. He and his co-founder then incorporated a company around information retrieval ideas from formal semantics. They arrived in San Francisco with $50,000, quickly learned that rent alone was expensive, and had to raise money before they had an MVP. His co-founder lived in a garage for two years. Coveo acquired the company in 2019.

07:15

Coveo has different search and recommendation problems under one business

Coveo began with enterprise search, putting information from different systems into a unified index. It later developed more specialized businesses, including customer service and e-commerce. In customer service, search and recommendations help people find answers so they do not open a ticket. In e-commerce, recommendations should keep people on the site and encourage a purchase. The use cases look similar from the outside, but their data and goals differ. Coveo therefore has many simpler models working together across different applications. Ciro says robustness matters more to this business than having one highly accurate model serving an enormous number of predictions.

12:23

Coveo treats the data portion of each ML application as a shared foundation

Ciro describes every machine learning application as a fork in a directed acyclic graph. The first part handles data ingestion and produces tables that different applications can consume. That work requires discipline and substantial manual effort. Coveo has customers who track data in different ways, so making the data consistently usable is difficult. The company created a standardized framework for tracking and ingesting data, while accepting that one standard cannot cover every line of business. Once the data foundation exists, the models can differ substantially by use case.

14:40

Personalization requires different data strategies for customer service and e-commerce

Customer service has relatively little browsing activity, but users often log in when they open a ticket. That makes it possible to build a user profile. In business-to-consumer e-commerce, only a small share of visitors log in, so a user profile would help very few people. Coveo instead needs to personalize from the current session. Ciro uses this contrast to explain why teams cannot assume that one model or data design will transfer directly between products. Some components can be shared, while others must reflect the details of the business case.

16:27

Ciro says better data produces more value than marginal model improvements

Ciro says that, in Coveo's situation, all marginal gains from improving models are much smaller than the gains from having good data. Snowflake made the company's life easier, but it did not solve the ingestion problem. Partners, professional services teams, and developers still implement tracking, and their choices can damage the integrity of the ingestion process. A warehouse technology cannot create the discipline required to keep data consistent. That discipline has to come from the organization and from the people designing the process.

18:38

Immutable raw data gives the warehouse a usable history

Ciro describes a data warehouse as a kind of time machine or Git for data. Traditional database habits make it natural to correct a bad event in place, but Coveo needs to preserve data in the rawest and most verbose form possible. Teams can then build an abstraction for an application, discard it when it proves unsuitable, and rebuild it from the raw record. Keeping the historical data also creates possibilities that were invisible when the warehouse contained only the latest state, including trend analysis, time series work, and prediction based on how data changes over time.

21:40

Machine learning engineers should own the final data preparation for their use case

Ciro says centralized data teams often remain centralized as they grow, even though data scientists become more connected to particular business cases. He is comfortable with data scientists speaking directly to clients because they need to understand what is special about the data and why it matters. The latest steps in data preparation should belong to the data scientist or ML engineer, since that person knows what the data is for. The company can still reduce operational complexity by agreeing on shared languages or parts of the stack, while allowing different teams to use tools that fit their work.

33:29

An end-to-end slice helps an engineer turn a business problem into something testable

Ciro's preferred sign of a strong data scientist is the ability to build an end-to-end slice. It may be limited or imperfect, but it has a beginning and an end, so the team can try it and decide what deserves more work. This requires going deeply into the business case. An engineer must speak with an e-commerce manager or product manager, understand their constraints, and avoid treating those concerns as inferior to technical ones. Ciro says this creates a culture of iteration and makes it easier to learn which parts should be expanded and which belong in technical debt.

38:12

Organizational permission matters more than tool choice when teams adopt MLOps

Ciro separates the visible tooling layer from the harder organizational layer. Young engineers may want familiar tools because those tools appear in job descriptions, but they also need the authority to investigate a business problem and propose an experiment. Security, privacy, customer data restrictions, unfamiliar tools, managers, and department structure can all block a technically sound plan. Coveo's models are important to its value proposition, but they are not single points of failure, so a weaker new recommender can often be rolled back. Ciro sees room for a culture where people try ideas, discover that they are wrong, and try again.

"The latest steps in the data preparation must belong to the data scientist, to the machine learning engineer."Ciro Greco24:23
Who should watch
  • You are building ML systems for several products whose data and business goals look similar but behave differently.
  • Your team has chosen warehouse or ML tools, yet developers and partners still break data ingestion or disagree about ownership.
  • You manage ML engineers who need to work more closely with product and business teams, and want a concrete view of what good end-to-end work looks like.