# How to Choose the Right ML Tool

Jose Navarro, Cookpad & Mariya Davydova, Neu.ro | MLOps Coffee Sessions | Episode 13 | 1:00:39

Source: https://www.youtube.com/watch?v=mmTCGkm3ZoQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-to-choose-the-right-ml-tool
Published: 2020-10-15
Tags: build-vs-buy, open-source, platform-teams

## TL;DR
- Jose Navarro starts tool selection by asking whether the team needs another tool at all, since adopting the latest software can create unnecessary operational work.
- The choice between managed and open-source tooling depends on requirements, scale, engineering capacity, migration cost, team expertise, security constraints, and the people or community behind the tool.
- For urgent gaps, Mariya Davydova recommends building a small internal MVP when waiting for a release or migrating carries more risk, then replacing it later if a suitable tool becomes available.

## Summary
Jose Navarro and Mariya Davydova discuss how ML teams should choose tools when managed services and open-source projects offer overlapping functionality. Jose recommends first checking whether an existing system can solve the problem, then comparing the expected value with the engineering work required to build and maintain a solution. The discussion covers scale, migration effort, team learning costs, security restrictions, tool maturity, release cycles, community support, and vendor risk. Jose describes Cookpad's decision around Feast, which was running on AWS but lacked production-ready AWS support at the time. The team had to weigh waiting for a release against building a small internal feature-store MVP or choosing a managed product. Mariya also discusses experiment tracking, where TensorBoard can become difficult to use with many parallel experiments and Weights & Biases introduces different infrastructure and data-location considerations. The conversation ends with organizational questions: large companies may need shared infrastructure standards, while product teams still need room to work independently.

## Key ideas
### Teams should check existing systems before adopting another tool
[03:17](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=197s)
Jose Navarro says tool selection starts with a step back. When a new problem appears, the team should ask whether an existing solution can handle it before integrating something new. He warns that engineers can get excited about the latest software and quickly install multiple components, Helm charts, and Kubernetes clusters without first proving that the extra tool is needed. The question is practical: will the new tool solve a problem that the current stack cannot solve? This first check can prevent the team from taking on another system to operate, learn, and maintain when a simpler change to the existing stack may be enough.

### Managed and open-source choices follow team maturity and company context
[06:34](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=394s)
Mariya Davydova describes a pattern in which newer or smaller teams often choose managed services because they want a usable stack without building every part themselves. Larger teams may later move toward open-source components as they hire ML engineers and gain the capacity to operate them. She also observes regional preferences in the markets where she works. Some Russian and Eastern European teams prefer open-source software and are willing to spend months building their own systems, while European and American teams more often accept managed products. Jose adds that Cookpad generally prefers managed services for work that does not directly deliver value, although the decision still depends on the company and the specific case.

### The comparison must include the full cost of building and maintaining a tool
[10:58](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=658s)
Jose says teams often underestimate the engineering effort behind an in-house solution. The calculation includes the cost of building the tool and the continuing work needed to keep it running and relevant. A managed product is worth its price only when the team will use enough of its capabilities to justify that cost. For a small company deploying one or two models, a managed deployment service with features such as canary releases may offer more than the team needs. An open-source option may be sufficient. David Aponte adds that the requirements should describe expected value, deployment-time goals, maintenance ownership, and the expertise available to support the system.

### Scale, migration cost, and team learning time can change the decision
[21:07](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=1267s)
The speakers add several criteria beyond the initial feature list. David says the team should estimate capacity requirements such as data volume, memory, bandwidth, usage, and streaming needs. Jose notes that experiment tracking tools can also run into capacity limits. Mariya says teams should consider the cost of migrating away if the team outgrows the product or the project becomes unsupported. A configuration file may be easy to replace, while a tool that imposes a specific project structure or links together steps in the ML lifecycle can make migration much harder. David also points to the time needed to teach the team, arguing that a Kubernetes-native product is a poor fit when nobody has experience operating Kubernetes.

### Tool ownership and community health are part of production risk
[22:13](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=1333s)
Jose says teams should investigate who stands behind a tool before putting it into production. A startup may run out of money, leaving a production system without a future. The team should ask how close the tool is to production use, whether a company or an active community supports it, whether the maintainers respond to issues, and whether releases are stable. He gives an example of a continuous-delivery tool where a teammate submitted a pull request for a bug and received a release candidate the next day. That kind of response can matter when the team lacks the expertise to solve problems alone. A large community helps, but the team's ability to work with the tool remains part of the risk assessment.

### Cookpad's Feast decision shows why urgent needs create difficult trade-offs
[25:55](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=1555s)
Jose describes Cookpad's feature-store work as a concrete example. The team built a proof of concept with Feast, an open-source project from Gojek, because it looked like a good match for their Kubeflow cluster. Feast had been developed around GCP products, so the team made manual changes to run it on AWS and EKS. The proof of concept worked, but parts of the system were not ready for production on AWS. A planned release was expected to add better AWS support, yet the team needed the capability sooner. Their options were to wait, choose a managed product, or build a small internal feature-store solution. Jose says the team had to weigh the urgency of its stakeholders, the stability of a new release, and its own capacity before planning the next quarter.

### A small internal MVP can reduce dependence when the need is immediate
[34:31](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=2071s)
Mariya says that, in the Feast situation, she would build an MVP because it gives the team control over the immediate solution. If the upstream release arrives on time, the team can migrate later. Waiting for a release means depending on another project, while changing tools introduces a different dependency and migration cost. She presents this as her personal view rather than a universal rule. The choice depends on how much customization the team needs and whether an existing managed product works in its cloud environment. Jose agrees that managed services can still carry risk because early products may fail or lack support for a particular use case.

### Large companies need shared standards without removing team autonomy
[41:57](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=2517s)
The discussion turns to companies with many ML teams using different clouds and tools. Jose says large organizations can end up with teams that do not know about one another, each building a separate stack. Later, a chief architect or similar role may discover that the teams cannot communicate easily and the company must spend months migrating them to a common system. David argues that a central team cannot support every tool, so it may need to define a supported set based on cost and usage. Mariya suggests a shared infrastructure team can define the main platform, tools, roles, and permissions, while DevOps-oriented people inside product teams handle local needs. The speakers accept that standardization can conflict with team independence, so the structure must fit the organization.

### The team's actual users and practices should shape the tool
[55:06](https://www.youtube.com/watch?v=mmTCGkm3ZoQ&t=3306s)
Jose says tool choice must account for the experience of the data science and research teams. If researchers mainly work in Jupyter notebooks and have little engineering experience, the infrastructure team cannot expect them to adopt a complex tool quickly. Cookpad has more flexibility because its researchers can document applications, build HTTP services around models, and write tests. David adds that tools can encode practices such as reusable deployments and separation of runtime environments. The final selection should start with user requirements, use cases, service-level agreements, and the team's ability to use the system. A tool that fits the technical stack but does not fit its users may go unused.

## Notable quotes
- Jose Navarro: "The first thing I like to think is, do we really need to adopt a new tool for solving this problem?" (05:12)
- Jose Navarro: "How much engineering effort is going to cost you to build something?" (11:26)
- Jose Navarro: "The decision has to be made by the end of the week." (32:34)
- Mariya Davydova: "In this particular use case it would be great to build an MVP and then, if they manage to release on time, then great, it will just migrate." (34:31)
- David Aponte: "It should always be about supporting the actual needs of the users." (57:41)

## Tools & references mentioned
- Cookpad
- Neu.ro
- Gojek
- Kubernetes
- Kubeflow
- Feast
- AWS
- GCP
- EKS
- TensorBoard
- Weights & Biases
- Jenkins
- GitHub Actions
- Netflix

## Who should watch
- You are choosing between a managed ML product, an open-source project, and a small internal build, and you need a way to compare their real operating costs.
- Your ML infrastructure team is facing an urgent gap while a promising tool is still missing support for your cloud or production environment.
- Several teams in your company use different ML stacks, and you are deciding how much to standardize without taking away their ability to work independently.

## Editor's note

Jose Navarro warns that separate teams can build incompatible stacks and leave a company with months of migration work. ZenML lets teams keep pipeline code in Python while choosing the orchestrator, artifact store, and other infrastructure through configuration. The same workflow can therefore run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Build vs Buy an ML Platform](https://mlopstalks.com/talks/build-vs-buy-an-ml-platform) (Diego Oppenheimer, Algorithmia, 57:20)
- [ML Platform Tradeoffs and Wondering Why to Use Them](https://mlopstalks.com/talks/ml-platform-tradeoffs-and-wondering-why-to-use-them) (Javier Mansilla, Mercado Libre, 53:57)
- [MLOps Critiques](https://mlopstalks.com/talks/mlops-critiques) (Matthijs Brouns, Xccelerated.io, 49:44)
- [The Shipyard: Lessons Learned While Building an ML Platform](https://mlopstalks.com/talks/the-shipyard-lessons-learned-while-building-an-ml-platform) (Joseph Haaga, Interos, 40:00)
- [Challenges Operationalizing ML (And Some Solutions)](https://mlopstalks.com/talks/challenges-operationalizing-ml-and-some-solutions) (Nathan Ryan Frank, WW Grainger, 52:28)