# Machine Learning at Atlassian

Geoff Sims, Atlassian | MLOps Coffee Sessions | Episode 34 | 58:23
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=MI0hqyYSO3c
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/machine-learning-at-atlassian
Published: 2021-04-12
Tags: feature-engineering, feature-stores, model-serving, platform-teams

## TL;DR
- Atlassian started its machine learning work without the infrastructure or organizational structure that mature ML companies already had.
- The first internal ML platform handled streaming events and online features, but it created a gap between production features and the features used for training.
- Moving to Tecton gave data scientists shared feature definitions, historical feature joins, and online serving without needing an engineer for each deployment.

## Summary
Geoff Sims describes Atlassian's move from scattered analytics work to product machine learning. When he joined, Atlassian did not depend on ML to sell its products, so it had little infrastructure for running jobs or deploying models. The team gradually built an online feature system for product events, but the system became difficult to scale and hard to align with offline model training. Streaming failures could leave features weeks or months behind, while feature definitions had to be recreated separately in SQL or PySpark. Atlassian later formed a central data organization and moved much of this work to Tecton. Geoff says this reduced the need for engineers to translate and deploy features, allowing data scientists to create historical training sets and production features from shared definitions. The remaining bottleneck is model serving, versioning, testing, and deployment. He is candid that Atlassian is still working through an ad hoc product development process.

## Key ideas
### A physics and astrophysics background led Geoff Sims into applied machine learning
[01:17](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=77s)
Geoff Sims studied physics and astrophysics, completed a four-year PhD, and worked on instruments deployed to Antarctica to assess astronomical observing sites. He left academic research after seeing how difficult it was to find stable research work and how often academic careers required moving countries. He joined Quantium, an Australian data science and analytics consultancy, where he worked on retail problems such as predicting what shoppers would buy for pricing and promotional decisions. After several years, he joined Atlassian.

### Atlassian began its ML journey without the infrastructure that mature ML companies took for granted
[04:33](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=273s)
Geoff says Atlassian was a major step backward in data and ML maturity compared with companies whose business models depended on recommendations or AI. Atlassian sold software products and did not need machine learning to operate. One of his first tasks was to run SQL every day and produce a report, but there was no server for a cron job. He learned to deploy microservices simply to schedule jobs, then carried that infrastructure knowledge into later machine learning work.

### The right expectations depend on the organization's maturity
[06:54](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=414s)
Geoff says a newcomer needs to understand what the organization expects from the role. At Atlassian, he had to be adaptable and resourceful because the surrounding systems did not yet exist. His PhD had prepared him for that kind of independent work. He had been given data, unusual formats, and old code and had to figure out how to make everything work. He does not think every data scientist now needs Kubernetes, although a broad skill set can improve employability.

### Product ML turned infrastructure into a two- to three-year engineering effort
[09:41](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=581s)
Geoff first worked in support, where he handled almost the entire production line himself and did not think much about MLOps. Later, Atlassian began adding ML-powered features inside its products, used by hundreds of thousands of cloud customers and millions of active users. The company had almost no systems for this work, so a small group of experienced engineers spent two to three years building an MLOps capability from the ground up. Geoff calls the result both an impressive achievement and an insufficient system for the scale Atlassian wanted.

### Real-time product recommendations required a feature system built around event streams
[21:42](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=1302s)
Atlassian's product models use behavioral signals from cloud product events, with billions of events arriving each day. Geoff gives Jira ticket assignment as an example, where a predictive list could suggest who should receive a ticket based on current work patterns. The features need to change faster than a daily batch because users and projects change. The team transformed events in real time, stored features in a Dynamo store, and made them available to models through APIs and wrappers.

### Separate online and offline feature definitions created a source-of-truth problem
[23:58](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=1438s)
The online feature system could process events and serve features quickly, but it did not solve model training. Data scientists had to recreate the online feature logic in an offline environment to produce historical training data. A feature defined in nested Java code had to be rewritten independently in SQL, PySpark, or another language. The two versions could be close without being guaranteed to match. Geoff says this led to duplicated definitions, confusing terminology, and difficulty answering even basic questions about which features a model used.

### Streaming failures exposed the limits of the home-built platform
[31:52](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=1912s)
As the team added more models, it found problems with streams choking, catching up, alerting, and autoscaling. The models could perform badly when the feature pipeline suffered a disruption. In some cases, streams became weeks or months old. Restarting was risky because pulling the plug on a streaming system could mean starting again from the beginning. The team eventually decided that the system was becoming unmanageable, even though building it had been a significant engineering accomplishment.

### Tecton gave data scientists shared features and reduced dependence on engineers
[36:41](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=2201s)
Atlassian investigated feature store and MLOps platforms and moved to Tecton. Geoff says its architecture was similar to what Atlassian had built, including streaming data, feature transformation, and Dynamo storage, but Tecton had more mature operational support. It provided feature configuration, a UI, logging, tracking, monitoring, and autoscaling. Data scientists could define features, join them onto historical labelled data, and use the same definitions for online serving. Geoff says this removed much of the engineering work needed to turn a training feature into a production feature.

### Once features stopped being the bottleneck, model deployment became the next problem
[44:37](https://www.youtube.com/watch?v=MI0hqyYSO3c&t=2677s)
Geoff describes ML platform work as a sequence of bottlenecks. After solving the feature problem, Atlassian's slower step became model libraries, serving, versioning, and deployment. Models still needed to be posted somewhere, switched on or off, and tested in A/B experiments. The team ran predictions in-house because the volume could make SageMaker expensive, and it was also considering Databricks as it expanded its ML tooling. Geoff rejects the assumption that data scientists are always the main constraint, since training-set curation and representative sampling remain difficult.

## Notable quotes
- Geoff Sims: "You certainly need to understand the maturity of the organization as to what should be expected of you." (07:54)
- Geoff Sims: "It took us three years to build all the supporting infrastructure and everything you needed to run one model." (32:12)
- Geoff Sims: "There are always bottlenecks of problems. There's something that's a bottleneck, and you look for where that bottleneck is, and then that's what you fix." (45:44)
- Geoff Sims: "The single biggest thing is independence from a data science perspective." (41:22)

## Tools & references mentioned
- Atlassian
- Jira
- Confluence
- Trello
- Quantium
- Walmart
- Hadoop
- Dynamo
- Tecton
- Databricks
- SageMaker
- PySpark
- MLOps Community

## Who should watch
- You work at a product company where machine learning is useful but is not the main business priority, and you need to set realistic expectations for data scientists.
- Your team is building online features from high-volume event streams and is struggling to keep training data aligned with production data.
- You have solved feature generation and now need to decide how to handle model serving, versioning, A/B testing, and deployment.

## Editor's note

Geoff Sims describes how Atlassian had to maintain separate online and offline feature definitions, so the features used in training were not guaranteed to match production. ZenML lets teams define workflows as Python pipelines whose runs record inputs, outputs, and code version. Its lineage makes artifacts traceable to the data and code that produced them, while the same pipeline can run on different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Lightweight Feature Platform](https://mlopstalks.com/talks/lightweight-feature-platform) (Matt Bleifer & Mike Eastham, Tecton, 1:03:58)
- [AI Innovations: The Power of Feature Platforms](https://mlopstalks.com/talks/ai-innovations-the-power-of-feature-platforms) (Mahesh Murag, Tecton & Jose Navarro, Cleo & Nikhil Garg, Fennel, 1:05:28)
- [Machine Learning at Reasonable Scale](https://mlopstalks.com/talks/machine-learning-at-reasonable-scale) (Jacopo Tagliabue, Coveo, 1:04:32)
- [Maturing Machine Learning in Enterprise](https://mlopstalks.com/talks/maturing-machine-learning-in-enterprise) (Kyle Gallatin, Etsy, 47:09)
- [MLOps at Volvo Cars](https://mlopstalks.com/talks/mlops-at-volvo-cars) (Leonard Aukea, Volvo Cars, 57:31)