Jacob Tsafatinos designed Uber's ad event processing system around the needs of its downstream clients, especially pacing and budget updates.
2
Exactly-once processing helped prevent duplicate event handling from causing merchants to be overcharged when Apache Flink jobs restarted.
3
Real-time processing adds complexity, so teams should first establish which customers and systems need fresh data and which can use batch processing.
Summary
Jacob Tsafatinos discusses the real-time ad event processing system he helped build at Uber. The system handled events from ads and supported data scientists, billing, pacing, budget updates, reporting, and advertiser dashboards. Jacob explains why speed, reliability, and accuracy mattered, especially because click data affected how much advertisers were charged and how quickly their budgets were paced. Exactly-once processing reduced the risk of duplicate events after Apache Flink jobs restarted from checkpoints, although different parts of the system used different approaches. He also describes the practical complexity of enabling this across Uber's regional Kafka infrastructure and Pinot's upsert feature. The conversation covers when real-time processing is useful for machine learning and analytics. Jacob's advice is to start with the product and its clients, then decide whether fresh data is needed. He also argues that engineers can learn unfamiliar systems while building them if they have enough support and time.
Band collaboration gave Jacob practice for greenfield engineering
Jacob connects his work in Good Kid with building Uber's new ad system through collaboration rather than through shared technical methods. A band of five people has to work together over a long period while solving different problems. He sees the same needs in a small engineering team doing greenfield work: people have to figure things out as they go, make sure others feel heard, and keep working through uncertainty. The Toronto team had not built this kind of system before, so these team habits mattered. His experience with Spotify analytics also gave him a practical sense of delayed data. He could tell when a reported stream increase probably came from a previous day rather than from the day shown.
The system grew from a greenfield Toronto project into Uber Ads infrastructure
When Jacob joined Uber, the Toronto engineering office was new and was given a greenfield project instead of inheriting a system with existing technical debt. The original project was grocery, but the team moved to ads after Uber acquired another company. The ad system had to support auctions, ad display, bidding, advertiser billing, and processing the events generated by ads. Jacob's blog post focuses on the real-time exactly-once ad event processing system. He describes the opportunity as unusual for his level at the time. He started the project as an L4 and says the system's scope helped him become a senior engineer. The team was small and had to build a substantial part of the platform with limited management support.
Speed, reliability, and accuracy came from the needs of many clients
Jacob starts system design by asking who depends on the system. In Uber Ads, those clients included data scientists, engineers, operations staff, billing systems, pacing and budget updates, and advertisers viewing performance dashboards. Their needs differed. Data scientists could tolerate delayed reports, while pacing needed click feedback quickly because an advertiser's budget depended on how many people clicked. Without timely feedback, a campaign could spend its budget too quickly. Reliability mattered because the system handled customer data and money. Accuracy mattered because incorrect event counts could overcharge or undercharge merchants. Jacob says the team avoided trading away these properties, but the choice produced a more complex system and required extra work.
Exactly-once processing protected accuracy when Flink jobs restarted
Jacob explains that exactly-once semantics were driven mainly by accuracy problems. No Uber team had implemented exactly-once processing before, so the team had to enable it in Kafka topics and Flink jobs while working within Uber's infrastructure limits. Flink jobs can fail for many reasons, such as running out of memory, and a restart reprocesses events from the previous checkpoint. Without protection, that could count the same click again and lead to an advertiser being overcharged. The whole system did not use exactly-once semantics in one identical way. Pinot used an upsert feature so duplicate writes did not create duplicate database records. Hive had no equivalent write-time deduplication, so queries used a primary key or idempotency key to remove duplicates.
Uber's regional Kafka setup created extra work for exactly-once processing
Uber's Kafka infrastructure had local topics for events in individual regions and aggregate topics that combined data across regions. The team could not enable exactly-once processing on the aggregate topics, so it had to apply the approach to the local topics instead. That required an additional Flink job to synchronize data from two regions. Jacob also found that Pinot's upsert behavior depended on messages with the same primary key reaching the same Kafka partition. The partitioner being used at Uber did not always guarantee that, which produced duplicate data during inspection. Fixing the problem required close work with the Kafka and data platform teams and changes to partitioner logic. These infrastructure details were a major source of the system's complexity.
Real-time processing is a spectrum defined by how data is handled
Jacob distinguishes real-time processing from instantaneous results. A system can use checkpoint intervals and event windows, which introduce delay, while still processing data continuously as it arrives. He describes batch processing as collecting data for a period, setting a cutoff, and then processing the bounded set in a warehouse or database. The real-time system does not wait for that cutoff. Instead, it processes incoming events as they arrive. The appropriate delay depends on the product. A data scientist's report may tolerate a longer delay, while budget pacing may need faster feedback. Calling a system real-time therefore requires more detail about its processing model and the latency its users actually need.
Real-time machine learning depends on whether new events change decisions
Jacob says the need for real-time data depends on the prediction and on how much a new event can change its outcome. He gives a recommendation example from Shopify, where a model suggested marketing campaigns for merchants. A product event arriving milliseconds earlier would probably not change that recommendation. Trading systems could be different because each new trade might affect the next decision. He also says that having fresher data can lead to new product ideas. In advertising, a system could detect that one campaign is performing worse than another during its run and notify the advertiser to change its title, description, or image. Those products are difficult to imagine without data that arrives quickly.
Small teams should learn real-time systems while controlling the scope
Asked about a data scientist being pressured to build a real-time fintech system, Jacob says he would learn the technology and build it if the environment allowed learning while working. He read an entire book on Flink, built an initially poor MVP, and then rebuilt the system from scratch. He also stresses the need for support, realistic timelines, and enough people. At Uber, one person was assigned to each core service, and a broader team helped with billing, pacing, consistency, and reliability. With more resources, he might have separated real-time billing from delayed analytics instead of building one broader system. Constraints made the single holistic design easier for that team.
Many engineering mistakes come from missing system context
Jacob recalls once pushing a bug at Amazon that paged everyone in the organization. His review process for incidents is to ask whether the change had the right reviewers, whether the team lacked context, whether tests should have caught it, and whether monitoring or alerting was sufficient. He says many of his mistakes came from misunderstanding how a system operated. His Uber project produced fewer incidents because he had built it from the ground up and understood its internal behavior. The mistakes that did occur there were usually infrastructure problems rather than errors in changes he made. His example makes system knowledge a practical part of safe engineering work, especially when changing unfamiliar infrastructure.
"I usually start going there and then working my way back and then once I know what does the system need to do, then I start thinking, what is the best way to do that?"Jacob Tsafatinos31:52
Who should watch
You are designing an event pipeline and need to decide whether users actually require fresh data or whether batch processing is sufficient.
Your Flink jobs restart and duplicate events could affect billing, budgets, or other financial records.
You are a data scientist or engineer being asked to build a real-time system and need a way to scope the work, learn the tools, and ask for support.
