Pinterest moved image similarity detection from a mature batch pipeline toward near-real-time processing with Apache Flink while keeping the consumer-facing behavior as stable as possible.
2
The system generates embeddings, searches an existing image corpus through Manas, expands the candidate set, scores candidates with a TensorFlow model, and stores cluster relationships across several storage systems.
3
Debug metadata, batch-versus-stream comparisons, rollback controls, and scheduled consistency checks are built into the system because real-time processing removes the global validation checks available in batch jobs.
Summary
Shaji Chennan Kunnummel explains Pinterest's near-real-time system for finding exact and near-duplicate images. A newly uploaded image is represented by embeddings, converted into LSH terms, searched against an existing corpus through Manas, and scored by a TensorFlow model. The result assigns the image to an existing cluster or creates a new one. Apache Flink coordinates the stream workflow, while RocksDB supports point lookups, graph storage holds cluster membership, and the search index is updated for later queries. The difficult part is moving away from batch processing. Batch jobs provide snapshots and make it easier to compare totals and detect inconsistencies. Streaming needs those checks, debugging data, rollback paths, and recovery procedures during processing. Shaji describes a gradual move from manual bootstrapping toward automation, along with separate short-term and long-term monitoring. The design is being generalized beyond images and is moving toward a unified interface for consumers.
Pinterest treats production ML signals as changing dependencies
Shaji says signals come from different teams and can change without the consuming team knowing. A field may be duplicated, a new field may appear, or a score can change from a 0-to-100 range to a 0-to-1 range. Pinterest therefore needs systems that detect issues as they happen, alert people, and support an actionable response. Depending on the incident, the response may be to stop serving data or roll back to a known good state. The team also wants metadata that explains where a signal came from and what transformations were applied.
The move from batch to streaming removes useful validation checks
The batch pipeline had run for years and had mature optimizations, monitoring, and many downstream consumers. The streaming replacement had to preserve the consumer experience while making historical data available to a serving system. Batch processing could validate a complete snapshot, compare one day's record count with the previous day, and spot an unexpected jump. Real-time processing has no simple whole-corpus check, so monitoring and debuggability have to be part of the pipeline itself. The system also needs a way to identify bad time ranges and roll back to a safe state.
Bootstrapping requires consistent population of several storage systems
The initial bootstrap populated search indexes, graph storage, and the RocksDB-based lookup system. Teams coordinated write rates, sometimes agreeing on a rate such as 10,000 queries per second, but the process could stall when another cluster experienced high CPU usage. The team first moved from rebuilding everything to calculating deltas, then automated the delta patching. Historical data is required because new images are compared with the existing corpus, and the same data must be present and consistent across the storage systems before real-time processing starts.
The pipeline turns embeddings into candidates and clustered images
For each new image, Pinterest receives embedding notifications through Kafka. After the required embeddings are available, the system generates LSH terms and sends them to Manas, the custom search engine. Manas returns roughly 200 to 300 results, after which a candidate enhancer expands the set by about 10 times. The candidates go to the Scorpion model hosting service, where a TensorFlow model produces a normalized score between zero and one. The highest-scoring candidate determines the cluster assignment. If no suitable cluster exists, the image can become its own cluster.
The search index requires a minimum number of overlapping LSH terms and orders results by the number of overlaps. Shaji says missing a small fraction of results is acceptable in many search systems, but it is much less acceptable here because the result affects duplicate detection. Pinterest therefore sacrifices some latency for correctness. The search engine can take a few seconds for larger searches. Candidate enhancement adds similar images to the result set so the TensorFlow model sees more variation before producing its scores.
Debug information makes batch and streaming differences explainable
Pinterest keeps metadata about search result counts, overlap counts, candidate-enhancement choices, model scores, and backend errors. This lets the team distinguish a scoring problem from a backend failure. The design also supports automated tracing from a difference between batch and real-time outputs back to the event where the divergence began. Some differences are expected because result tails can be nondeterministic or because floating-point precision differs. Debug data is retained in Kafka and S3, while some metadata is removed before writing to serving storage so payloads do not become too large.
Storage consistency is checked after the stream has written its results
The system uses RocksDB for low-latency point lookups, graph storage for paginated cluster membership, and the Manas index for search. Each system has its own consistency and replication behavior. Hourly and daily jobs compare the final results and find records that have gone out of sync, after which scripts patch the differences. Kafka-derived hourly tables are retained for 90 days. Daily snapshots from graph storage and RocksDB support longer-term comparisons and make it possible to find the delta between successive days.
The next stage is broader entity support and safer consumer access
Shaji describes plans to add a CI/CD workflow, generalize the architecture beyond images, and handle near-duplicates in formats such as idea pins. Pinterest is also working toward a single consumer interface. Historically, some consumers queried a signal delivery service while others accessed storage directly. A unified interface would let the team change the underlying storage without requiring each consumer to perform an expensive migration. Another planned capability is to stop serving data for a damaged time range while continuing to process new Kafka input.
"The debug ability and then the monitoring part, it actually has to be built right into the system."12:29
Who should watch
You are designing a streaming ML pipeline that has to replace or coexist with a batch system and need to think through bootstrapping, replay, and recovery.
Your model depends on signals or services owned by other teams, and you need concrete patterns for debugging changes and tracing bad outputs.
You operate several serving stores with different consistency models and want to compare hourly results with longer-term snapshots.
