# Vector Similarity Search at Scale

Dave Bergstein, Pinecone | MLOps Coffee Sessions | Episode 52 | 49:48

Source: https://www.youtube.com/watch?v=Gh-kZ4tHzNE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/vector-similarity-search-at-scale
Published: 2021-08-31
Tags: build-vs-buy, embeddings, platform-teams, search

## TL;DR
- Vector similarity search turns model-generated embeddings into a way to find similar images, products, or pieces of text.
- Production vector search needs approximate nearest-neighbor algorithms, low latency, memory management, sharding, replicas, persistence, fault tolerance, and live updates.
- Platform teams should define clear boundaries and support interoperability with other tools instead of trying to bring every capability in-house.

## Summary
Dave Bergstein explains how Pinecone provides managed vector similarity search for production applications. Neural networks can turn images or text into vectors whose distances capture useful similarities. Dave uses a photo of his dog Zeus to explain how an image model's final layer can find visually similar dogs, then describes applications such as product recommendations, duplicate-person detection, and semantic text search. The engineering problem begins when a brute-force search over thousands, millions, or billions of vectors must still respond in milliseconds. Teams need approximate nearest-neighbor methods, distributed storage, replicas, persistence, fault tolerance, update handling, and enough RAM to keep searches fast. Dave also discusses his experience moving from hardware and scientific software into product work. His lesson for platform teams is to decide what belongs inside the platform, leave room for an ecosystem, and make interoperability a product requirement.

## Key ideas
### Hardware production taught Dave to think about software failure as a systems problem
[05:10](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=310s)
Dave traces his production mindset to manufacturing electronic ballast products. The factory produced about 100,000 units a year of the product he worked on, while the wider factory produced millions of products. He remembers a quality engineer showing the team roughly 200 components and asking them to consider the combinations created by each component's tolerances. Software systems have a similar source of complexity: many parts interact, and the possible failure cases multiply. That experience shaped how he thinks about taking a prototype toward something reliable at scale.

### Embeddings turn images and text into numbers that can support similarity search
[08:41](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=521s)
Pinecone provides managed vector similarity search. Dave explains that a neural network can reduce an image or sentence to a set of numbers that captures useful properties learned during training. In his Zeus example, he takes the final fully connected layer of an image model trained to distinguish dogs, cats, birds, and elephants. The resulting roughly 1,000-number vector captures features that help find other dogs that look like Zeus. Similarity can be measured with Euclidean distance, dot product, or cosine distance.

### Semantic vectors let search match meaning across products and reviews
[25:27](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=1527s)
Dave describes several applications beyond finding similar dog photos. A networking platform can compare images to detect cases where the same person may appear in multiple profiles. E-commerce systems can recommend visually similar products. Text embeddings can match a shopper's description of soft, light-colored khakis with product information and reviews. A query about something suitable for the office can match a review saying that an item is good for work, even when the exact word 'office' is absent.

### The production gap appears when data science prototypes must meet application latency
[19:06](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=1146s)
Dave describes a common customer path. A data science team builds embeddings and demonstrates what is possible, then works with DevOps or an MLOps team to put the system into production. A brute-force dot-product search may run in milliseconds for a hundred images on a laptop, but it slows significantly at thousands of images. Customer-facing systems may need around 100 milliseconds for responsiveness, and Dave says Pinecone has gone down to 10 or 5 milliseconds. That requirement pushes teams toward approximate nearest-neighbor search.

### Scaling vector search requires database behavior around the search algorithm
[23:07](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=1387s)
Open-source approximate nearest-neighbor libraries solve only part of the problem. Teams still need to distribute the index across machines with shards and replicas, then provide database operations such as create, read, update, and delete. Production systems also need persistence, fault tolerance, eventual consistency, and live updates. Dave says vector data usually needs to stay in RAM for fast search, even though it is much smaller than the original images. RAM and cloud compute become significant costs when the system stores billions of vectors.

### Index updates and software updates need separate operational machinery
[28:28](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=1708s)
Dave distinguishes updates to vector or metadata from software updates. Pinecone supports different engine types, including FAISS, HNSW, and its own proprietary engine. Dave says the proprietary engine can update vectors and rebuild the index live in tens of milliseconds, while HNSW updates can disturb its graph structure. Pinecone therefore builds machinery to swap in new vectors and indexes while the service is running. Similar mechanisms can support software updates through Kubernetes. He also describes private connectivity through AWS PrivateLink and says security is a major part of the service.

### Platform teams need to choose boundaries and preserve interoperability
[39:59](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=2399s)
From his MathWorks experience, Dave says a platform must decide what belongs inside it and what it will work with from outside. MATLAB grew an ecosystem in which people built tools on top of the platform, and MathWorks also brought some toolboxes in-house to improve performance, consistency, and features. Dave presents this as a trade-off rather than a universal rule. His preference is to keep interoperability strong, since platforms can benefit from innovation happening elsewhere. Pinecone follows a similar approach by supporting multiple approximate-nearest-neighbor engines instead of relying only on its proprietary engine.

### Vector search will keep splitting between tailored internal systems and managed services
[46:10](https://www.youtube.com/watch?v=Gh-kZ4tHzNE&t=2770s)
Dave expects the underlying libraries to keep advancing through active research. Large companies such as Facebook, Google, and Amazon can build and tailor their own systems because they have the incentive and resources to do so. Other companies may prefer a managed service that spreads the cost of expertise across many customers. Dave does not expect one solution to fit every use case. Tailoring and optimization can reduce compute cost, so there is room for several providers and for systems tuned to different workloads.

## Notable quotes
- "It's similarity search at scale." (08:41)
- "The fastest way to do this is to keep all your vector data, not necessarily the images, but the vector data, you want to keep it in RAM." (25:23)
- "I think one thing I think of when I think a platform is interoperability with other platforms." (39:59)
- "I don't know that we'll see one kind of solution that fits all." (47:38)

## Tools & references mentioned
- Pinecone
- MLOps Community
- Lutron Electronics
- Boston University
- Tesseract
- MathWorks
- MATLAB
- The Cathedral & the Bazaar
- BERT
- Flickr
- Amazon
- Google
- Elasticsearch
- Kubernetes
- FAISS
- HNSW
- AWS PrivateLink
- Python
- Facebook

## Who should watch
- You are considering semantic search or recommendations and need to understand what changes when embeddings move from a laptop to a live product.
- Your data science team has working embeddings, while the DevOps or MLOps team is worried about latency, RAM cost, updates, and distributed operation.
- You are designing an internal platform and need to decide which capabilities to build while keeping the system compatible with outside tools.

## Related talks

- [Using Vector Databases: Practical Advice for Production](https://mlopstalks.com/talks/using-vector-databases-practical-advice-for-production) (Sam Partee, Redis, 29:58)
- [Why Purpose-built Vector Databases Matter for Your Use Case](https://mlopstalks.com/talks/why-purpose-built-vector-databases-matter-for-your-use-case) (Frank Liu, Jiang Chen & Yujian Tang, Zilliz, 59:01)
- [Information Retrieval & Relevance](https://mlopstalks.com/talks/information-retrieval-relevance) (Daniel Svonava, Superlinked, 56:05)
- [Vector Databases and Large Language Models](https://mlopstalks.com/talks/vector-databases-and-large-language-models) (Samuel Partee, Redis, 13:10)
- [The Vector Search Engine Powering AI Agents](https://mlopstalks.com/talks/the-vector-search-engine-powering-ai-agents) (Thierry Damiba, Qdrant, 27:45)