# Building Say Less: An AI-Powered Summarization App

Yoav Zimmerman, Model Zoo | MLOps Meetup | Episode 32 | 53:08
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=RoYl0waMfMc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-say-less-an-ai-powered-summarization-app
Published: 2020-09-08
Tags: build-vs-buy, deployment, model-serving

## TL;DR
- Say Less uses an off-the-shelf abstractive summarization model to shorten long emails without custom training or GPUs.
- Yoav Zimmerman moved from a small Python prototype to a real-time HTTP API, then built a Chrome extension that integrates with Gmail.
- The main model weakness comes from transferring a model trained on news articles to email, especially because the model favors the beginning of a message and lacks email-specific training data.

## Summary
Yoav Zimmerman walks through Say Less, a tool that shortens long emails. He starts with the product idea, explains extractive and abstractive summarization, and describes why he chose Facebook's BART model through Hugging Face. The first prototype took about ten lines of Python and ran locally without training or GPUs. He then deployed the model as an HTTP API using Model Zoo, whose architecture uses AWS API Gateway, AWS Lambda, Amazon Elastic Container Service, a load balancer, CloudWatch metrics, and Amazon S3 for model artifacts. A Chrome extension provides a Gmail integration and only appears when an email exceeds 100 words. Yoav is direct about the limitations: the model was trained on CNN and Daily Mail articles, so it favors the beginning of emails and sometimes produces poor summaries. He says an email dataset with human summaries would improve the system, though privacy makes collecting one difficult.

## Key ideas
### Say Less began with a simple email-length problem
[02:05](https://www.youtube.com/watch?v=RoYl0waMfMc&t=125s)
Yoav Zimmerman frames Say Less around a common annoyance: long email. The product lets someone writing an email trigger a shorter version of the message. His idea was influenced by the Five Sentences email manifesto and by Greg Kogan's post, "To Get More, Apply to Say Less." Yoav says Kogan found that cutting email length by 90 percent increased conversions and increased replies by eight times in a particular email-marketing use case. This gave him evidence that shorter email could be useful beyond personal preference. He presents the project as an exercise in taking an idea from initial observation through a working production application.

### The first version used an existing abstractive model instead of custom training
[02:58](https://www.youtube.com/watch?v=RoYl0waMfMc&t=178s)
Yoav separates summarization into extractive and abstractive approaches. Extractive systems remove material from the original passage, while abstractive systems synthesize a new summary and can produce words that were absent from the source. He chose Facebook's BART because he could not find an easy-to-use Pegasus implementation at the time. The model had benchmarks for abstractive summarization, and Hugging Face provided a public version fine-tuned on the CNN and Daily Mail dataset. That dataset contains about 300,000 news articles with editor-written summaries. Yoav's prototype used the Hugging Face pipeline and required about ten lines of Python.

### A small local prototype was enough to test whether news-trained summarization transferred to email
[13:50](https://www.youtube.com/watch?v=RoYl0waMfMc&t=830s)
Yoav did not assume that a model trained on news articles would work for email. He ran several personal emails through the prototype on his own computer. The model sometimes chose the wrong sentences or made mistakes, but it worked well enough to justify building a product around it. On his computer, producing a result took a few seconds, and he says it could run on a MacBook. This prototype stage gave him a practical answer before he invested in custom data collection or model training. It also exposed a limitation that became clearer after the product was released: the source and target domains had different writing patterns.

### Model Zoo turned the model into an HTTP endpoint
[14:52](https://www.youtube.com/watch?v=RoYl0waMfMc&t=892s)
After the local prototype, Yoav deployed the model as an HTTP API and built the Chrome extension on top of it. Model Zoo provides a client library where the user specifies deployment resources while the platform handles the back end. Its AWS architecture routes requests through API Gateway and Lambda, uses an API path for deployment and a prediction API for applications, and runs model containers on Amazon Elastic Container Service behind a load balancer. Prediction calls send metrics to CloudWatch, and model files are stored in Amazon S3. The example API accepts an input string and an API key, then returns a generated summary.

### Model Zoo gets simplicity by constraining the model interface
[28:25](https://www.youtube.com/watch?v=RoYl0waMfMc&t=1705s)
Yoav explains that Model Zoo can make deployment close to a single-line experience because it limits the supported model patterns. For a Hugging Face model using the library's pipeline abstraction, the platform can know which Docker image to use, which code should run inside it, what HTTP schema the service needs, and which resources may be appropriate. The container downloads model artifacts from Amazon S3 in an expected format, loads them with custom web-service code, and runs on Amazon Elastic Container Service. Yoav says this approach differs from general-purpose open-source deployment frameworks because it trades flexibility for ease of use.

### The Chrome extension hides the feature until an email is long enough
[20:24](https://www.youtube.com/watch?v=RoYl0waMfMc&t=1224s)
Yoav says the product needed a front end that people without machine learning experience could use. Say Less has a pop-up interface and an integration inside Gmail. The Gmail version avoids interrupting users who are writing short messages. Its front end tokenizes the email and shows the widget only when the message exceeds 100 words, an arbitrary threshold that roughly corresponds to five sentences. Even when users do not use the summarization service, the prompt can remind them that an email has become long. Yoav built the interface with React, jQuery, and the open-source gmail.js library, which helps bind code to Gmail's changing DOM elements.

### The project reached production in about two weeks by choosing low-effort components
[24:18](https://www.youtube.com/watch?v=RoYl0waMfMc&t=1458s)
Yoav estimates that the full project took about two weeks, with most of his time going into the front end because it was less familiar to him. He was also working on Model Zoo with customers, so this was part-time effort. His method was to find the lowest-hanging fruit: use a model someone had already trained, use Model Zoo for deployment, and use gmail.js rather than building the Gmail integration from scratch. He describes this as getting 80 percent of the result with 20 percent of the effort. The prototype was the point where he decided the idea could become a real product.

### The model's main failure comes from the gap between news articles and emails
[41:00](https://www.youtube.com/watch?v=RoYl0waMfMc&t=2460s)
Yoav says users found that Say Less did not always select the right content. The model also favors the beginning of an email, which he connects to the structure of CNN articles, where the first paragraph often summarizes what follows. Email does not follow that distribution, so information is lost when a news-trained model is transferred to email summarization. A better system would need real email data and human-written summaries. That is difficult because email is privacy-sensitive. The public Enron email dataset provides messages, but Yoav points out that it does not readily provide the corresponding summaries. He says Say Less does not log or retain users' email text and throws it away immediately.

## Notable quotes
- Yoav Zimmerman: "The basic idea is you're writing an email, it might be getting long, you do this thing and it gets shorter." (02:26)
- Yoav Zimmerman: "The broad point here is that thanks to these new deep learning architectures abstractive summarization is actually starting to get kind of good enough." (09:14)
- Yoav Zimmerman: "Can you get to 80% of the result with 20% of the effort?" (25:03)
- Yoav Zimmerman: "The model favors the beginning of the email over the end of it." (41:48)
- Yoav Zimmerman: "I don't log any of your email text on my servers or in my logs or anywhere, I throw it away immediately." (44:19)

## Tools & references mentioned
- Say Less
- Model Zoo
- Facebook BART
- Pegasus
- Hugging Face
- CNN/Daily Mail dataset
- Five Sentences
- Greg Kogan
- Google
- Determined AI
- Product Hunt
- Hacker News
- AWS API Gateway
- AWS Lambda
- Amazon Elastic Container Service
- Amazon CloudWatch
- Amazon S3
- React
- jQuery
- gmail.js
- GPT-3
- OpenAI
- OtherSideAI

## Who should watch
- You are building a small machine learning product and need a practical path from an existing model to a usable application.
- You want to understand what an ML deployment platform can automate when it supports a narrow model interface.
- You are deciding whether a prototype is worth extending before collecting private, domain-specific training data.

## Related talks

- [Using Large Language Models at AngelList](https://mlopstalks.com/talks/using-large-language-models-at-angellist) (Thibaut Labarre, AngelList, 51:42)
- [Small Models, Big Ideas: The Next Frontier in AI](https://mlopstalks.com/talks/small-models-big-ideas-the-next-frontier-in-ai) (Korri Jones, Chick-fil-A Corporate Support Center & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Lihu Chen, Imperial College London & Binoy Perera, MLOps Community, 58:44)
- [Making LLM Inference Affordable](https://mlopstalks.com/talks/making-llm-inference-affordable) (Daniel Campos, Snowflake, 32:07)
- [DSPy: Transforming Language Model Calls into Smart Pipelines](https://mlopstalks.com/talks/dspy-transforming-language-model-calls-into-smart-pipelines) (Omar Khattab, Stanford, 1:05:40)
- [LLMs Mini Summit // MLOps Mini Summit Meetup #3](https://mlopstalks.com/talks/llms-mini-summit-mlops-mini-summit-meetup-3) (Thomas Capelle, Weights & Biases & Boris Dayma, Craiyon & Jonathan Whitaker & Robbie McCorkell, Leap Labs, 1:45:28)