Ghostwriter uses a user's past emails to reproduce their writing style and include facts relevant to the current draft.
2
Shortwave finds similar emails with semantic search, adds them to a prompt, and asks an LLM to copy their style and content.
3
Autocomplete required fine-tuning because general LLM instructions could not reliably handle continuation, spacing, punctuation, and stopping at the end of a draft.
Summary
Jonny Dimond explains how Shortwave built Ghostwriter, an AI writing system for email that learns from a user's own messages. The system embeds incoming emails, stores them in a vector database, and retrieves similar messages when the user replies to a thread or starts a draft. Those examples give the model both the user's writing style and facts that belong in the response. Shortwave uses GPT-4 for drafting and a fine-tuned model for autocomplete, where output formatting and continuation behavior need tighter control. Dimond says a few hundred synthetic examples produced a large improvement. The team also added examples for fact lookups and cases where information is missing, so the model would avoid inventing an answer. He describes future work on ranking, snippets, multiple embeddings, more training data, and per-user fine-tuning. Shortwave requires user confirmation before sending emails or taking other actions.
Ghostwriter has to sound like the user and write the right content
Shortwave found two separate problems in AI email writing. The generated message has to sound like the person who will send it, since a generic AI email is easy to reject. It also has to contain the information needed for the actual reply. A long block of plausible text is not useful if it leaves out the answer. Dimond describes early attempts that gave the model a description of the user's style, including sentence structure, emoji use, salutations, sign-offs, and technical language. Fine-tuning on a user's emails worked well too, although applying it at Shortwave's scale was difficult. The team then found that examples of the user's own writing could provide both style and content.
Relevant past emails give the model style and facts at the same time
Shortwave's approach is to retrieve emails that show how the user writes and contain information relevant to the current message. The prompt tells the LLM to copy the style and content from those examples and generate a reply. For a draft about a promo code, the system can retrieve earlier messages about promo codes. In the demo, it used a code the user had described in an earlier email and inserted the actual offer instead of inventing one. The same mechanism supports fact lookups, such as finding an office Wi-Fi password and adding it to a reply. Dimond's point is that retrieval examples reduce the need for users to write detailed instructions themselves.
Semantic search turns a user's mailbox into a source of writing examples
Shortwave indexes the user's emails by running them through an embedding model and storing the results in a vector database. When the user replies to a thread or starts a draft, the current thread or draft goes through the same embedding model. The system searches for emails with similar meaning, then selects roughly five to ten messages, depending on their length and other conditions. It combines those examples with a system prompt and user preferences such as language or preferred sign-off. The resulting prompt goes to an LLM, which produces the draft. Dimond says the same underlying search infrastructure also supports Shortwave's AI search feature.
Autocomplete needed fine-tuning because prompting could not control its output reliably
Shortwave uses GPT-4 for ordinary assistant drafts, but fine-tuned a model for autocomplete. Autocomplete must continue exactly where the user stopped. If the user types "happy to", the model should add the next text rather than repeat those words. The team also had to handle spaces, punctuation, mid-paragraph continuation, and the point where a completed draft should produce no further suggestion. System-prompt instructions did not make this behavior reliable. Dimond says the team saw a large improvement with only 400 or 500 examples. They created training data from real emails by hiding different portions of each message and varying the cursor position, including cases at the start, middle, and end of a draft.
Synthetic examples teach the model when to include facts and when to stay silent
The training data included handwritten examples for factual information such as a Wi-Fi password, an office address, or a phone number. These examples taught the model to include a fact when it appeared in the available context. Shortwave also added examples where the required information was missing. Those cases taught the model not to make up an answer. This matters because the autocomplete feature is expected to insert concrete content into an existing draft, rather than only produce general prose. The promo-code demonstration used the same idea: the suggestion came from information previously written by the user. Dimond presents fact handling as part of the writing system, rather than as a separate text-generation problem.
The system is built so individual components can be replaced
Dimond says Shortwave tries to make each part of the pipeline replaceable because models and techniques change quickly. The team does not want to depend permanently on one embedding model. It has infrastructure that can switch to another model, even though changing models requires updates and computation. The email cleaning pipeline is another important part of the design. It extracts the actual text from messages and replies, giving the team a cleaner representation for search and fine-tuning. Dimond describes this as a practical lesson from building the product over several years. The team uses both open-source models and OpenAI models, choosing what fits the task.
User confirmation limits what the AI can do with sensitive email data
Shortwave's AI can read past email content and use it in future suggestions, which raises the risk of exposing sensitive information in a draft. Dimond says the product follows a rule of requiring user confirmation before actions. The system writes the email but does not send it. It may suggest an event or calendar action, but it does not add it automatically. He connects this restriction to the risk of hallucinations and prompt injection. The user remains the final check before the product takes an external action. Dimond also says Shortwave is available as a full email client for Gmail accounts, while Outlook and Microsoft 365 support was not available at the time of the talk.
The team wants better retrieval, snippets, and user-specific training
Dimond lists several directions for improving Ghostwriter. Shortwave could train on more varied examples, writing styles, languages, and larger datasets. It could use whether a user accepts a suggestion as a training signal. Per-user fine-tuning had become possible enough to investigate, although it was previously too expensive. Retrieval could also improve beyond basic similarity ranking, including cross-encoding and more focused ranking methods. The current system includes whole emails, while future versions could retrieve smaller snippets for fact lookups. Multiple embeddings per email could capture different meanings. Dimond also says email is messy and that data cleanup remains a substantial area of work.
"If you're able to find emails that both show how the user writes and contains relevant information, then it's able to produce emails that sound very much like you and include the right content."06:11
Who should watch
You are building an AI writing feature and need retrieval to provide both a user's style and the facts needed in a reply.
You are working on autocomplete and need examples of how to train a model to continue text, stop at the right point, and format its output.
You are designing an email assistant that can access sensitive data and want a concrete approach to user confirmation before external actions.
