# LLMs as Intelligent Assistants

Sarah Aerni, Salesforce | LLMs in Production 2023 | 28:45

Source: https://www.youtube.com/watch?v=E0929WqB72k
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/llms-as-intelligent-assistants
Published: 2023-06-29
Tags: guardrails, human-in-the-loop, privacy, tool-use

## TL;DR
- LLMs can assist people across sales, service, marketing, commerce, and development by working inside existing business workflows.
- Salesforce grounds generated content in customer data and keeps people responsible for reviewing, editing, and approving the output.
- Secure data separation, masking, toxicity detection, explainability, and human review are central to deploying these assistants.

## Summary
Sarah Aerni presents LLMs as assistants embedded in Salesforce workflows rather than as fully autonomous replacements for employees. She walks through examples for sales, service, marketing, analytics, and development. A seller can summarize a new account, create contacts, draft an email, adjust its tone, and add a Slack link. Service staff can receive suggested replies based on knowledge articles and generate a case summary or new article. Marketers can create and edit landing-page content and images. Developers can generate code and test scaffolding from comments. Aerni also explains Salesforce's AI Cloud architecture and its trust controls, including secure retrieval, data masking, toxicity detection, tenant separation, and human review. She is direct that generated output must be checked for accuracy. Salesforce's approach keeps customer data separate and uses it to ground responses without retaining it to train or fine-tune models, according to Aerni.

## Key ideas
### Customer expectations make AI assistance part of the standard product experience
[01:28](https://www.youtube.com/watch?v=E0929WqB72k&t=88s)
Aerni says customers now expect fast responses, personalized content, and automation for time-consuming tasks. She cites a survey in which 62% of respondents expect businesses to anticipate their needs. At the same time, she says 76% of executives still struggle to deploy AI in production. She attributes the gap to data quality, integration problems, and the difficulty of connecting new technology to existing processes. Her point is practical: an assistant has to fit into the systems and work that employees already use.

### Salesforce presents LLMs as a broad extension of its existing AI platform
[04:08](https://www.youtube.com/watch?v=E0929WqB72k&t=248s)
Salesforce already ships what Aerni corrects as 1 trillion predictions per week, including opportunity scores and case labels for triage. She describes earlier investments in machine learning research, acquisitions such as RelateIQ, PredictionIO, and MetaMind, and work on automated machine learning, forecasting, and language models. LLMs expand access because developers and business users can try natural-language interactions themselves. Aerni says this makes it easier for people to imagine where an assistant could reduce effort or help them work faster.

### The AI Cloud architecture places language models inside a controlled platform
[08:00](https://www.youtube.com/watch?v=E0929WqB72k&t=480s)
Aerni describes Salesforce's AI Cloud as a unified architecture built on Hyperforce, with a platform layer that gives developers low-code tools. Large language models are embedded into that platform, allowing builders to create applications and Salesforce to ship its own applications above the same foundation. She frames trust as part of the architecture rather than a later add-on. The controls she names include secure retrieval, use of customer data, masking, and toxicity detection.

### Sales assistants can turn account research and outreach into an in-product conversation
[10:49](https://www.youtube.com/watch?v=E0929WqB72k&t=649s)
In the sales demonstration, a seller asks about a newly transferred account and receives a summary with recent expansion news. The assistant can update the account description, identify people connected to the expansion, create a new contact, and draft an email grounded in Salesforce information. The seller can ask for a shorter or less formal version, then add a link for connecting in Slack. Aerni's example keeps the seller in control while removing several searches and manual drafting steps from the workflow.

### Service assistants reuse conversations and knowledge articles to speed case handling
[13:35](https://www.youtube.com/watch?v=E0929WqB72k&t=815s)
For a question about the temperature rating of a K3 Alpine Jacket, the assistant retrieves relevant information and proposes a reply that the representative can edit. It can also summarize the interaction when the case is closed. Aerni says the same conversation can produce a knowledge article for future cases. The workflow therefore helps with the immediate response and can add reusable material for the wider service team, subject to review.

### Marketing and developer assistants support iterative creation rather than one-shot generation
[14:35](https://www.youtube.com/watch?v=E0929WqB72k&t=875s)
The marketing example creates a landing page for a VIP store opening, then lets the user revise the campaign message, add information about meeting the US Olympic team, generate a snowy header image, add a title, and add a form. In the developer example, comments describe a controller that retrieves up to 10 contacts, and the assistant generates code with the requested invocable behavior. It also generates test scaffolding. Aerni presents both workflows as cycles of asking, editing, and checking.

### Human review remains necessary because assistance does not guarantee correct output
[23:35](https://www.youtube.com/watch?v=E0929WqB72k&t=1415s)
Answering a question about replacing or augmenting workflows, Aerni says human involvement is critical for checking accuracy, reviewing generated material, and deciding whether it can be trusted. She also points to explainability for traditional and newer models, including reports about the data driving predictions and the accuracy of individual predictions. Assistants can surface knowledge articles or create drafts, but people still curate the results and handle complex requests.

### Customer data is kept separate and used for grounding rather than model training
[26:49](https://www.youtube.com/watch?v=E0929WqB72k&t=1609s)
Aerni says Salesforce treats the customer's data as the customer's own data, not as Salesforce's product. Models and builders operate on a customer's data, with data kept separate between organizations. She says customer data is not retained to train or fine-tune models. Instead, the system retrieves relevant information from the customer's data and uses it to generate content for that customer. She presents separation, relevance, and trust as connected parts of the design.

## Notable quotes
- "The assistant is helping you get through this task quickly." (12:50)
- "The importance of humans in the loop cannot be understated." (24:24)
- "Your data is yours. Your data is not our product." (26:49)
- "Your customer, your data, it's not being retained to train and fine-tune models." (27:54)

## Tools & references mentioned
- Salesforce
- Einstein GPT
- Tableau
- Hyperforce
- RelateIQ
- PredictionIO
- MetaMind
- Slack
- Marketing Cloud
- Einstein
- AI Cloud

## Who should watch
- You are designing LLM features for sales, service, marketing, analytics, or developer workflows and need concrete product examples.
- Your team is deciding where an assistant should support a person while leaving review and approval with that person.
- You need to think through tenant separation, grounding, data privacy, masking, toxicity detection, or explainability before deploying generative AI.

## Related talks

- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [Guiding LLMs While Staying in the Driver's Seat](https://mlopstalks.com/talks/guiding-llms-while-staying-in-the-drivers-seat) (Jacob van Gogh, Adept AI, 10:02)
- [AI Squared: Breaking LLMs out of the Chat Application](https://mlopstalks.com/talks/ai-squared-breaking-llms-out-of-the-chat-application) (Benjamin Harvey, AI Squared, 52:20)
- [Real LLM Success Stories: How They Actually Work](https://mlopstalks.com/talks/real-llm-success-stories-how-they-actually-work) (Alex Strick van Linschoten, ZenML, 48:20)
- [LLM Use Cases in Production](https://mlopstalks.com/talks/llm-use-cases-in-production) (Agnieszka Mikołajczyk-Bareła, Chaptr & Jason Liu & Arjun Kannan, Resides, 30:49)