# Linguistically-informed LLMs Perform Better

Chris Brousseau, Mastercard | LLMs in Production 2023 | 19:34

Source: https://www.youtube.com/watch?v=7Hvy8j3j-9s
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/linguistically-informed-llms-perform-better
Published: 2023-07-27
Tags: embeddings, prompt-engineering, voice

## TL;DR
- LLM teams should define the language goal they need before optimizing metrics such as precision, recall, or F1.
- Linguistic structure gives practical ways to improve tokenization, embeddings, context handling, and speech generation.
- In one biology-question example, pragmatic prompting raised Falcon 7B Instruct from 7 correct answers out of 20 to 17, while reducing runtime to about two seconds locally.

## Summary
Chris Brousseau argues that large language models should be designed with language itself in mind. He uses syntax, morphology, semantics, pragmatics, and phonetics to explain where current systems work and where they lose information. He considers syntax largely solved by generative models, while tokenization and embeddings are partly solved. Pragmatics can improve inference by giving a model context, rules, and opportunities to reason through a task. In his biology example, Guidance applied pragmatic instruction to Falcon 7B Instruct, increasing the score from 7 out of 20 to 17 out of 20 and cutting the local runtime to about two seconds. Brousseau also explains how frequency-based tokenization can mishandle numbers, newly popular words, and borrowed words. The final example concerns speech: text alone loses emphasis and melody, so systems that combine text with phonetic and audio information can preserve more of the intended meaning.

## Key ideas
### LLM projects need a language goal before they optimize metrics
[03:13](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=193s)
Brousseau compares training an LLM with growing a beard. A person can measure length, but that does not say whether the result matches the intended style. He makes the same point about precision, recall, and F1. If a KPI is treated as the goal, a team may be late in the process or may have no clear destination. The model should be built toward a use case and a desired language behavior, with metrics helping measure progress toward that goal.

### Language has several structures that models need to handle
[05:42](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=342s)
Brousseau divides language into syntax, morphology, semantics, pragmatics, and phonetics. Syntax concerns grammar and structure. Morphology concerns how smaller units of meaning form words. Semantics concerns literal definitions. Pragmatics concerns the surrounding situation and how context changes interpretation. Phonetics concerns the production of speech, including articulation, sounds, and the relationship between writing and speech. These categories give teams a way to locate the source of a model's failure.

### Current LLM methods handle syntax well and morphology partly well
[07:02](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=422s)
Brousseau says generative LLMs can produce infinitely many combinations, which makes him think much of the syntax problem is already handled. Tokenization gives models a representation of word structure, while embeddings turn vectors into representations with meaning for the model. He estimates that morphology and embeddings are roughly 75 to 80 percent solved. He places pragmatics and phonetics further behind, since they require context and information that ordinary text may omit.

### Dictionaries describe changing usage rather than permanent definitions
[09:18](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=558s)
Brousseau presents the dictionary problem as a warning against treating dictionaries as final authorities on meaning. He describes Dictionary.com, Merriam-Webster.com, and the Oxford English Dictionary as updating their vocabularies over time to reflect current usage. That creates a design question for an LLM: should it describe language at one point in time, or should it keep adapting? A financial model may need less frequent language updates than a general-purpose model because the language in its domain changes more slowly.

### Frequency-based tokenization can hide useful word structure
[11:21](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=681s)
Brousseau connects the yeet problem to morphology and predictability. Byte pair encoding, SentencePiece, and ChatGPT's encoding rely on statistical patterns and frequency. He argues that this can produce bad representations for numbers and newer words. English has constraints on which sounds and letters can appear together, and those constraints can make a word predictable even when the word itself is new or uncommon. Using linguistic structure can therefore improve tokenization and the model's view of the input.

### Multilingual training can help models segment borrowed words
[13:07](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=787s)
The kimono example shows how a tokenizer trained around English can assign misleading meaning to parts of a borrowed word. Brousseau says an English-oriented tokenizer may see 'mono' as meaningful while failing to treat the rest of 'kimono' appropriately. He connects this to the observation that multilingual models can outperform monolingual models on the same tasks. Exposure to more languages gives the tokenizer more evidence about how words are formed and borrowed.

### Pragmatic instructions can improve both answers and runtime
[14:14](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=854s)
For a set of 20 college-level biology questions, Brousseau says a vanilla ChatGPT setup answered 7 correctly and initially took about four minutes, or about one minute after the connection warmed up. Running Falcon 7B Instruct locally with Guidance added pragmatic instruction and chain-of-thought prompting. The system prompted itself several times and used system prompts to draw out the model's knowledge. It answered 17 of 20 correctly and took about two seconds. He also mentions LangChain and vector databases as ways to add retrieval and more context.

### Text-only speech systems lose emphasis that changes meaning
[16:55](https://www.youtube.com/watch?v=7Hvy8j3j-9s&t=1015s)
Brousseau uses the sentence 'I never said I loved you' to show how emphasis changes meaning while the written words stay the same. He compares text-to-speech systems such as Tortoise and 11 Labs with a speech-to-speech model and a model that combines phonetic information with an audio reference. He says the text-to-speech systems lose melody and phonetic information, while the combined approach sounds much better. His point is that reducing speech to text removes information that language models may need.

## Notable quotes
- "LLMs are solving for language, right? They're large language models." (04:55)
- "The problem that I see with that is you aren't hitting your goals when you're measuring for KPIs." (04:16)
- "We don't determine morphemes based on frequency, and so they ran into problems with numbers where it's grouping commonly grouped numbers together." (11:42)
- "It increased by 10 which increased by 50% accuracy and increased in speed like almost a hundredfold just from using some pragmatic instruction on the inference side." (16:09)

## Tools & references mentioned
- Mastercard
- Matthew Sharp
- MLOps Community
- Fred Jelinek
- Shoebox
- Chomsky
- Dictionary.com
- Merriam-Webster.com
- Oxford English Dictionary
- byte pair encoding
- SentencePiece
- ChatGPT
- GPT-4
- Goat 7B
- Falcon 7B Instruct
- Guidance
- LangChain
- Tortoise
- 11 Labs
- SVC
- International Phonetic Alphabet
- LinkedIn

## Who should watch
- You are choosing evaluation metrics for an LLM but have not written down the language behavior the system should produce.
- Your tokenizer performs poorly on numbers, borrowed words, newly popular terms, or multilingual input.
- You work with speech or text-to-speech systems and need to understand what is lost when audio becomes text.

## Related talks

- [Want High Performing LLMs? Hint: It Is All About Your Data](https://mlopstalks.com/talks/want-high-performing-llms-hint-it-is-all-about-your-data) (Vikram Chatterji, Galileo, 33:56)
- [Incorporating LLMs in High-stake Use Cases](https://mlopstalks.com/talks/incorporating-llms-in-high-stake-use-cases) (Yada Pruksachatkun, Moonhub, 11:01)
- [LangChain: Enabling LLMs to Use Tools](https://mlopstalks.com/talks/langchain-enabling-llms-to-use-tools) (Harrison Chase, LangChain, 11:43)
- [Boosting LLMs: Performance, Scaling, and Structured Outputs](https://mlopstalks.com/talks/boosting-llms-performance-scaling-and-structured-outputs) (Tom Sabo, SAS & Matt Squire, Fuzzy Labs & Vaibhav Gupta, Boundary ML, 1:01:24)
- [Why Language Models Need a Lesson in Education](https://mlopstalks.com/talks/why-language-models-need-a-lesson-in-education) (Stephanie Kirmer, DataGrail, 14:54)