# Alignment is Real

Shiva Bhattacharjee, TrueLaw Inc | MLOps Podcast | Episode 260 | 38:36
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=O0F3RAWZNfM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/alignment-is-real
Published: 2024-09-13
Tags: build-vs-buy, fine-tuning, orchestration, rag

## TL;DR
- Shiva Bhattacharjee uses DSPy to iteratively improve prompts, query rewriting, retrieval, and synthetic data generation for legal applications.
- TrueLaw fine-tunes embedding models and generation models when legal users need better retrieval or answers that follow a firm's preferred style.
- TrueLaw buys foundation models and workflow infrastructure, while building its own data-generation, feedback, orchestration, and messaging systems around them.

## Summary
Shiva Bhattacharjee explains how TrueLaw builds AI systems for lawyers, where precision matters more than instant responses and users are not usually prompt engineers. The company began with prompting and LangChain, then adopted DSPy because its modules and optimizers made iterative improvement easier. DSPy supports query rewriting, multi-hop retrieval, custom rankers, and synthetic data generation. TrueLaw fine-tunes embedding models with contrastive data for domain-specific retrieval, and fine-tunes generation when firms expect answers in a particular style. Shiva describes the surrounding system as a data flow that collects prompts, outputs, feedback, and training examples. TrueLaw does not train foundation models. It uses external training services, keeps its orchestration layer independent of the training environment, and can run work in its own cloud or a customer's Azure environment. The company also uses Temporal for long-running inference and training workflows. Shiva connects these choices to earlier work in distributed systems, data management, and performance engineering.

## Key ideas
### DSPy replaced brittle prompt editing with modular iteration
[01:17](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=77s)
TrueLaw first experimented with zero-shot prompting and LangChain, then ran into prompts that broke when models or prompt wording changed. Shiva Bhattacharjee adopted DSPy after seeing its modular design and optimizers. The optimizers still need examples that describe the desired behavior, so DSPy does not remove the need for useful data. Its programming model made it easier to swap rankers and other components. TrueLaw also used it to generate synthetic data. Shiva compares its modularity to PyTorch and says the framework was easier to modify than the layered chains he had encountered in LangChain.

### An open-source framework needs an operational service to attract a company
[04:57](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=297s)
Demetrios Brinkmann asks whether DSPy is risky because it lacks a large venture-backed company to guide it. Shiva says the community has accepted changes from TrueLaw quickly, and the code can be inspected or changed because it runs on the company's servers. He sees DSPy and LangChain as systems with similar limitations, since both ultimately involve prompt engineering. DSPy had a steeper learning curve, partly because its iterative prompting is less visible. Shiva thinks a company could form around services such as hosting, state management, tracing, and observability. LangSmith is an example of this service layer around a framework.

### Legal retrieval needs query rewriting and adjustable search depth
[12:26](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=746s)
TrueLaw builds for lawyers and law firms, where precision and answer quality usually matter more than the lowest possible latency. Lawyers do not necessarily write optimized prompts, so their questions need contextualization before retrieval. TrueLaw uses DSPy for query rewriting and can adjust the number of retrieval hops, retrieval depth, and top-k document count. Those settings expose the tradeoff between recall, precision, quality, and response time. The system exposes intermediary query-rewriting steps to users, so they can see how their initial question becomes more contextualized against the material being searched.

### Fine-tuning helps when retrieval or answer style is domain-specific
[14:58](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=898s)
TrueLaw fine-tunes embedding models when the search corpus and user queries contain domain-specific language. Shiva says the encoder does not need to be very large, but creating useful contrastive training data is difficult. Fine-tuning is also useful for generation when a firm expects answers in a particular format or voice. Shiva relates this to work that junior associates might previously have done, where a lawyer would request a search and expect the results to be presented in a familiar way. Basic off-the-shelf RAG approaches did not perform well enough in TrueLaw's legal experiments. The company added metadata extraction and other context before aggregating retrieved information.

### Training infrastructure is bought while data flow remains an internal focus
[18:10](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=1090s)
Shiva says training has become cheaper and more standardized, so TrueLaw does not focus on building training infrastructure. It uses an orchestration layer that is independent of where training runs and gives the company a choice of infrastructure. Training can happen in TrueLaw's cloud or in a customer's Azure environment. The company's own work centers on generating data, collecting feedback, constructing training sets, and incorporating that information into models. Shiva connects this to his earlier work with data and metadata flows at Confluent. Prompts, outputs, and feedback are unstructured, but the engineering needs around versioning, replaying data, and moving it through systems remain familiar.

### A small startup cannot justify training a foundation model
[20:13](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=1213s)
TrueLaw had to balance limited money with the need to ship features quickly. Shiva says building a foundation model is extremely difficult, even when it is technically possible, because of the resources required. The company therefore focused on fine-tuning and on compound systems that coordinate smaller language models or other components. This approach fit the company's infrastructure and product constraints better than building a large model. Shiva also says that scale and price require economies that a small company cannot create for its own limited training volume, which made external model-training services a practical choice.

### Temporal replaced custom handling for long-running workflows
[27:35](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=1655s)
TrueLaw initially built a messaging system between its microservices to handle asynchronous work. That worked until production workloads included large inference jobs over thousands of emails and other runs that could take hours. The company then used Temporal for durable workflows. Temporal handles interruptions, retries, notifications, and long-running execution after TrueLaw defines the state-machine logic. Shiva says a small engineering team would have needed months to make an internal system reliable, and that system would have been specific to their use case. Temporal made the workflow code simpler and reduced infrastructure problems around training and inference.

### Distributed-systems principles carry across the AI stack
[32:06](https://www.youtube.com/watch?v=O0F3RAWZNfM&t=1926s)
Shiva draws connections between his earlier systems work and current AI infrastructure. File systems, multithreading, data platforms, and language-model pipelines all involve concerns such as buffering, caching, parallel work, and managing memory or state. He recalls working on Grand Central Dispatch at Apple after earlier experience with a macro-based approach to asynchronous I/O at Riverbed. He also describes Spark as an aggregation of practices such as caching, data distribution, and immutability, made accessible through a declarative data-frame API. Shiva is direct about one gap in his background: he has less depth in GPU internals, although he recognizes familiar performance patterns in the papers he reads.

## Notable quotes
- Shiva Bhattacharjee: "The optimizers make a lot of sense in terms of how you can actually iteratively improve this." (03:26)
- Shiva Bhattacharjee: "Precision and quality matters trumps over latency a lot." (12:54)
- Shiva Bhattacharjee: "The main thing there in an embedding model is to kind of figure out the training data." (17:43)
- Shiva Bhattacharjee: "Building foundational model is very very difficult." (20:53)
- Shiva Bhattacharjee: "Everything is data and what you're removing with the context of data changes quite a bit." (25:56)

## Tools & references mentioned
- TrueLaw Inc
- TrueLaw
- DSPy
- LangChain
- LangSmith
- PyTorch
- Temporal
- Confluent
- Azure
- Apple
- Riverbed
- Grand Central Dispatch
- Spark
- Databricks

## Who should watch
- You are deciding whether a domain-specific RAG system needs prompt optimization, embedding fine-tuning, or generation fine-tuning.
- You are building an AI product at a small company and need to choose what to buy versus what to build.
- You run long inference or training jobs and want a practical reason to use durable workflow infrastructure.

## Editor's note

Shiva Bhattacharjee says TrueLaw's work centres on generating data, collecting feedback, constructing training sets, and replaying that information through models. ZenML records each pipeline run's steps, inputs, outputs, and code version, so training artifacts and models can be traced to the data and code that produced them. Its pipeline code can run on different infrastructure through configuration.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [LLM Evaluation with Arize AI's Aparna Dhinakaran](https://mlopstalks.com/talks/llm-evaluation-with-arize-ais-aparna-dhinakaran) (Arize AI's Aparna Dhinakaran, 55:48)
- [Impact of LLMs on the Tech Stack and Product Development](https://mlopstalks.com/talks/impact-of-llms-on-the-tech-stack-and-product-development) (Anand Das, Bito, 55:31)
- [From the Legal Trenches to Tech](https://mlopstalks.com/talks/from-the-legal-trenches-to-tech) (Nick Coleman, LexMed, 35:52)
- [Making Your Company LLM-native](https://mlopstalks.com/talks/making-your-company-llm-native) (Francisco Ingham, Pampa Labs, 56:14)
- [Building Robust AI Systems with Battle-tested Frameworks](https://mlopstalks.com/talks/building-robust-ai-systems-with-battle-tested-frameworks) (Vaibhav Gupta, Boundary ML & Charles Frye, Modal, 1:08:10)