# Enabling Defense Missions with Local LLMs

Gerred Dillon, Defense Unicorns | LLMs in Production 2023 | 23:07

Source: https://www.youtube.com/watch?v=MnDPFaTQuHE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/enabling-defense-missions-with-local-llms
Published: 2023-08-16
Tags: deployment, model-serving, open-source, security

## TL;DR
- Regulated defense environments need local generative AI because their data, networks, and model artifacts cannot be handled like ordinary cloud workloads.
- LeapfrogAI provides open-source APIs and tools for deploying, operating, fine-tuning, observing, and scaling models in air-gapped and access-controlled environments.
- Local models need compatibility with familiar tools and APIs, along with support for text, speech, images, embeddings, CPUs, GPUs, and distributed Kubernetes deployments.

## Summary
Gerred Dillon explains why defense and other regulated environments need local language models. These environments may be air-gapped, placed at the edge, tightly controlled for ingress and egress, and subject to authorization and artifact-management requirements. Hosted APIs are difficult to use when data and model weights must remain inside the environment. Dillon argues that innovation still has to happen in these settings because generative AI can change the cost of operations and conflict across domains. His team at Defense Unicorns is building LeapfrogAI, an open-source collection of APIs and tools for deploying and operating models locally. The system supports OpenAI-compatible access, gRPC and HTTP backends, speech-to-text with Whisper large V2, embeddings, vector search with Weaviate, and CPU, GPU, and Apple Metal deployments. Demos show translation, summarization, code-generated plots, model listing, completions, and embeddings. The roadmap includes quantized models, a model zoo, a registry, fine-tuning APIs, and document ingestion.

## Key ideas
### Regulated environments restrict both access and movement of data
[02:59](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=179s)
Dillon defines regulated environments as mission-focused systems built around specific goals. They may be fully air-gapped, deployed at the edge, or surrounded by strict controls on who can access them and how data enters or leaves. Defense, healthcare, finance, and civilian government systems can contain information whose loss creates serious security problems. Deployment also requires formal steps, including authorization to operate, and teams must control model weights and other artifacts as carefully as they control containers.

### Generative AI still has to advance inside defense missions
[06:03](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=363s)
Dillon rejects the idea that regulation means innovation should stop. He describes AI as a competitive area that changes rapidly, with new open-source models and tools appearing constantly. He connects this competition to power projection, arguing that generative AI can reduce the cost of operations across domains while raising the resource cost for adversaries. For defense organizations, failing to develop these capabilities creates an imbalance.

### Small local models can handle specific tasks well
[09:03](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=543s)
Dillon points to a previous presentation about a roughly three-billion-parameter local model that performed well on healthcare question answering. He describes working with Wizard 40B for code and looking at StarCoder Pro only days earlier, using the examples to show how quickly the model ecosystem changes. Tools for regulated environments therefore need to keep pace with model releases and make new models usable without sending sensitive data to hosted services.

### Open-source tooling should work across multimodal workloads
[10:28](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=628s)
Dillon says mission-oriented tooling needs to support more than text generation. Speech-to-text, image understanding, image generation, embeddings, and language models can each provide value in these environments. The surrounding MLOps tools should cover fine-tuning, observation, deployment, and day-two operations. Dillon also wants local systems to work with familiar application frameworks and APIs, including LlamaIndex, LangChain, and Microsoft's Guidance.

### LeapfrogAI packages local model operations for restricted networks
[12:01](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=721s)
LeapfrogAI is presented as an open-source, open-contribution suite from Defense Unicorns. It targets environments with limited egress and ingress, strict data ownership, restricted API access, and security requirements. The project includes hardened containers and is being developed with authorization to operate in mind. Its goal is to provide the components needed to deploy and run generative AI while keeping data and services inside the controlled environment.

### An OpenAI-compatible API makes local models easier to adopt
[12:27](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=747s)
LeapfrogAI exposes an OpenAI-compatible API and supports model backends over gRPC and HTTP. Dillon explains that adding a model can involve creating a new gRPC backend through the project's harness. In the demo, a client lists available models, requests a completion, receives streamed tokens through the front-end API and gRPC backend, and calls an embeddings endpoint. This lets existing application patterns work with locally hosted models.

### The system is designed for distributed hardware from the start
[13:30](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=810s)
Dillon says LeapfrogAI supports models running on GPUs, CPUs, and Apple Metal. Its default deployment instructions use Kubernetes and assume a cluster of machines rather than one large monolithic host. The project uses Weaviate for vector database support so users can add embeddings to prompts quickly. This design allows teams to scale across available machines in restricted environments.

### The demos connect local models to concrete mission applications
[16:59](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=1019s)
An internal hackathon produced applications that combine several local models. A translation tool processes Yuri Gagarin's Soyuz launch speech, supports summarization, and takes about ten seconds per minute of audio. Another application uses a code model to retrieve air-quality data for June 6, 2022, and generate a plot. Dillon uses these examples to show why a local model zoo and broad model availability matter for mission work.

### The roadmap adds model management and fine-tuning capabilities
[19:09](https://www.youtube.com/watch?v=MnDPFaTQuHE&t=1149s)
The planned work includes ggml and quantized model support, more models and deployment targets, a model zoo, and a registry. Dillon also lists APIs for fine-tuning and making models available to missions, along with document management and ingestion APIs. He closes by inviting contributions through the project's repositories, Discord community, and open issues, and credits Zarf for helping deploy tools into air-gapped environments.

## Notable quotes
- "What is absolutely true is there's very high requirements for deployment and operation in these environments." (05:09)
- "Innovation and the availabilities models does not change anything and pushing for that Innovation especially in these regulated environments in these defense missions is very critical." (06:03)
- "We want to build open source tools designed for Mission oriented environments highly regulated environments." (10:28)
- "We wanted this experience right out of the gates." (15:19)
- "We wanted to be able to support more models in more places." (19:22)

## Tools & references mentioned
- Defense Unicorns
- LeapfrogAI
- Mark Andreessen
- Wizard 40B
- StarCoder Pro
- Whisper large V2
- Hugging Face
- LlamaIndex
- LangChain
- Microsoft Guidance
- Kubernetes
- Weaviate
- Chroma
- Zarf
- gRPC
- OpenAI-compatible API

## Who should watch
- You are building AI systems for defense, government, healthcare, finance, or another setting where data and model weights must remain inside a controlled network.
- Your team needs local language models that can run across air-gapped, edge, CPU, GPU, or Kubernetes environments.
- You want to understand how an OpenAI-compatible API, embeddings, speech-to-text, and model backends can fit into a regulated deployment.

## Related talks

- [Building Defensible AI Apps](https://mlopstalks.com/talks/building-defensible-ai-apps) (Gregory Kamradt, DataIndependent, 1:05:34)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)
- [The Next Million AI Apps](https://mlopstalks.com/talks/the-next-million-ai-apps) (Mark Huang, Preemo, 1:03:23)
- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)
- [Extending AI: From Industry to Innovation](https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation) (Sophia Rowland & David Weik, SAS, 1:01:37)