# Fireside Chat with LLM Startups Paul van der Boor & Sandeep Bakshi, ProsusGroup & Shriyash Upadhyay, Martian & Lars Maaløe, Corti & Pietro Gagliano, Transitional Forms | LLMs in Production 2023 | 30:46 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=Q9wpSwUOrEU Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/fireside-chat-with-llm-startups Published: 2023-10-18 Tags: cost, healthcare, model-serving, startups ## TL;DR - Martian routes each prompt to the model that offers the best performance at the lowest cost, rather than sending every request to one provider. - Corti builds the full audio-to-clinical-prediction stack for medical conversations, with real-time assistance for healthcare professionals. - Transitional Forms combines different AI systems with Unity to create interactive, real-time entertainment such as automatically generated TV shows. ## Summary This panel brings together founders building companies around different uses of generative AI. Pietro Gagliano describes Transitional Forms as an entertainment innovation company that connects models, game engines, and interactive formats to generate shows in real time. Lars Maaløe explains how Corti listens to medical conversations, understands them, and gives healthcare professionals prompts during urgent calls. Its stack covers speech recognition, language processing, and clinical prediction. Shriyash Upadhyay describes Martian as a routing layer for language models. It studies what different models are good at and sends each request to the most suitable one. The discussion also covers competition, proprietary technology, and fundraising. The founders advise technical builders to focus on their own motivation, listen carefully to investors without letting fundraising displace building, and stay close to the conviction behind their idea. They also discuss the difficulty of describing new products before the market has settled on the right language. ## Key ideas ### Transitional Forms treats AI as a set of creative building blocks for interactive entertainment [02:29](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=149s) Pietro Gagliano says Transitional Forms began in 2018 as a creative machine intelligence company. The team has experimented with reinforcement learning, language models, and music Transformers, and has grown to about 10 people. Its TV platform lets users generate shows from templates, prompts, story beats, music, sound effects, camera filters, sets, and character backstories. The company's live generative variety show, Robots Make TV, produces autonomous outputs in real time, so the same setup does not create the same show twice. Gagliano describes the company as connecting different generators rather than building one core model. ### Corti uses AI to help healthcare professionals understand urgent medical conversations [05:25](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=325s) Lars Maaløe says Corti focuses on the medical conversation, where patients may struggle to explain symptoms and healthcare professionals may misunderstand them. The first use case is emergency calls, such as 911 in the United States and 112 in much of mainland Europe. Corti listens to the conversation, understands what is being said, and gives the professional prompts that support a decision within seconds. Maaløe says the company has about 100 people and had recently raised its Series B, with Prosus among the lead investors. ### Martian routes requests across language models to reduce cost without giving up performance [07:45](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=465s) Shriyash Upadhyay compares using language models to using electricity. Developers should be able to connect an application to language-model intelligence without having to decide where every request comes from. Martian receives a request, chooses the model with the best performance at the lowest cost, and returns the result. Upadhyay says the company can beat GPT-4-level performance on OpenAI's own evaluations at a lower cost by routing requests to the right model. The company's purpose is to make a multi-model architecture easier to use. ### Each founder defines competition around a different technical advantage [09:36](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=576s) Gagliano sees Runway ML, Midjourney, and Stable Diffusion as tools that can contribute to creative pipelines, but says few companies take Transitional Forms' real-time, interactive approach. The company uses Unity and draws on Gagliano's background in VR. Maaløe identifies Nuance as a major competitor in healthcare, while distinguishing Corti's focus on predicting outcomes from conversations rather than mainly providing dictation. Upadhyay says language-model providers are limited because they naturally prioritize their own models, so they will not route a request to a competitor when that competitor performs better. ### Corti builds and controls the full path from speech to clinical prediction [16:27](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=987s) Maaløe says Corti has built its stack from audio understanding through speech representation learning, speech recognition, natural language processing, and clinical predictions. The team follows research closely, reproduces published models, and replaces components when newer approaches perform better. Its system must handle difficult audio and complex inputs while producing predictions in less than a second. Corti also fine-tunes and restructures language models for medical use. Maaløe argues that a vertical model can perform very well on a focused domain because it does not need to handle unrelated requests such as pizza-delivery calls. ### Martian maps model behavior so routing can be based on more than surface-level benchmarks [20:22](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=1222s) Upadhyay says good routing requires understanding what each model is good and bad at. Language models have unknown behaviors and can hallucinate, so treating them as black boxes limits the quality of routing. Martian's core work, which it calls model mapping, turns Transformers into a more interpretable representation. The team studies how models work internally instead of competing to build ever-larger models. This understanding lets Martian decide which model should handle a particular request. ### Founders should protect time for building while choosing investors with real conviction [24:20](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=1460s) The founders advise builders to use investors as a source of information about competition and the market, while being selective about who receives their time. Maaløe recommends asking venture capitalists questions and testing whether an idea is as strong as it feels from inside the company. Upadhyay says high-conviction investors can help founders focus on building. Gagliano says founders with ideas that are several years ahead of the market should seek people who think about the future and avoid being discouraged when others do not immediately understand the idea. ### Technical founders should follow internal motivation and adopt language that helps others understand the product [28:42](https://www.youtube.com/watch?v=Q9wpSwUOrEU&t=1722s) When asked what they wish they had known before starting an AI company, Upadhyay says founders should focus on what motivates them internally rather than chasing every exciting development. Gagliano wishes he had known the terms that would later describe his work, since explaining an invention without accepted language took much longer. He advises founders to connect their idea to the words that culture and the market use for it. The closing advice is to follow personal conviction, keep doing work that matters to the founder, and avoid letting others redefine that conviction. ## Notable quotes - Shriyash Upadhyay: "We want to make it as easy as plugging your application into an LLM, just as you would plug an electrical socket into an outlet." (08:04) - Lars Maaløe: "Our core AI is basically the entire value chain from the audio all the way to the clinical predictions and be able to do that in real time." (17:33) - Shriyash Upadhyay: "The core technology we're developing is actually understanding these Transformer models." (21:21) - Shriyash Upadhyay: "Building is the most important thing." (26:22) - Pietro Gagliano: "As you're inventing things, really try to latch on to what culture is calling what you're doing." (29:44) ## Tools & references mentioned - ProsusGroup - Martian - Corti - Transitional Forms - Runway ML - Midjourney - Stable Diffusion - Unity - Nuance - GPT-4 - OpenAI - Transformers - LSTM - Robots Make TV ## Who should watch - You are building an application that may need to choose between several language models based on quality, latency, or cost. - You work on speech, language, or clinical systems and want an example of a focused, real-time AI product in healthcare. - You are a technical founder deciding how to explain an unusual AI product and how to spend time with investors. ## Related talks - [AI in Education Fireside Chat](https://mlopstalks.com/talks/ai-in-education-fireside-chat) (Klinton Bicknell, Duolingo & Bill Salak, Brainly & Yeva Hyusyan, SoloLearn, 31:01) - [LLM Valley](https://mlopstalks.com/talks/llm-valley) (Lilly Chen, Contenda, 11:57) - [AI Is Fast. AI Projects Are Slow. Let's Fix That.](https://mlopstalks.com/talks/ai-is-fast-ai-projects-are-slow-lets-fix-that) (JRocketRide's Joe Maionchi, 56:48) - [A Playground for AI Engineers](https://mlopstalks.com/talks/a-playground-for-ai-engineers) (Paulo Vasconcellos, Hotmart, 54:42) - [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg, 50:12)