LLM infrastructure changes so quickly that systems built with Kubernetes can become unnecessary within a few months.
2
A production LLM application needs data access, retrieval, software workflows, risk checks, user feedback, and an interface around the model.
3
Teams still lack a simple, established way to evaluate open-ended text output and manage risk across rapidly changing models.
Summary
Pascal Brokmeier, Daniel Herde, and Viktoriia Oliinyk discuss what they learned while building LLM applications for clients. Pascal describes running large models across Kubernetes nodes with Ray and Alpa after cloud platforms could not host them, then explains how quickly managed services and smaller models made that architecture less necessary. Daniel focuses on document question answering with LangChain, retrieval from company data, permissions, feedback, and safeguards around inputs and outputs. The panel argues that an LLM is only one part of a product. Teams still need vector stores, data integration, workflow code, user interfaces, evaluation, and human review. They also discuss the difficulty of evaluating free-form answers, the cost of model calls, open-source model risks, data sovereignty, and the need for data engineers to develop stronger software engineering skills. The speakers are enthusiastic about the technology, while being candid about immature evaluation and compliance practices.
LLM infrastructure can become obsolete within months
Pascal Brokmeier describes how the team first tried to host large models in late 2022. Google Cloud Platform could not host them through Vertex AI, and SageMaker and Azure support was still unclear. They used Kubernetes, Ray, and Alpa to spread models across multiple nodes, including a 300-gigabyte model. A few months later, Vertex AI, SageMaker, and Azure had added support, while smaller seven- to nine-billion-parameter models had become useful. Pascal says teams should choose technology that gets the product working rather than automatically taking on Kubernetes complexity.
Document question answering depends on retrieval and domain context
Daniel Herde identifies document question answering as one of the strongest use cases. Organisations in areas such as finance, sustainability, healthcare, and education have large collections of documents but need specific answers from them. LangChain lets teams connect a language model to a particular problem domain and provide supporting resources. Daniel says this can reduce hallucinations and gives teams a way to validate and check model outputs. The system must bring relevant company information into the prompt instead of relying only on the model's general knowledge.
A production application contains much more than the language model
Pascal explains that inference is becoming easier, while fine-tuning remains harder and training from scratch is a separate problem. For a question-answering product, the team still needs a vector store, a process that enriches the user's question with context, and checks for risk and ethics. It also needs access to company data, which may be split across many silos. In this design, the LLM is one component inside a larger software product. Classic engineering work around data, permissions, workflows, and interfaces remains substantial.
Data engineers need stronger software engineering skills
Pascal divides data engineers into people who came from software engineering and people whose work is mainly SQL and Spark code. He expects the first set of skills to matter more as teams connect complex software systems and make data available to LLM applications. He also points to vector databases as a new area that many practitioners did not learn at university. The work involves bringing data assets together, connecting services, and building the systems around the model.
Open-ended text is difficult to evaluate with standard machine learning metrics
Daniel says traditional machine learning often has a clear metric such as R-squared, followed by feature work and hyperparameter tuning. Prompt engineering and generated text feel more like a soft science. The panel discusses automatic validation, using the probabilities produced by the underlying model, and evaluation methods inspired by reinforcement learning from human feedback. Daniel is clear that these approaches are still developing, and he has not seen a good off-the-shelf solution for structured quality evaluation.
Risk controls must cover both user inputs and model outputs
For a document question-answering system, Daniel says teams need infrastructure for user feedback and iterative model improvement. They also need checks on incoming questions to reduce harmful or inappropriate requests, followed by review of generated answers before they reach a broad internal or external audience. Pascal adds that open models may not have gone through the same alignment work as models from OpenAI. Organisations using raw open-source models may therefore need to recreate safeguards themselves. The panel does not identify a drop-in risk and compliance framework.
Human review remains appropriate for consequential outputs
Daniel says organisations are enthusiastic about generative AI, partly because ChatGPT has made the technology easy to try. That enthusiasm can create unrealistic expectations about cost and capability. He recommends keeping humans in the loop, especially when generated outputs are used downstream or may affect individuals. A public chatbot is easier to imagine than a reliable product, because the latter also needs review, permissions, evaluation, and controls.
Local and packaged systems could address data sovereignty concerns
Pascal responds to concerns about European organisations using US cloud providers and APIs. He imagines a 'document QA in a box' that a small or medium-sized company could install on its own virtual machine or appliance. Users could place files on a Windows file system or Linux server and access a web interface without sending data to a cloud provider, another country, or a competitor. He presents this as a way to make document question answering more accessible while keeping data closer to the organisation.
"The space is moving so quickly that it's actually really fun to see that code that you thought was a really good idea two months ago and soft infrastructure that you've built two months ago you can pretty much get rid of."Pascal Brokmeier07:53
Who should watch
You are deciding whether to run LLM inference yourself or use a managed cloud service, and need to understand how quickly that decision can change.
You are building document question answering and need a concrete view of the retrieval, permissions, feedback, evaluation, and safety work around the model.
You are responsible for deploying generative AI in a regulated organisation and want an honest discussion of human review, open models, cost, and data location.
