33 sessions
PodcastHow A Manager Became a Believer in DevOps for Machine LearningClaimKeith Trnka says projects usually fail because they do not address user or business needs, or because the surrounding deployment, reliability, monitoring, and rollback work is inadequate.5:31
PodcastML Scalability ChallengesClaimWaleed Kadous says machine-learning infrastructure should let developers move from development to scalable serving while retaining their preferred Python libraries.25:18
MeetupDeclarative MLOps: Streamlining Model Serving on KubernetesClaimRahul Parundekar says Kubernetes lets an ML engineer declare a target layout, including model-server and backend replicas, and then handles scheduling and orchestration.7:24
Cost Optimization and PerformanceClaimBringing models in-house gives a company more control over latency optimization, API rate limits, and service availability.6:40
No Rose Without a Thorn - Obstacles to Successful LLM DeploymentsClaimTanmay Chopra divides the obstacles to deploying LLMs into production into infrastructure challenges and output-related challenges.0:46
PodcastWhy is MLOps Hard in an Enterprise?ClaimMaria Vechtomova and Basak Eskili say their standardized deployment process makes it possible to deploy a model to another Ahold Delhaize brand in a few minutes, although the model still needs to be adapted to the brand's data.13:30
PodcastClean Code for Data ScientistsClaimMatt Sharp says Merlin provides command-line tools, dependency management, image creation, boilerplate code, and automatic scaling for machine learning services.31:43
PodcastThe Long Tail of ML DeploymentClaimQuantumBlack is technology agnostic and helps organizations improve the process of building and deploying AI while reducing friction between teams.2:03
Large Model Training and Inference with DeepSpeedClaimDeepSpeed is a library for training, compression, and inference of large models.1:59
Scalable Evaluation and Serving of Open Source LLMsPushed backThe displayed model cost and performance estimates do not fully account for batching, which can change results substantially.10:19
Taking ImgFlip's 'This Meme Does Not Exist' to the Next Level with a LLMClaimThe project uses ImgFlip's meme dataset to build a content-aware LLM that can create quality memes and cover more templates than the existing service.4:55
PodcastOpen Source and Fast Decision MakingClaimGiving narrowly focused services access to entire repositories, drives, or communication systems creates a serious security risk.11:26
Build and Customize LLMs in Less than 10 Lines of YAMLPushed backTravis argued that fine-tuning a smaller model can match or outperform a much larger model for a sufficiently bounded task at lower latency and cost.17:17
Challenges in Providing LLMs as a ServiceClaimServing large language models can require splitting them across multiple accelerators, which creates communication and computation trade-offs.1:52
LLM on KubernetesPushed backRahul Parundekar argues that companies should prioritize a repeatable deployment platform and model iteration over over-optimizing Kubernetes autoscaling while GPUs are scarce.16:42
Unleashing Code Completion with LLMsClaimModel selection for code completion should consider the objective, training data, evaluation benchmarks, model weights, tuning frameworks, cost, and latency.5:54
End-to-end Modern Machine Learning in ProductionClaimOmar Sanseviero aims to increase awareness of tools that make it easier to use state-of-the-art machine learning models in products and services.0:47
PodcastUsing Large Language Models at AngelListPushed backThe idea that off-the-shelf machine-learning services were sufficient for AngelList was rejected because they were too expensive, could not scale as needed, and lacked the required extraction capabilities.12:57
Considerations and Optimizations for Deploying Open Source LLMs at Your CompanyClaimDeploying an open-source LLM as a fast, secure, scalable API endpoint is broadly a software engineering problem similar to deploying other machine-learning models, but it may require much more memory.1:03
Enabling Defense Missions with Local LLMsClaimRegulated environments are often restricted, isolated, and controlled for ingress and egress, and some are completely air-gapped or deployed at the edge.2:59
Preemption Chaos and Optimizing Server StartupClaimServing large language models at low latency requires powerful GPUs, and larger models require better and more expensive GPUs.2:30
LLMs vs LMs in ProductionClaimVoiceflow's custom NLU model outperformed GPT-4 on both cost and accuracy in one test because GPT-4 inference cost much more.22:43
PodcastBuilding an ML Platform: Insights, Community, and AdvocacyPushed backStephen Batifol says deployed machine learning models should be treated as ordinary software for operations and on-call rather than as a separate special category.43:20
PodcastMLOps at GetYourGuidePushed backJean Machado argued that people should not be blocked from using LLMs because they do not use the platform's Python-centered templates, although the platform will provide managed tooling for use cases that need observability and controls.54:11
Efficient Serving of LLMs for Experimentation and Production with Fireworks.aiClaimFine-tuning can reduce serving costs because it enables shorter prompts and sometimes smaller models for the same quality.1:59
Building RAG-based LLM Applications for ProductionClaimRay Serve was used to deploy the application because it supports composing embedding models, ranking models, and language models, as well as end-to-end streaming.15:59
The State of Open Source AI: Deployment Engines, Licences, & HardwareClaimCasper da Costa-Luis says international web-service providers may need to handle different legal requirements across countries, or avoid serving some countries.4:53
PodcastDesigning for Forward Compatibility in Gen AIClaimRohit Agarwal says open-source model deployments are difficult because compute demand is hard to manage when traffic has spikes and troughs.13:56
PodcastChallenges Operationalizing ML (And Some Solutions)Pushed backNathan Ryan Frank presents notebooks as useful for exploration but argues that their work should be moved into shareable, testable, deployable structures rather than left as an unstructured production artifact.21:47
