Cost Optimization and PerformancePushed backThe best cost-reduction approach was disputed between selecting and optimizing the right model and hardware, versus pruning and distilling a larger model.8:45
Threads / Paying for it / 2023
Paying for it in 2023
27 sessions
Cost Optimization and Performance
Efficiently Scaling and Deploying LLMsClaimThe cost and difficulty of training large language models are lower than commonly assumed when appropriate tooling is used.12:07
Using LLMs to Punch Above Your Weight!ClaimCameron Feenstra says relying on an API can make iteration on business logic, prompts, and model inputs expensive or even cost-prohibitive.19:07
PodcastWhy is MLOps Hard in an Enterprise?ClaimMaria Vechtomova says Ahold Delhaize reduced infrastructure costs by standardizing MLOps processes and changing how clusters were created, used, and monitored.50:10
PodcastThe Long Tail of ML DeploymentClaimTuhin Srivastava believes smaller, targeted models are preferable when a product requires concentrated value, speed, cost control and performance.28:27
PodcastFrom Arduinos to LLMs: Exploring the Spectrum of MLClaimBuilding an application around an LLM API is initially cheap and easy, but complexity, reliability problems, and costs increase as the product grows.21:58
The Emerging Toolkit for Reliable, High-quality LLM ApplicationsClaimReliable LLM applications must address operational issues such as cost, performance, availability, model drift, timeliness, and privacy.5:14
Understanding the LLM EconomicsClaimNikunj Bajaj says GPT-4 pricing has separate charges for prompt tokens and response tokens.6:12
It Worked When I Prompted ItClaimAs an LLM application becomes more complex, longer prompts can cause API costs to rise quickly.5:51
PodcastTreating Prompt Engineering More Like CodeClaimMaxime Beauchemin says Promptimize can compare different prompts, models, and parameter settings and report differences in accuracy, speed, and cost.17:52
Building ProductsClaimSahar Mor says prompt chaining, asking a model to critique its own answer, and requiring citations can help mitigate hallucinations, although some of these methods increase cost and latency.23:39
Making LLM Inference AffordableClaimThe reduction in summarization cost allowed Neeva to summarize its entire index offline instead of doing the work online.9:15
The Confidence Checklist for LLMs in ProductionClaimRohit Agarwal recommends captchas, rate limits, and monitoring to limit the cost of abusive traffic and DDoS attacks.7:47
Preemption Chaos and Optimizing Server StartupClaimUsing preemptible GPUs can cut cloud costs by two-thirds while maintaining uptime for users.2:05
PodcastTecton Round-table // Get your ML Application Into ProductionClaimDerek Salama describes an Uber pricing model that cost 50 cents in additional compute per ride and was not deployed in markets where the profit margin was about 10 cents per ride.11:06
Finetuning Open-Source LLMsClaimIn Sebastian Raschka's speed test, low-rank adaptation trained a 7-billion-parameter model on one GPU in about one hour, while full fine-tuning took about nine hours on six GPUs.14:49
Fireside Chat with LLM StartupsClaimMartian routes each user request to the model that provides the highest performance at the lowest cost.8:22
What Drives GenAI Development in the Next 3 YearsClaimThe cost of achieving a given level of model quality tends to decrease over time, while model quality continues to improve.5:38
AI in Education Fireside ChatClaimKlinton Bicknell says running language models naively at large scale can be cost-prohibitive, so Duolingo uses them to pre-generate options or create inexpensive rules and detectors for real-time use.21:26
Amplifying Impact with Generative AI: Insights from 10,000 ColleaguesClaimPrompt economics are affected by language because tokenization can require substantially more tokens for some non-English languages and for code than for English.16:40
Exploring the Latency/Throughput & Cost Space for LLM InferenceClaimTimothée Lacroix says the talk focuses on the cost of inference, throughput, latency, and deployment of open-source large language models.1:13
Efficient Serving of LLMs for Experimentation and Production with Fireworks.aiClaimFine-tuning can reduce serving costs because it enables shorter prompts and sometimes smaller models for the same quality.1:59
Current State of LLMs in ProductionClaimQuery routing can reduce cost by sending complex queries to more capable models and simpler queries to less expensive models, and caching can reduce the number of LLM calls.9:14
Building RAG-based LLM Applications for ProductionClaimRay Data was used to scale document processing and embedding because it supports different data sources and can use CPU and GPU resources together.13:55