LinkedIn has expanded its GPU fleet and training capacity as LLM use cases move from experiments into products such as profile summarization and the LinkedIn Hiring Assistant.
2
Inference has become a larger cost problem than training because user-facing workloads need low latency, redundancy, and capacity for peak traffic.
3
Elastic workloads, fast hierarchical checkpointing, custom GPU kernels, and shared platform layers help LinkedIn use its GPU fleet more efficiently.
Summary
Animesh Singh describes how LinkedIn moved from early LLM experiments to products such as profile summarization, personalized recruiter emails, and the LinkedIn Hiring Assistant. He says training costs became easier to manage through larger GPU infrastructure, open source models, fine-tuning, and prompt optimization. Inference is harder because it has to meet latency expectations while carrying redundancy for failures and data-center outages. Singh explains why recommendation and ranking workloads create a different scale problem from user-invoked generative AI. He covers elastic scheduling, disruption readiness, hierarchical checkpointing, GPU memory limits, KV caching, and custom kernels built with Triton. LinkedIn is also using a less rigid training platform for both LLM and traditional recommendation workloads. The lower infrastructure layers can be shared, while inference engines, feature processing, hardware, and orchestration diverge as workloads become more specialized.
LinkedIn is using LLMs in products where personalization is visible to members and recruiters
Singh points to profile summarization, LinkedIn Learning assistants, personalized recruiter emails, and the LinkedIn Hiring Assistant. The recruiter assistant works from criteria set by recruiters, searches for relevant candidates, summarizes their experience and profiles, and lets recruiters choose whom to contact. He says candidates respond more often to recruiter emails that account for the candidate's profile, role, and recruiting company. Singh expects more LinkedIn features to use agents, while cautioning that agents should be applied where the use case benefits from them.
Open source models and fine-tuning reduced the barrier to training
Singh says training was once the main barrier, but LinkedIn invested in scale-out infrastructure and grew its fleet by 7x from the level he found when he joined. The fleet includes A100s, H100s, and H20s, and LinkedIn has completed 1 million training runs on the platform. Open source models also changed the economics. A company with specific data can start from a model that has already learned broad information, then use fine-tuning, supervised fine-tuning, few-shot learning, or prompt optimization instead of training on huge amounts of world data.
Inference costs grow when applications need low latency and redundancy
Singh says the main cost concern has shifted toward inference. Reasoning models may make several inference calls while analyzing and refining an answer, although users may accept the extra latency for complex questions. Recommendation and ranking systems are different. Feeds and recommendations must update as a person browses, so there is little tolerance for delay. User-facing services also need spare capacity and failover across multiple data centers. Those requirements can leave GPUs underused while still making them expensive.
Foundation models could simplify recommendation architecture, although the idea is still unproven
Traditional recommendation systems use many smaller, targeted models, real-time feature ingestion, online training, and graph processing. Singh says a foundation model may already have seen most relevant patterns, reducing the amount of online updating needed for a particular user. A central team could curate organizational data, create a small number of foundation models, and distill smaller models for individual use cases. That could reduce the number of models that need compliance review and make hiring easier by reducing the number of specialized systems. Singh is clear that this approach remains a hypothesis, and recommendation workloads operate at a much larger and more latency-sensitive scale than user-invoked generative AI.
GPU efficiency requires workloads that tolerate maintenance and failure
Singh explains that GPU maintenance decisions are harder than CPU maintenance because idle capacity is expensive. Distributed training jobs are commonly gang scheduled, so losing a few nodes can force a job to return to the queue and restart from a checkpoint. LinkedIn is exploring workloads that can shrink and expand as capacity changes. For inference, Singh argues that provisioning for peak traffic leaves GPUs idle during much of the day, so elastic architecture should scale capacity up and down. Long foundation-model runs also face failures in storage, networking, messaging, Kubernetes, or GPU health checks.
Hierarchical checkpointing reduces pauses and speeds recovery
LinkedIn changed checkpointing so a training job first writes a checkpoint in memory, then asynchronously streams it to block storage. This lets training resume sooner instead of waiting for a remote write to finish. Singh says the team is also exploring distributed caches built from SSDs attached to GPU nodes, which could avoid moving large checkpoints out of the GPU network during restore. Checkpoint frequency depends on the workload and the amount of progress a modeler is willing to lose. Planned maintenance can trigger an automated checkpoint, move the workload, and put it near the front of the queue when it is rescheduled.
Kernel fusion can improve training without rewriting every model by hand
After using data parallelism, model parallelism, tensor parallelism, and ZeRO++ to address GPU constraints, LinkedIn examined the training code and custom CUDA kernels. Model-by-model kernel rewriting did not scale, so the team used Triton, a Pythonic interface for GPU programming, to create custom kernels for distributed training workloads. Singh describes kernel fusion as a way to combine several operations instead of launching separate kernels for each one. He reports a 300% increase in efficiency in one case and more than a 50% reduction in memory use in many cases. LinkedIn later open sourced the work and integrated it with the Hugging Face ecosystem.
Memory movement is often the limiting factor in LLM workloads
Singh frames machine learning as a data-processing problem in which performance depends on moving data through GPU HBM, SRAM, CPU memory, and storage. Larger context windows and longer generated outputs increase the amount of data being processed. KV caching can avoid recalculating attention scores for token sequences, although the cache competes with the model for GPU memory. Singh says 80 GB per H100 can be low for workloads that use GPU memory for both computation and caching. Grace Hopper and Grace Black architectures address this pressure by linking CPUs and GPUs with high-bandwidth data transfer.
LinkedIn shares the lower platform layers while allowing LLM and traditional ML systems to diverge
LinkedIn moved its machine learning training pipelines from a rigid TensorFlow-based design to an open source orchestration engine called Flyte. The new platform supports short experiments, fine-tuning, remote debugging from VS Code, and versioning that bundles configuration, pipeline, model, and data lineage. Singh says the same training pipeline engine now supports both recommendation workloads and LLM workloads. Divergence appears higher in the stack. LLM systems may use vLLM, TGI, or SGLang for inference, different hardware, vector databases, embeddings, and LangChain or LangGraph, while traditional recommendation systems retain specialized inference and feature-processing systems.
"Inference is becoming very costly because you are optimizing a lot for latency, throughput, and you have a lot of failover mechanisms which you need to build."Animesh Singh11:45
Who should watch
You run a GPU fleet and need to reduce idle capacity, handle planned maintenance, or recover long distributed training jobs.
Your team is deciding whether LLMs belong in recommendation and ranking workloads where latency and request volume are high.
You are combining traditional machine learning with LLM systems and want a single training platform without forcing every workload into the same upper-layer architecture.
