Replit cut GPU serving costs by two-thirds by using preemptible nodes while keeping uptime.
2
Reducing container size and enabling GKE image streaming cut much of the startup delay caused by node replacement.
3
Switching from Alpine to a slim Google Cloud SDK image fixed a gsutil multiprocessing problem and reduced model loading from about four minutes to less than 30 seconds.
Summary
Bradley Heilbrun explains how Replit used preemptible GPU nodes for LLM serving even though Google warns against using them for highly available services. An A100 cost about $3,000 per month at the listed price, compared with about $1,000 for a spot instance. The tradeoff was frequent preemption, only 15 seconds of warning, and occasional lack of capacity in an entire zone. Replit focused this talk on making servers start quickly enough to recover. Their original startup took about 18 minutes: two minutes for the node, 11 minutes for application containers, and five minutes to load model weights. Container cleanup helped a little. GKE image streaming reduced container startup from minutes to seconds. Model loading improved after moving downloads to local NVMe storage and then fixing a gsutil issue caused by the Alpine image. Total startup fell to about two minutes, allowing Replit to use cheaper nodes without losing uptime.
Preemptible GPUs cut the listed cost of an A100 by two-thirds
Bradley Heilbrun says Replit needs high-end GPUs to keep latency low as its models grow, but those GPUs are expensive. The pricing example in his slide puts an A100 at about $3,000 per month without discounts, while the spot price is about $1,000. That creates a possible two-thirds cost reduction. The service still has to tolerate interruptions. Heilbrun describes spot instances as preemptible and best effort, with real preemptions and zone-wide stockouts. Google documentation says not to run highly available services on spot nodes, but Replit chose to do so anyway.
Preemptions force the service to recover within seconds rather than minutes
Preemptible nodes come with weaker guarantees. Heilbrun says a node may give only 15 seconds of notice before it goes down, and Kubernetes pod disruption budgets do not work in this situation. Replit spreads capacity across as many availability zones as possible and considers falling back to more expensive nodes or using commitments for baseline capacity. This talk focuses on startup speed because nodes can appear and disappear dynamically. A 15-minute boot process is too slow when the warning window is only 15 seconds, so the replacement server has to become useful quickly.
Replit's original server startup took about 18 minutes
Before optimization, bringing a serving node online took about two minutes for the node and its drivers. Application containers then took about 11 minutes to start. Loading model weights and reaching a healthy serving state took another five minutes. Heilbrun gives this as roughly 18 minutes in total. That delay made preemptible capacity difficult to use because a replacement could spend most of its useful lifetime starting up. The first step was to reduce the container itself. Replit removed a pip cache, dependencies used only for development and testing, unnecessary CUDA libraries pulled in with PyTorch, and unused framework support from its Triton inference server image.
GKE image streaming changed container startup from minutes to seconds
Shrinking the container removed about 10 gigabytes from its compressed size, but it only saved around a minute or two. Replit then enabled GKE image streaming. Heilbrun explains that the feature streams file contents in the background as the container reads them instead of waiting to pull the whole image first. This fit Replit's workload because the serving process did not need every file in the container. The application container portion of startup went from minutes to seconds. Heilbrun also says the feature helped system and Kubernetes containers on the node, which removed additional minutes from the overall startup time.
Local NVMe storage only helped after the download tooling was fixed
Replit initially fetched model files from Google Cloud Storage onto a remotely attached spinning disk. Heilbrun moved the destination to a locally attached NVMe SSD, expecting the faster disk to improve loading, but saw no improvement. The download was limited to about 50 megabytes per second even though the network and disk should have supported more. The team found that changing the container image increased transfer performance by more than five times. The image still used the Google Cloud SDK, but it moved from Alpine to slim. This change exposed a problem in gsutil, whose multiprocessing mode could hang on Alpine. Replit had disabled multiprocessing without realizing that the image choice was responsible.
The combined changes reduced startup from 18 minutes to about two
After the tooling change, the model loading portion fell from about four minutes to less than 30 seconds for the model Heilbrun describes. Replit's complete pod startup went from 18 minutes to two minutes, and he says he had seen it go well under two minutes. The result came from several practical changes: trimming the container, enabling GKE image streaming, using ephemeral local SSDs, and changing the container image so gsutil could use multiprocessing. Heilbrun presents the faster startup as part of Replit's broader move to preemptible nodes. That move cut costs by two-thirds while allowing the team to maintain uptime.
