Varun Mohan moved Exafunction from GPU virtualization and inference infrastructure to Codeium after deciding that generative AI would change who consumes machine learning.
2
Codeium trains its own models for code completion, applies context from across a codebase, and runs inference in the cloud rather than on a developer's laptop.
3
Codeium evaluates generated code by deleting functions from open-source repositories and checking whether the model can restore functionality well enough for the unit tests to pass.
Summary
Varun Mohan describes how Exafunction became Codeium. Exafunction started with GPU virtualization for large deep learning workloads, eventually managing more than 10,000 GPUs for a small number of customers. Varun and his co-founder then concluded that generative AI would shift machine learning toward ordinary software developers and that their inference infrastructure addressed too small a part of the customer problem. They built a free coding assistant, trained their own models, and used their existing systems to run inference at scale. Varun explains why code completion needs different training and evaluation methods from ordinary text generation. Codeium generates code inside existing lines, uses embeddings to find relevant files across a codebase, and supports self-hosting for enterprises. Its code evaluation runs unit tests against partially deleted repositories. Varun is also cautious about building elaborate products around fast-changing models. He recommends investing in primitives such as better context and fine-tuning, which can improve as the surrounding system improves.
Exafunction began by moving GPU workloads across machines
Exafunction started with GPU virtualization because GPUs had limited memory and deep learning models consumed much of it. The system let workloads that normally ran on GPUs run on CPUs while CUDA kernels were offloaded to remote GPUs. If a machine failed, Exafunction reconstructed the state on another machine without the user seeing the change. The company managed more than 10,000 GPUs for a handful of customers and, at its peak, handled over 20% of Google Cloud Platform's GPU inference capacity in one region. Autonomous vehicle simulation was an important workload because collecting real-world miles was expensive and unsafe, while offline simulation could run the full deep learning stack.
Generative AI changed the customers Exafunction wanted to serve
Varun said the company saw generative AI narrowing and homogenizing the model workloads that it had been optimizing. Instead of many specialized machine learning engineers tuning models such as BERT for individual tasks, ordinary software engineers could ask a large generative model for an answer. He expected most workloads to move in that direction, while recognizing that autonomous vehicles would still need very high precision and recall. Exafunction therefore shifted from providing value at the inference layer to building an application. The team had already used Copilot and believed programming involved much more than the product's existing capabilities.
Making Codeium free forced the team to own the full model stack
The team decided that Codeium had to be entirely free. Varun said using OpenAI would have made the economics fail, so the company trained its own models from scratch, evaluated them, deployed them, and ran A/B tests. Codeium was not built on OpenAI or Anthropic. The product's code completion also required tasks that standard generative models did not handle by default. It sometimes generated 20 to 30 lines, needed code both before and after the cursor, and had to use context from other files. Training specifically for those tasks gave Codeium control over the model, data, and infrastructure.
Codeium applies codebase context and runs inference in the cloud
Codeium appears as an extension for VS Code, IntelliJ, Vim, Eclipse, and other IDEs. Community contributors added extensions for tools such as Emacs, Vim, and Xcode. On every keystroke, the extension sends information to the cloud for processing. The models do not run on the developer's machine because even a high-end consumer Mac has far less compute than an H100, while many users work on older Windows laptops. Codeium uses embeddings to find relevant files across the codebase, rather than limiting context to open files. The company also built an in-house chat system that uses codebase context.
Codeium scaled by combining its old GPU expertise with aggressive optimization
Varun said scaling Codeium across users was familiar because Exafunction had already managed large GPU fleets. The company obtained GPU reservations and added capacity as usage grew, while optimizing the workload so it needed fewer GPUs than other companies in the space. Codeium uses batching, quantization, model parallelism, and other techniques. Varun described a workload that runs a large multi-billion-parameter model on every keystroke. The product had grown from about 1,000 users at the start of the year to hundreds of thousands of users, with close to 400,000 downloads and five-star reviews on the VS Code Marketplace. Users accepted more than 100 million lines of code per day.
Enterprise self-hosting is tied to privacy and model personalization
Codeium lets enterprises self-host the product because companies often treat their source code as intellectual property and already self-host systems such as GitLab. Varun said customers can start with one GPU, and Dell published a case study about running the workload entirely on premises in an air-gapped environment. Self-hosted customers can personalize and fine-tune Codeium on private code. The company built systems that continuously fine-tune on the same hardware as the customer's codebase. Training its own models lets Codeium offer this without asking a customer to send its code to OpenAI. Varun also said the company avoids training on non-permissively licensed code such as GPL code and can provide attribution for permissively licensed code.
Code evaluation checks functionality through unit tests
Varun described evaluation as one of Codeium's hardest domain-specific problems. The team takes open-source repositories, runs their unit tests, removes parts of the code, and checks whether the model can reconstruct code that makes the tests pass. This works because code can be executed. The company does not require generated code to match the original word for word; it cares whether the functionality is correct. These evaluations test context selection, model improvements, and code generation. Codeium also compares offline evaluations with behavior in production, where it runs tens of millions of inferences each day. For enterprise customers, it checks whether fine-tuning and personalization improve the product on private code.
Varun measures reliable developer value instead of attention-grabbing features
Varun said Codeium tracks whether users spend time getting value from the tool, how much code they accept, and how the product helps across the software development lifecycle. That includes chat, terminal work, reading code in an IDE, and reviewing or summarizing code in source control systems. He criticized features that look impressive but are unreliable. In his example, a pull request summary could be wrong while its short haiku was amusing. A wrong summary damages trust, so Codeium watches whether users accept more code and whether they keep using the product. The company will not ship a feature simply because it attracts attention in a demo.
Future-proof systems should invest in primitives that improve with better models
Varun argued that model and hardware progress moves too quickly to justify elaborate interfaces built around today's limitations. He recalled that GPUs available for autonomous vehicles had roughly 10 to 20 teraflops when he joined Nuro, while consumer GPUs later reached hundreds of teraflops. His advice was to build for technology that works today, then choose primitives that can compound as the system improves. Better context selection and fine-tuning can improve many parts of the product as the base model and data sources improve. Extra interfaces are easier for competitors to copy and do not automatically benefit from better models.
