Code completion quality depends on whether suggestions are honest, harmless, and helpful to the developer's goal.
2
A production architecture can combine pretrained and third-party models with prompt processing, validation, ranking, and continuous evaluation.
3
Historical code commits can provide reference outputs for evaluating model suggestions without relying entirely on manual review.
Summary
Monmayuri Ray describes GitLab's early work building code completion tools and the decisions involved in taking LLMs into production. She evaluates suggestions against three practical questions: are they consistent with facts, harmless to the developer, and helpful for completing the intended task? Her proposed architecture combines pretrained open-source models with third-party models, data processing, tokenization, prompt libraries, model gateways, validators, and post-processing. The central engineering problem is evaluation at scale. Historical code commits can be used to identify acceptable developer outputs, which then provide reference points for comparing model completions. Ray argues that prompt engineering and model tuning should run in a continuous loop with version control, CI, feedback, ranking, and rate limits. She closes by saying that code completion has a steady source of new evaluation data because developers keep writing code, while human judgment remains necessary for some tasks.
Code completion must be judged by whether it helps the developer
Ray defines code completion as writing or recommending code for a developer. She says model output should be examined in three ways. It should be honest, meaning consistent with facts and useful to the person coding. It should be harmless. It should also help the developer accomplish the intended goal. Correctness alone is too narrow. A suggestion can compile and still fail to help with the feature the coder is trying to build. Ray uses this practical standard to frame later choices about models, prompts, evaluation, latency, and cost.
Choosing a model requires more than checking benchmark quality
For raw LLM selection, Ray says teams need to examine the training data, model parameters, existing evaluation benchmarks, model weights, tuning options, cost, and latency. Open-source models may allow teams to add data or change how the model is tuned, while third-party models impose different constraints. She questions how teams can assess quality at scale rather than relying on occasional examples. The production goal is to make open-source or third-party models produce useful code for a specific use case, with prompt engineering and evaluation treated as ongoing work.
The serving architecture combines models with a prompt and validation layer
Ray describes an architecture that can use several models instead of choosing only one. Additional data is collected, preprocessed, tokenized, and used to train or tune a pretrained model, with checkpoints kept along the way. In parallel, a prompt engine examines the code a developer has written, breaks it into tokens, and uses a prompt library or prompt database. A gateway then applies post-processing and validation before calling either a third-party model or a model trained by the team. The same architecture can support continuous evaluation and inference.
Historical commits can create reference outputs for evaluation
Ray explains that a company codebase contains many examples of developers implementing similar work in different ways. The team can inspect committed code, compare examples by similarity, and agree on an acceptable developer output for a particular completion task. That reference can then be compared with output from open-source, third-party, or tuned models. She mentions using similarity-based techniques to connect the prompt and generated code to a quality assessment. This turns past development work into evaluation material instead of leaving prompt and completion review in manual spreadsheets.
Continuous evaluation should run as an automated engine
Ray proposes treating prompt input, tokenization, similarity checks, storage, evaluation, and output generation as connected services. The process can run repeatedly in the background rather than only when a person reviews a new prompt. This allows the team to keep measuring usefulness, harmfulness, and other benchmarks as the system changes. She describes the benefit as an evaluation loop that can run at large scale and feed its results back into code completion. The point is to make quality assessment part of the production architecture.
Prompt tuning and model tuning form a feedback loop
After evaluation identifies where the model performs poorly, the team can test prompt templates and prompt tuning. A prompt validator and rate limiter can inspect user input and generated output. If prompt changes are insufficient, the process can move toward fine-tuning the models. Ray describes these pieces as version-controlled services connected through CI. She compares the repeated process to a recommendation engine that keeps adjusting its output. The model starts with some acceptance level, then uses evaluation and continuous prompting to improve the usefulness of its suggestions.
Code completion produces a continuing source of evaluation data
Ray says code completion has a built-in stream of new material because developers continue writing and committing code. That completion data can support evaluation, prompt inference, ranking, and analysis of which decisions affect the quality of suggestions. The loop can therefore keep growing as the product is used. She ends by arguing that data remains central to LLM development, while some work will still require human ability even when sufficient data is available.
"Imagine this having this in scale as engines and that's the beauty of having this sort of microservice engines added to your layer of full architecture."12:40
Who should watch
You are building an AI code assistant and need a way to judge suggestions beyond whether they are syntactically correct.
Your team is deciding between open-source and third-party LLMs and needs to account for training data, tuning, cost, latency, and evaluation.
You want to turn committed code and developer feedback into a repeatable evaluation and prompt-tuning process.
