Ilya Reznik recommends prompt engineering and retrieval before fine-tuning because fine-tuning can reduce a model's performance on other tasks and often produces uncertain gains.
2
NeurIPS has shifted toward applied work on how Transformer models can be used in real systems, while evaluation remains unreliable because benchmark data can leak into training sets.
3
Reaching staff engineer is optional after senior level, and engineers should choose it for the work and responsibility rather than for the title alone.
Summary
Ilya Reznik discusses two connected subjects: practical choices around large language models and the career path for machine learning engineers. He argues that teams should try prompt engineering before fine-tuning. Fine-tuning takes substantial effort, can make a model worse overall, and is most useful for narrow output or style requirements. Retrieval can help with facts, though it does not remove every failure caused by the model's learned probabilities. At NeurIPS, he saw more applied Transformer work, renewed interest in curriculum learning, and widespread concern about weak evaluations and leaked benchmark data. The second half focuses on progression from senior to staff engineer. Reznik explains that staff roles vary by company and by archetype. Some engineers lead across the model lifecycle, while others are deep specialists. Staff engineers guide projects, unblock teams, and hand work off once it is moving. He advises people to pursue the next level only when they want its responsibilities, since senior engineer can be a better fit than a poorly matched staff role.
Fine-tuning should follow simpler ways to control model output
Ilya Reznik says fine-tuning can help when a model must produce a particular form, such as JSON, HTML, markdown, or a formal style. It can also help when the language distribution is unusual, as in medical terminology. The cost is uncertainty and lost general ability. A model that performs well across many tasks may become better at one narrow task while becoming worse elsewhere. Reznik recommends prompt engineering first. Fine-tuning does not magically improve a model's mathematical ability, but it may help when the model repeatedly returns markdown instead of a required structured format.
Retrieval puts facts near the model without changing its learned behavior
Reznik says language models do not store facts in the ordinary sense. They store probabilities, so fine-tuning for a particular body of knowledge has not proved useful in his experience. Retrieval can place relevant facts closer to the model's context, though it is not a complete fix. A strong prior in the language model can still overwhelm retrieved information. He also rejects the usual implication in the word hallucination that the answer is random. In his explanation, an incorrect answer can be a reasonable prediction from the model's training distribution. Future systems will combine language models with retrieval, graphs, reinforcement learning, or other techniques.
Machine learning practice is changing too quickly for permanent best practices
Reznik describes current ML work as being on shifting ground. Techniques that were considered advanced a year earlier may later be called naive, as happened with early retrieval systems. The field changes quickly because there is strong financial pressure and a large benefit to getting systems to work. Engineers cannot learn one body of material and rely on it for a twenty-year career. They need to keep learning. Reznik expects language models to remain useful, but he does not expect them to be the only technique used in future systems.
NeurIPS is moving toward practical Transformer applications
Reznik says NeurIPS, once mainly an academic meeting between neurobiology and machine learning researchers, now contains a majority of applied papers, posters, and presentations in his view. Much of the work asks how Transformer models can become useful in real systems. He describes a movement from having many model architectures, toward convergence on Transformers, and then toward divergence into applications. He also mentions work on different token directions, token importance, and reinforcement learning as researchers consider how to build systems around language models.
Curriculum learning could make training more data-efficient
Reznik explains curriculum learning through the way children are taught. Children receive sounds, words, and sentences in a progression, while models are often trained on shuffled data and left to discover the order themselves. Curriculum learning would curate data so a model learns basic concepts before building on them. That could make training more data-efficient and help models learn symbols and concepts that relate to the real world. Reznik says the approach is expensive and difficult because it requires substantial data curation, but it may reduce reliance on statistics alone and could reduce some model errors.
Benchmark scores can be high while real-world performance remains poor
Reznik is skeptical of many current evaluations. At NeurIPS, he saw repeated discussions of retrieval failures and weak evaluations. A model can score around 95 percent on a benchmark and still be unable to do useful work, which means the benchmark is measuring the wrong thing. He describes data leakage as a central problem. At internet scale, test examples may already be part of training data. Better benchmarks could use physical separation, such as problems created by mathematicians away from the internet, or time separation, where predictions cannot be judged until future data arrives.
Staff engineer is a responsibility and work pattern, not simply the next number
Reznik says companies reach staff decisions in different ways. Some value public talks and publishing, while others emphasize broad technical impact. Staff engineers need depth in at least one area and enough breadth to understand the wider ML lifecycle. In his technical-lead archetype, the job involved helping a large team succeed, setting project scope, guiding work with sensitive context, and contributing to difficult projects. He learned that taking the largest implementation task can make a staff engineer a bottleneck. A better pattern is to take an uncertain early piece, establish direction, then hand it to another engineer.
Promotion beyond senior engineer should be an intentional choice
Reznik argues that senior engineer is the required progression at many large technology companies, while staff and higher levels are optional. A strong senior engineer may earn more than a weak staff engineer because performance affects bonuses and refreshers. The next level also brings more responsibility and work. Reznik advises engineers to ask whether they want the responsibilities, rather than chasing the numerical level. He left technology employment to become self-employed and help ML engineers with their careers because that is the work he would choose even without much payment. He is building that guidance through one-to-one work, a YouTube channel, and a podcast.
