Verena Weber says teams should define a business problem before choosing an AI model, then begin with the simplest solution that can meet the need.
2
At Amazon Alexa, her team used positive congruent training to reduce negative flips when updated models changed correct predictions into incorrect ones.
3
Multimodal models expand what applications can accept and produce, but Verena Weber says their extra complexity must be justified by measurable gains and a team's ability to maintain them.
Summary
Verena Weber discusses how AI systems move from experiments into production, drawing on her work as a data scientist, consultant, and research scientist on Amazon Alexa. She describes the operational work around retraining, deploying, and testing natural language models, including the problem of negative flips after an update. Her team used positive congruent training and generated variations of user utterances with a T5 model to make models less sensitive to small wording changes. The conversation then turns to multimodal models, legacy systems, and the temptation to use the newest generative AI model for every task. Weber argues for starting with a clearly defined problem, testing a simple baseline, and comparing the cost and maintenance burden of more advanced systems with their actual performance gains. She also explains why she coaches women in technology and why representation affects who feels able to stay and progress in the field.
Production work starts with maintaining models after the proof of concept
Verena Weber says her earlier consulting work involved time series, anomaly detection, NLP, and computer vision, but much of it stopped at proof of concept. At Amazon Alexa, she moved into a setting where machine learning was the core product. Her team maintained natural language understanding models for German, French, and later British English. The work included retraining models, deploying releases, checking training data, and improving model and deployment processes. She also had research responsibilities, including reading papers, attending academic conferences, and publishing work. Alexa's older architecture made adopting newer technologies difficult because the system had accumulated legacy tools and design decisions.
Amazon's Alexa team worked backwards from customer problems
When Demetrios Brinkmann asks whether papers drove the team's work, Verena Weber says the usual process went in the other direction. The team first identified problems that affected the specification or customer experience, then searched for papers and methods that could help. She gives positive congruent training as an exception. The method addressed negative flips, where a previous model handled a training instance correctly but an updated model handled it incorrectly. This mattered for frequent Alexa utterances because users expected familiar requests to keep working after retraining.
Positive congruent training reduced unwanted changes during model updates
Verena Weber explains that the team added another term to the loss function during training to prevent some negative flips before they happened. Alexa had a complicated process for checking that frequent utterances still worked after each retraining. The model had a group of very frequent requests that needed to work consistently, alongside a long tail of infrequent requests. Weber is careful about the method's limits. It could prevent some negative flips, but it could not prevent all of them. The example shows why production model updates need tests for behavior that users already depend on.
Synthetic wording variations made the model less sensitive to small input changes
Another project focused on cases where small changes in an utterance, such as adding 'please', changed the model's prediction. The team trained a T5 model on utterances with small variations and used it to generate more examples. They added suitable generated examples to the training data and also used the approach for testing. Weber says the generated data could not all be used directly. The team did exploratory analysis and spot checks, then created heuristics and other cleanup methods. Alexa received transcribed text rather than raw audio, so the natural language understanding model could analyze text.
Breaking a multimodal system into steps can make it easier to maintain
The conversation contrasts Alexa's text-based pipeline with newer multimodal systems. Weber says a pipeline that transcribes audio and then sends text to a language understanding model can be easier to maintain and debug because each stage has a defined role. A direct audio-to-intent system might reduce steps, but it also changes the trade-offs around accuracy, speed, maintenance, and debugging. She describes the move from models built for one modality toward systems that accept video, audio, and text as a major expansion of possible applications. She gives examples involving book pages and help with physics problems.
Teams should compare advanced AI with a simple baseline before adopting it
Weber advises companies that are starting with AI to avoid choosing the newest technology first. They should define the problem clearly, find a simple solution, and then compare more complex approaches against that baseline. The comparison should include the cost of the advanced system, its performance gain, and the team's capacity to maintain it. She says the most sophisticated option often does not justify its cost. A simple classification problem, for example, may not need ChatGPT. A generative model can be easy to use for a quick proof of concept, but a system with significant traffic may need a cheaper and more scalable approach.
Representation affects who stays in technology and who feels able to participate
Verena Weber says she began coaching women after reading that more than half of women leave technology careers around the midpoint of their careers. She connects this to feeling pressure to prove themselves, limited growth opportunities, and a lack of visible futures in the industry. Her coaching focuses on actions individuals can take, such as building confidence, becoming more visible, asking for opportunities, managing stress, and planning career growth. She also says the industry needs more women so the technology reflects the wider population. Demetrios Brinkmann describes trying to reach balanced representation at MLOps Community events, while Weber explains that a more balanced room can make speaking and participation feel safer.
"What we can do is we can adopt certain strategies to work on our confidence, be more visible, be more strategic about our career growth, ask for the things that we want, and go after the growth opportunities."Verena Weber42:45
Who should watch
You are putting a generative AI proof of concept into production and need to think through updates, testing, maintenance, and cost.
Your team is considering a multimodal or large language model for a problem that may have a simpler solution.
You want an example of production NLP work and a candid discussion of career support and representation in technology.
