Maher Hanafi's team reduced LLM latency by up to 70% with TensorRT LLM, which rewires a model for the target NVIDIA GPU architecture.
2
Using larger GPUs, one model per GPU, larger KV caches, in-flight batching, and scheduled and dynamic scaling reduced total GPU runtime and spending.
3
Betterworks treats AI as a shared platform and improves it through a flywheel of planning, building, running, and optimization, with human review and safeguards in production.
Summary
Maher Hanafi describes how he moved from having no AI experience to leading self-hosted LLM infrastructure for an HR technology company. He explains the hidden work behind AI features, including GPU selection, latency, throughput, cost, cold starts, model quality, privacy, and language support. His team used AWS FSx and container images with embedded models to reduce cold-start time. TensorRT LLM delivered up to 70% lower latency by adapting models to NVIDIA GPU architectures. They also learned that one model with a large KV cache and in-flight batching could produce more throughput than multiple tightly packed models. Maher connects infrastructure work to product planning through a flywheel, and argues for reusable AI platform components rather than isolated features. He also discusses an AI engineering lab, where engineers developed review rules for coding agents and changed their software development process. Enterprise compliance remains a constraint, especially around model training data and Chinese-origin models.
Engineering leaders need to learn AI alongside their teams
Maher says he started with no AI background and learned by asking junior machine learning engineers to explain how their systems worked. When engineering leaders are asked to build AI without enough specialist resources, he believes learning has to come first. His approach is to avoid vague directives such as telling a team to build an MCP server or add RAG without understanding the basics. He learns with the team and asks them to learn with him. He sees this as both a personal need and a response to a market where traditional technology leaders are increasingly expected to lead AI work.
The AI iceberg is the operational work hidden behind the feature
Maher calls the less visible work behind an AI product the AI iceberg. Once a company self-hosts models, it must choose GPUs, architectures, and open-source models, then manage performance, latency, throughput, response quality, accuracy, and cost. He says these factors form a trade-off pyramid, and no use case can optimize every edge at once. For Betterworks, an HR technology product, privacy and responsible AI matter, while efficiency, accuracy, latency, and cost must be balanced for its particular workloads. The right trade-off depends on the use case.
Scheduled and dynamic scaling reduce GPU waste in predictable HR workloads
Betterworks examined when customers used its HR product and found lower demand at night and on weekends. Maher's team introduced scheduled scaling for those patterns, then added dynamic scaling for spikes at the end of a week, quarter, or year, when customers upload more feedback, conversations, and recognitions. This differs from using an API provider, where the customer does not manage GPU capacity. Self-hosting required the team to spin GPUs up quickly and turn them off when demand fell. The same approach also had to be adapted for different regions with different traffic patterns.
Cold starts improved when storage and model packaging were changed
Maher's team used AWS FSx to download model files faster than their previous approach with S3. They also stopped starting an empty container and downloading the model after launch. Instead, they embedded the model in the container image. A new Kubernetes node could then start with the model already present. Maher says this removed minutes from the cold-start process. The GPU itself still took time to start, but model downloading was the largest part of the delay that the team could address. He says they focused on the individual startup steps that consumed the most time.
One model per GPU with KV cache and in-flight batching produced more throughput
Maher initially tried to fit the smallest possible model on a GPU and even considered running two models on one GPU to handle more requests. He later learned that the remaining GPU memory could be used for KV cache, which enables batching. With in-flight batching, completed requests can leave a batch while new requests enter without waiting for every request in the original batch to finish. The team therefore chose one model per GPU and used the remaining memory for KV cache. This was counterintuitive compared with ordinary CPU and memory planning, where unused capacity often suggests running another system.
More expensive GPUs can lower total cost when they finish work faster
Maher says his team moved from smaller GPUs to larger, more expensive machines because the relevant unit was total runtime rather than hourly machine price. If a new GPU cost 30% more but ran for 50% less time, he considered it cheaper overall. Better throughput meant the team did not need as many machines. Scheduled scaling reduced capacity further, with fewer GPUs during low-demand periods and more during working hours. Maher reports that TensorRT LLM, larger GPUs, one model per GPU, and higher KV-cache capacity let the team serve traffic with fewer machines while also improving latency and allowing larger models.
AI features need a planning and optimization flywheel
Maher describes a four-part flywheel: plan, build, run, and optimize. Planning brings together product, engineering, customer success, legal, ethics, sales, and customer councils to find AI work with high impact and low technical effort. The team starts with a small scope instead of committing to a long project before proving its value. After building and deploying through stages, the team measures usage, customer feedback, quality, accuracy, and performance. Optimization includes new models, lower latency, new languages, safeguards, and evaluation checks. Maher says optimization takes at least half of the flywheel because models and techniques keep changing.
A shared AI platform prevents every product feature from becoming a separate system
Maher distinguishes vertical AI features, such as goal setting or feedback assistance, from a horizontal AI platform that can combine those capabilities into larger workflows. A manager program could use information from goals, feedback, conversations, and skills to suggest actions across several product areas. The team can also preprocess summaries during low-demand periods, then use that prepared data later instead of running every operation at request time. Maher wants the infrastructure, safeguards, and model integrations to be reusable across features. He says building products in silos would force the team to rebuild the same systems repeatedly.
AI coding tools changed the engineering process through shared review rules
Maher created an AI engineering lab where engineers could test coding tools on lower-priority projects within budget limits. Junior and mid-level engineers adopted the tools quickly because they could attempt work that previously required more training. Senior engineers were more skeptical, then became involved after reviewing generated code and finding weak quality, missing design patterns, and components that did not match the organization's standards. The team agreed that time saved by AI should be spent reviewing the output. Engineers still need to understand and own the code. They also developed shared instructions, memory, and skills for agents so the tools would follow the team's patterns instead of creating isolated approaches.
"I was doing this iteratively with my team, getting into talking to AWS, talking to NVIDIA, understanding what tools we have available to go and optimize one at a time."Maher Hanafi17:35
Who should watch
You are self-hosting LLMs and need to reduce GPU cost, latency, or cold-start time without handing inference to an API provider.
Your AI feature works as a proof of concept, but you need a way to choose production work, measure customer value, and reuse infrastructure across product areas.
Your engineering team is adopting coding agents and needs practical rules for review, model spending, design patterns, and code ownership.
