On-device AI improves response time and privacy because data can stay on the phone, car, PC, or other edge device.
2
Qualcomm AI Hub translates, optimizes, profiles, and validates models on physical Qualcomm devices in the cloud, then provides a deployable package.
3
Memory capacity, battery use, model compression, and differing hardware generations limit what generative AI can run locally.
Summary
Krishna Sridhar explains why deploying AI at the edge is difficult and why it is useful. Phones already run many models for photography, speech, and assistants under tight latency and privacy constraints. Newer devices also run language models for tasks such as summarization and text completion. The deployment problem sits between two systems that keep changing: model frameworks and architectures on one side, and CPUs, GPUs, and neural processors on the other. Qualcomm AI Hub tries to make this process simple. A developer can submit a trained model, select target devices, receive performance and accuracy measurements from physical devices in the cloud, and download a package for deployment. Sridhar says memory has become a larger constraint than raw compute for many language models. Battery use also forces trade-offs between speed and efficiency. He describes use cases including cricket ball tracking, local security analysis, music tools, and computer search features.
Edge AI makes fast and private device experiences possible
Sridhar says his work on Apple's Core ML showed why edge deployment matters. Face identification needs to process biometric data locally, without sending it to the cloud, and must respond within roughly 300 milliseconds. Smartphone cameras also run many models for framing, face detection, recoloring, and selecting better frames. Speech processing and text-to-speech for assistants can happen locally too, while heavier question answering may still use servers. The device already has substantial compute, so the challenge is putting it to work within a short response time and a limited power budget.
Deployment infrastructure has to connect two fast-changing systems
Qualcomm does not build every model used in phones, cars, or other devices. Sridhar says its role is to build systems that let model developers bring their work onto hardware quickly. The cloud-side ecosystem keeps changing through new frameworks and model architectures. Hardware changes just as quickly, with newer phones and computers adding dedicated neural processors. The infrastructure therefore needs to translate current models onto current and older device generations without forcing every application developer to understand each hardware target in detail.
Similar model architectures simplify support across different use cases
Sridhar says generative AI has added language and image-generation workloads to the existing computer vision and speech workloads. Many of these models use Transformer-like architectures, which makes it possible to build more shared deployment systems instead of entirely separate systems for cameras, language models, and driver assistance. The hardware requirements still differ. Language summarization may tolerate four-bit weights, while driver assistance needs higher precision for calculations involving pedestrians and nearby vehicles. These precision choices affect how models are mapped to device hardware.
Heterogeneous compute trades flexibility for speed and power efficiency
Modern devices combine CPUs, GPUs, and neural processing units. The CPU is flexible and programmable, although it may be slower or less efficient for some workloads. A neural processor is specialized, so it can provide roughly an order of magnitude better performance and power efficiency for suitable operations, while being harder to program for arbitrary tasks. The GPU sits between those options. Hardware and software teams must decide which operations deserve dedicated hardware and which can remain in a more flexible software implementation.
Memory and battery life now constrain local generative AI
Sridhar says he did not expect memory to become such a fast limit for language models. The ability to run larger models depends on memory technology and algorithms that consume less memory, not only on raw compute. Power is another constraint across phones, PCs, industrial devices, and cars. A model that can generate 50 tokens per second may waste battery when a person can comfortably read only 10 tokens per second. Local generative AI also needs more work before systems comparable to ChatGPT can fit within constrained devices.
Shared computation can keep multiple on-device models small
When Demetrios asks whether devices will need many small language models, Sridhar compares this with image processing models that share a backbone and use different heads. Sharing compute and memory can avoid storing and running completely separate models for every task. Storage matters too. Users do not want many gigabytes of their phone consumed by models because they still need room for photos and other data. Sridhar expects future systems to share more compute, memory, and storage across related use cases.
Qualcomm AI Hub automates model translation and physical validation
Sridhar describes Qualcomm AI Hub as a system for developers who want to deploy models across phones, PCs, cars, and industrial devices. The stated goal is deployment within five minutes and five lines of code. A developer selects a trained model and target devices. AI Hub translates the model for the relevant processor, applies optimization, runs it on physical devices provisioned in the cloud, and reports latency and accuracy. The developer can then download the model package and deploy it locally. The system also supports programmatic automation and integrations with AWS and other services.
AI Hub can deploy the same model across multiple generations of Qualcomm hardware. A component on the device detects the generation and maps operations to the features available on that chip. The model may run faster on a newer refrigerator, phone, or other device and slower on an older one, while remaining deployable. For efficiency work, developers can inspect whether operations run on the CPU, GPU, or neural processing unit. Moving more work onto the neural processor generally improves power efficiency.
Edge deployment enables applications that filter events before using the cloud
Sridhar describes a cricket application that uses a phone camera for real-time ball tracking and automatically clips a practice session down to important moments. He also mentions local music tools, Microsoft's Recall feature, code generation, and security systems that analyze camera feeds on-site. In the security examples, the device filters out routine footage and sends only important events to the cloud. He says this approach can make large deployments, such as airport security systems, more economical because less raw video needs cloud processing.
"You have two fast moving things one on the top the bottom so that's what makes this a pretty interesting problem because you got to map a fast moving thing at the top to a fast moving thing at the bottom."Krishna Sridhar11:08
Who should watch
You are choosing between cloud and on-device inference for a phone, PC, car, camera, or industrial product.
Your team has a trained PyTorch model and needs performance, accuracy, and processor-placement data before shipping it to edge hardware.
You are working with language models on devices where memory, battery life, storage, or support for several hardware generations limits deployment choices.
