TinyML becomes difficult when a model has to fit into a few kilobytes of RAM and run for a year on battery power in a remote location.
2
LLM applications are easy to prototype with an API, but reliability, cost, latency, and provider limits become serious problems as the product grows.
3
Soham expects teams to fine-tune and eventually own smaller, domain-specific models, while treating prompts and output tests as sensitive product assets.
Summary
Soham Chatterjee describes a career that began in electrical engineering and moved through IoT, deep learning, edge computing, and TinyML. He explains how an Arduino Nano may have only a few kilobytes of RAM, which must hold the model, application code, sensor data, and processing logic. Remote deployments add battery and connectivity limits. He then compares those constraints with large language model systems. An OpenAI API can help a startup validate an idea quickly, but long prompts, chained calls, unreliable outputs, rising token costs, and the absence of published service-level agreements make production use harder. Soham and Archana are building a Chrome extension to learn about those problems in public. He argues that teams will need to fine-tune and deploy models they control, rather than depend entirely on external APIs. He also explains that compression can increase bias by preserving performance on common examples while losing less frequent cases.
Soham entered machine learning through IoT data and electrical engineering
Soham Chatterjee studied electrical engineering rather than computer science. During his undergraduate studies around 2014 or 2015, IoT was becoming popular, but many people were not yet using the data from those devices to build intelligent systems. He started exploring how sensor data could support machine learning and deep learning. A student-run university lab exposed him to IoT, deep learning, and virtual reality. A talk by Malay, who later led a deep learning lab at Sama, led to an internship opportunity and then work as a deep learning researcher. Soham also experimented with quantum computing, although he says he is not qualified to discuss it in depth.
TinyML means fitting the whole system into severe hardware limits
For Soham, the smallest form of TinyML uses devices such as an Arduino Nano with only a few kilobytes of RAM. That memory has to contain the model, the code that executes it, sensor-reading logic, and data processing. A remote deployment may have no utility power, so the device has to run from a battery and operate autonomously for at least a year. TinyML devices form a range. The constraints ease as systems move from very small devices to Raspberry Pi computers, phones, and other embedded hardware. The smallest devices may have no useful connectivity, so they cannot send a query to a remote server.
TinyML applications can keep sensitive data inside a physical site
One early application involved a medicine manufacturer that needed to check whether boxes leaving a production line contained the medicine and the required information sheet. The company could not send confidential factory data to a remote model. Space on the factory floor was also expensive, so the system had to be small. Soham used a neural compute stick connected to a Raspberry Pi as an accelerator. The accelerator handled the compute-intensive machine learning work efficiently. In another project, a neuromorphic camera and a custom chip consumed only a few milliwatts while monitoring movement, such as cars or illegal activity in a jungle.
Large language model systems inherit many of the constraints found at the edge
Soham says latency is a major challenge in both TinyML and LLM deployments. Teams also have to consider power, privacy, control over the model, model compression, and cost. In a business setting, provisioning one or two GPUs may not make financial sense for a model that does not generate enough value. Fine-tuning and compression can reduce the size of a model. He compares this with knowledge distillation in TinyML and mentions the Alpaca model as an example of a model compressed from a LLaMA model. Compression also creates a risk because it can increase bias.
An API makes LLM prototypes easy while production systems become expensive and fragile
Soham and Archana started building a Chrome extension because they wanted to understand how LLM products work instead of only seeing closed-source applications built around them. Their experience showed that a startup can create an application quickly and cheaply by calling an API and adding prompts. That approach becomes harder as the product gains features and needs more reliable output. Few-shot prompting may give way to chain-of-thought prompting and longer prompts, which increase cost. Chains and agents make reliability worse when the output of one API call becomes the input to another. Some teams add watcher models to check errors, prompt injection, trust issues, and incorrect outputs, but Soham does not think that approach scales well.
Prompt engineering must be maintained like application code
Soham describes how prompts that worked well with the DaVinci 002 API stopped working as well after newer models became available. Moving an API endpoint is simple, but migrating the prompt engineering and checking every output is not. His team stayed with the older endpoint until reduced capacity made responses take a minute or more, forcing a move. A new model can change reliability, latency, and benchmark results, and an external provider may not publish service-level agreements or performance guarantees. Soham calls prompts a company's current differentiator and says teams should protect them like API keys. They should also keep benchmark outputs and test new models against them.
Small domain-specific models may replace dependence on general-purpose APIs
Soham expects teams to move toward fine-tuned models that they can deploy and control themselves. He does not think models will make data scientists obsolete. APIs make it possible to build and release a product quickly, even when a company lacks training data, but increasingly complex applications will require custom models. A feature such as changing the tone of text or summarizing it would have taken months to build without existing LLMs. Soham says the next step for his own product is to fine-tune and perhaps compress a model. He does not add vector databases simply because they are available; he wants a concrete need first.
Model compression can discard rare cases and reduce a model's richness
Soham explains the bias problem through an animal-recognition camera trap. A large model may recognize both common animals and rare animals such as a snow leopard. During compression, the model optimizes for examples that appear often and can forget classes with fewer examples. The compressed model therefore keeps performance on common cases at the expense of the long tail. Soham expects a similar effect in language models, where compression could remove vocabulary and some of the richness of a larger model. He connects this concern to Sarah Hooker's research on model compression and bias.
"I don't think that that's something that's scalable, and I don't think that that is something that makes a lot of sense from the cost perspective as well."Soham Chatterjee23:31
Who should watch
You are deploying models on small devices where memory, power, connectivity, privacy, or physical space limits the design.
You are considering an LLM API for a product and need a candid account of what changes when prompts, chains, traffic, and costs grow.
You want to compare edge ML constraints with the practical problems of fine-tuning, compressing, testing, and operating LLMs.
