Hugging Face Cofounder on AI Agents, LLMs and Open Source

Thomas Wolf, Hugging Face, Euro Beinat, Prosus Group27:31 · Dec 2024 · 913 views
Thumbnail for Hugging Face Cofounder on AI Agents, LLMs and Open Source Watch on YouTube
TL;DR
  1. 1

    Hugging Face grew from transfer learning and NLP into a broad platform for models, datasets, demos, and research across text, images, speech, and robotics.

  2. 2

    Agent projects should start with a specific business or technical domain and a defined set of tools, since general-purpose agents are still unreliable.

  3. 3

    Smaller open models will run on phones and laptops, while synthetic data and test-time compute will help improve models for specific tasks.

Summary

Thomas Wolf describes Hugging Face as a community-driven platform that grew from early work on transfer learning and NLP into a home for models, datasets, and demos across many modalities. He expects robotics to become a major area for open-source work. On agents, he argues that the word covers very different systems, from coding assistants to business automation and robots. Tool use is shared across these systems, but teams should currently build around a specific domain and tool set rather than wait for a general agent. Wolf expects model development to spread across many sizes and applications. He points to small models that can run on edge devices, synthetic data for specialised fields, and test-time compute that gives models more time to reason. He is especially optimistic about affordable open-source robotics, including a low-cost robotic arm that can perform tasks such as folding clothes.

Key ideas
02:56

Hugging Face grew through transfer learning and an expanding model community

Thomas Wolf connects Hugging Face's early direction to transfer learning, where one trained model could support many tasks. This helped teams share training compute instead of repeatedly building everything from scratch. The company first grew through NLP, then gained wider visibility with image generation and Stable Diffusion. Speech interfaces are also becoming more active, and Wolf expects robotics to become the next major area. He says the platform has passed one million public models, alongside a larger number of private models.

05:07

The platform follows fast-moving releases across models, datasets, and demos

Wolf says Hugging Face's trending page changes quickly because new releases move attention across different modalities. He cites Flux for photorealistic image generation, Pyramid Flow as an early open-source video model, Qwen2.5-Coder for code generation, and Microsoft's OmniParser for helping agents understand screen content. He also describes Spaces, where people can try models through demos. In his view, the release of a model increasingly includes an accessible demonstration, so users can test it rather than only read about it.

07:34

Agents share tool use but cover very different kinds of systems

Wolf rejects the idea that one agent framework will cover every use case. A code-completion agent, a business-process automation system, and a robot moving around a kitchen have very different requirements. They do share a need to interact with tools, APIs, software, and eventually the physical world. Hugging Face has started exploring hosted tools, but Wolf advises teams to focus on the tools required by their own vertical. General agents may arrive later, while current systems can still spend too long thinking without producing a good answer.

09:59

A tool marketplace will need more than a simple function directory

Euro asks whether agent tools will develop common standards or a marketplace. Wolf says several startups are working on the problem, and OpenAI has explored generic function calling. He expects the problem to be more complicated than a basic marketplace because agents need the right details for interacting with each tool. Hugging Face prefers to remain a low-level platform and encourage other companies to build products on top of it. He describes an ecosystem company built on the platform as a success, even if Hugging Face is not the application provider.

12:17

Open models can matter without winning every general leaderboard

Wolf questions whether next year's most capable general models will be open source, since closed companies can learn from open models while keeping their own work private. He also questions whether a general leaderboard will matter for every business or agent use case. Different tasks may reward different models, and the industry is moving toward many model sizes with different costs and performance. He expects small models to become much more useful, including models that run directly on phones and laptops. Wolf says this area is being pushed heavily by Meta, Alibaba's Qwen team, Hugging Face, and Mistral.

19:39

Current models already leave room for many useful applications

Wolf agrees that teams should build with current model capabilities instead of waiting for a future model. He says models around the level of GPT-4 and Llama 3.2 can support many projects when integrated into a useful interface. ChatGPT's impact came partly from making a model easy to access, while chat interfaces themselves can become tiring when users want assistance embedded in daily work. Wolf estimates there is a large amount of startup and product work available around today's models, even if model scaling slows.

21:14

Synthetic data helps specialised tasks more than general model training

Wolf describes synthetic data as useful in some settings, but not as a complete answer to general model improvement. Hugging Face explored this through a project called Cosmopedia. Generating large amounts of data from an existing model does not automatically keep general capabilities improving. It can work well for particular areas such as mathematics and code generation. Wolf also recommends it for business applications with limited data, where carefully designed synthetic examples can expand a fine-tuning or adaptation dataset.

22:57

Test-time compute gives models room to reason before answering

Wolf identifies test-time compute as another important direction. Instead of training a model to produce the right answer immediately, a system can give it more time to think before responding. He relates this to the difference between fast, direct answers and slower reasoning. He also connects this approach to agents, which can use tools and multiple steps to solve a task. The practical limitation is cost. More computation at response time only helps if teams can make it affordable.

24:00

Cheap open-source hardware could bring capable robots into homes

Wolf is excited by recent robotics work because robots have begun handling tasks such as cooking and folding clothes, including on hardware far cheaper than high-end systems. Hugging Face is working on both software and low-cost hardware. He says Remy pushed an open-source robotic arm priced at about $100, and the team has started using it for tasks such as folding clothes. Wolf compares the current research progress to an early breakthrough phase and expects a later point when ordinary people can use robots at home for tasks that machines previously could not perform.

"I think today the good way to tackle an agent project is really to understand which tool do I need, in which framework should I be, and be kind of specific until this day comes of a generic agent model."Thomas Wolf09:37
Who should watch
  • You are deciding whether an agent project should use a general framework or a narrow set of tools for one domain.
  • You want to understand where open models may matter when the best general-purpose models are closed.
  • You are evaluating small language models, synthetic data, or low-cost robotics for products that need to run outside a large data centre.