Scaling your data and AI from 0-100 with open source

Maarten Breddels, Pycafe, Pranav Aurora, Mooncake, Simba Khadder, Featureform1:09:51 · Feb 2025 · 422 viewsHosted by Demetrios Brinkmann
Thumbnail for Scaling your data and AI from 0-100 with open source Watch on YouTube
TL;DR
  1. 1

    Pycafe runs Streamlit, Dash, and Python apps in the browser with WebAssembly, so users can share applications without maintaining a server for each one.

  2. 2

    Mooncake combines Postgres, columnar storage, DuckDB, and open table formats so teams can query and serve data while keeping it in Parquet-backed storage.

  3. 3

    Featureform turns feature definitions into managed production pipelines on a team's own infrastructure, using Iceberg for versioning, lineage, and interoperability.

Summary

This session presents three open-source-oriented tools for moving from small experiments to production data and AI systems. Maarten Breddels introduces Pycafe, a browser-based environment for creating and sharing Streamlit, Dash, and Python applications. Its WebAssembly runtime means shared apps do not need a dedicated server, and projects can also be exported as self-contained HTML files or run on a self-hosted server. Pranav Aurora presents Mooncake, which keeps Postgres as the starting point while adding columnar execution and Iceberg or Delta Lake tables backed by Parquet files. His example moves data from Postgres into Python, applies an LLM transformation, and reads the resulting table back through Postgres. Simba Khadder explains Featureform's virtual feature store. It lets data scientists define features in SQL, PySpark, or another supported form while Featureform manages execution, versions, lineage, monitoring, and serving. Iceberg provides the shared table layer across execution engines.

Key ideas
02:37

Pycafe runs Python applications in the browser instead of assigning each app a server

Maarten Breddels describes Pycafe as a platform for creating and sharing Streamlit, Dash, and Python apps online. The editor, file browser, preview, and terminal run in the browser through WebAssembly and Pyodide. A shared app starts when someone opens its link, so having hundreds of published apps does not mean running hundreds of servers. Maarten says this makes the running cost close to zero apart from a small amount of platform infrastructure. The browser sandbox also limits the damage from arbitrary code execution to the virtual environment in that browser.

10:15

Pycafe makes reproducible examples and pull request previews easy to share

Maarten focuses on two open-source uses. A maintainer can answer a GitHub issue with a working Pycafe example instead of sending a code snippet that only works in their own development environment. Users can open the link, edit it, and test the behavior immediately. Pycafe can also create a playground for each pull request. Reviewers can open the playground, install the proposed change, and perform visual or API checks. Maarten says this can support many open pull requests without a server running for each preview.

22:17

Pycafe supports several ways to keep application data private or portable

Pycafe snippets store their data in the URL, while projects require a login and can hold larger files in the platform database. For organisations that cannot send internal applications to the public service, Maarten describes a self-hosted Python server that can be installed with pip. He also shows an export to a self-contained HTML file. The file contains the application and its CSV data, so it can be placed in an approved file store or sent to another person. His argument is that companies already understand how to protect files, while introducing another database can require more decisions.

24:24

Mooncake keeps Postgres as the entry point while moving analytical data into open lake formats

Pranav Aurora frames Mooncake around a path from Postgres to Postgres plus a data lake. Postgres provides extensibility, while a lake stores files in object storage with metadata that allows stateless querying. Mooncake adds a column store and DuckDB's vectorized execution to Postgres. The same table can appear as a Postgres column-store table and as an Iceberg or Delta Lake table outside Postgres. Those tables use Parquet files, so other engines can read them. This lets teams continue using Postgres semantics for transactions and joins while keeping data in an open format.

29:54

Mooncake uses metadata and file skipping to make Parquet-backed analytics practical

Mooncake stores detailed Parquet metadata in Postgres, including statistics such as minimum and maximum values for columns and aggregated statistics for row groups. During a query, segment elimination or file skipping can reduce the amount of data that must be read. Pranav presents this as a way to run analytical query shapes in Postgres without copying the data into a proprietary storage format. In his workflow, a column-store table is created in Postgres, its data is stored in an S3 bucket, and Python reads the resulting Delta Lake table directly. A processed data set can then be written as another table and queried through Postgres.

34:13

Mooncake treats the lake as a shared storage layer for feature work and LLM transformations

Pranav demonstrates a workflow over Reddit comments. He groups comments by author and subreddit, calls GPT-3.5 Turbo to produce a short description of each user, and writes the result to a new Delta Lake table. Postgres can read that table from the S3-backed Parquet files. He presents this as a simple feature extraction or feature engineering path: query through Postgres, process with Python or another engine, and serve the resulting data through Postgres. The benefit is that teams can bring different execution engines to the same open table rather than maintaining separate copies in a closed database.

49:35

Featureform connects notebook-style feature experiments to production pipelines

Simba Khadder describes a gap between data science notebooks and production machine learning. In production, teams must deal with changing data, errors, monitoring, streaming, backfills, and features that are only available at inference time. Featureform's virtual feature store lets data scientists define features in SQL, PySpark, or another supported form while Featureform orchestrates the work on their infrastructure. It records versions, documentation, runs, lineage, and monitoring. Simba wants data scientists to deploy and iterate on their own features instead of handing notebooks to another team that rewrites them as production code.

01:01:00

Iceberg gives Featureform one table layer across different execution engines

Featureform originally had separate code paths for different providers, including different Spark environments. Simba says Iceberg lets the project focus on a common table layer. Data files are stored as Parquet, and metadata points to the files that make up each table version. Predicate pushdown lets an engine skip files that cannot match a query or join. Featureform can write Iceberg while using Spark, Snowflake, BigQuery, or another supported system for execution. Users can retrieve the results as data frames, train models from training sets, and serve online features from systems such as DynamoDB.

"We think that data scientist one should be able to deploy their own and should be kind of managing and have the freedom to try things, release things, iterate by themselves without being dependent on another team."Simba Khadder53:13
Who should watch
  • You maintain Python or Streamlit projects and need reproducible examples, interactive demos, or pull request previews without deploying a server for every change.
  • Your team is deciding whether Postgres can remain the operational entry point while analytical and machine learning workloads move onto an open lake format.
  • Data scientists are passing notebooks to another team for productionisation, and you need feature definitions, lineage, versioning, monitoring, and serving on your own infrastructure.