Carl Steinbach argues that data platforms should bring back database abstractions such as tables, views, and declarative languages while keeping the storage and execution flexibility introduced by Hadoop.
2
Views provide a reusable and versioned interface over data, while UDFs are harder to distribute and appear as black boxes to query optimizers and lineage systems.
3
Data lakes and data warehouses are moving toward convergence, with systems adding both flexible programmatic access for data scientists and SQL interfaces for analysts.
Summary
Carl Steinbach describes a career spent making Hadoop and Spark analytics feel more like databases. He explains LinkedIn's Dolly work, which decouples business logic from execution engines by translating relational languages and supporting portable UDFs. This lets teams migrate old Pig Latin scripts and run views across engines such as Spark and Presto. Carl prefers views to UDFs for reuse because views can be catalogued, versioned, optimized, and inspected for lineage. He compares data sets to services with versioned APIs, allowing schema changes without breaking every downstream user. The conversation then moves to data lakes, data warehouses, access control, and the need for a table or record layer above files. Carl argues that this layer makes fine-grained permissions and caching practical. He expects data lakes and warehouses to converge. He also uses Richard Gabriel's "Worse Is Better" essay to explain why Hadoop spread before gradually adopting database-style abstractions.
Carl Steinbach came to data systems through databases and Hadoop
Carl says his college offered no database course, so he assumed databases were a finished subject. His first job at Oracle changed that view. He found databases interesting because they connect systems, algorithms, and languages. At Cloudera, where he was one of the first ten engineers, he chose Hive and argued that Hadoop needed a database-like interface. His broader work has focused on making offline analytics with Hadoop and Spark look more like a database, with tables and views instead of files and higher-level languages instead of MapReduce.
Dolly decouples business logic from execution engines
At LinkedIn, Carl started Dolly, originally meaning data access at LinkedIn. He describes it as a "post-modern database" that adds indirection and optionality across the stack. The team translates one relational language into another and provides portable UDFs. A view can therefore be written in SQL and executed on Spark, Presto, or other engines. This separates the logic people write from the engine that runs it, which avoids forcing every team to rewrite its work whenever an engine changes.
Transpiling made migration away from Pig practical
Carl explains that Hadoop introduced choice in storage formats, storage backends, and programming interfaces, but languages remained tied to particular execution engines. LinkedIn had a large collection of Pig Latin scripts and an aging Pig engine. A transpiler could automate migration instead of asking engineers to manually translate 10,000 scripts into SQL. Carl says the larger benefit is that teams can write reusable views once and execute them in different contexts. He connects this approach to software engineering practices such as source control, dependency management, and reusable components.
Views provide a better reuse and compatibility mechanism than UDFs
A view looks like a table but is defined by a query. In Carl's system, most views are virtual and run when queried. A database compiler can rewrite a query against the view definition and apply optimizations such as predicate pushdown and column pruning. Views also live in a catalog, so users can reuse them by name without copying implementation details. When an underlying table changes, the owner can update the view definition while keeping its users' references stable. UDFs can express imperative logic that SQL cannot, but they are harder to distribute and are opaque to query optimizers and lineage systems.
Versioned views let data schemas evolve without breaking every consumer
Carl compares a data set to a microservice with a schema as its API. A table usually cannot expose multiple API versions at once, so removing a column or changing its type can immediately break downstream users. This produces old, duplicated, and confusing columns. LinkedIn publishes a new version of a view to its catalog whenever the definition changes, appending the version number to the view name. Consumers can migrate from version one to version two during a transition period. A pointer to the latest view is also available for users who want automatic updates.
Data lakes and warehouses are moving toward the same architecture
Carl describes a data warehouse as a SQL analytics database where data is loaded into a system's format. A data lake puts data on disaggregated storage such as S3 or GCS and allows different tools and languages to analyze it. He sees Databricks and Snowflake moving toward each other's markets. Databricks has added capabilities for SQL analysts, while Snowflake is working on a programmatic API alongside SQL. His thesis is that data lakes and data warehouses will eventually become the same kind of system, combining flexible storage and execution with database-style interfaces.
A table or record layer makes permissions and caching manageable
Carl argues that file systems expose the wrong abstraction for managing very large data sets. Files force users to think about directories, partitions, formats, and bytes. A record service can hide those details and expose tables, records, and views instead. That layer can enforce column-level and record-level access controls, which are difficult to express with file permissions. It can also cache data close to compute. Carl notes that LinkedIn's batch workloads are predictable enough to pre-warm caches, and that caching can reduce storage I/O costs in some cloud environments.
"Worse Is Better" explains why Hadoop spread before database ideas returned
Carl uses Richard Gabriel's essay "The Rise of Worse Is Better" to explain technology adoption. The approach associated with Bell Labs accepts a less complete interface when it keeps the implementation simple and available. Carl says Hadoop delivered much of what users needed quickly, even though it exposed too many low-level details. Over time, data platforms have moved back toward database ideas such as declarative queries, tables, views, and managed access. He contrasts this with systems such as search and databases, where complex implementations support simple interfaces.
"The thing that provides 90% of what people want and which is available right away and which will incrementally approach the better thing over time is the thing that's more likely to succeed."Carl Steinbach52:20
Who should watch
You maintain analytics code that is tied to one execution engine and need a migration path without rewriting every job by hand.
Your data platform exposes files and storage details directly, and you need clearer interfaces for schema changes, reuse, permissions, or lineage.
You work across data science and data analyst workloads and want to understand why data lakes and warehouses are converging.
