Streaming systems become expensive and difficult to operate when teams connect Kafka or Kinesis, stream processors, storage, and serving systems themselves.
2
Freshness and cost should be treated as adjustable trade-offs, with checkpoint frequency and stream retention as practical controls.
3
Batch and streaming will continue to coexist because each handles different workloads better, while open storage formats and BYOC deployments can reduce vendor lock-in.
Summary
Rohit Agrawal explains why real-time ML data systems become difficult to build and expensive to run. A typical pipeline connects Kafka or Kinesis to Spark or Flink, then to storage and a serving layer. Each component needs different skills, and teams often own only part of the end-to-end reliability problem. Rohit recommends tuning the system around the freshness each application actually needs. Checkpointing, stream retention, storage layout, and the split between batch and streaming all affect cost. He also describes the appeal of object storage such as S3, managed services, and BYOC deployments. Rohit expects Iceberg to become a common storage layer that lets organizations choose different compute vendors without moving their data. He rejects the idea that every workload will become streaming. Batch systems are better for historical processing, while streaming systems handle recent data and ongoing state. The conversation is practical and honest about the operational tax behind self-managed infrastructure.
Streaming pipelines are difficult because the ecosystem is fragmented
Rohit describes streaming systems as a set of connected tools rather than a complete application platform. A common design starts with Kafka or Kinesis, uses Spark or Flink for processing, writes to a key-value store or Iceberg, and adds a serving layer for applications. Every step needs different skills, and the system needs ongoing reliability work after launch. Large companies may manage Flink at their scale, but smaller teams often lack simpler ways to use real-time data. The result is a high operating burden for teams building fraud detection, recommenders, loan scoring, and insurance claim systems.
Freshness should determine how much the system costs
Rohit frames freshness and cost as two sides of the same decision. Millisecond-level data can justify higher spending, while an application that accepts an hour of freshness should not pay for the same setup. Companies often end up with either an expensive low-latency system or a cheaper system that cannot meet application needs. He compares the desired architecture to a vehicle that can be fast when needed and practical at other times. The useful design has controls that let teams choose different freshness and latency levels for different use cases.
Checkpointing is a simple place to inspect a high streaming bill
Checkpointing lets a failed streaming application recover from a recent saved state, but frequent writes to storage add cost. Rohit says many streaming applications use aggressive default settings, and customers are often surprised when checkpointing appears in their bill. He suggests asking whether the application needs a checkpoint every second, 10 seconds, 30 seconds, a minute, or an hour. Moving from one-second checkpoints to 10-second checkpoints can reduce spending when the application can accept that recovery point. The setting must be judged against the freshness and recovery requirements of the full pipeline.
Object storage can separate compute growth from storage growth
Rohit points to systems that use S3 instead of disks attached to processing nodes. Warpstream, which Confluent acquired, is one example he gives, describing it as Kafka on S3. This design lets teams scale compute and storage independently. With attached disks, adding servers also adds storage, even when the workload mainly needs more compute. Object storage can reduce that coupling, lower infrastructure cost, and reduce some operational work. Rohit presents this as a way to build systems for cloud elasticity instead of copying architectures designed for very large companies.
Stream retention and batch boundaries are cost decisions
Teams need to decide how much history belongs in a stream and how much belongs in batch storage. Rohit gives examples ranging from a day to seven or 30 days of stream retention. An application may need recent activity from the last five minutes alongside statistics from the last seven days or a lifetime total. Those requirements often produce separate streaming and batch pipelines, which duplicate transformations and increase infrastructure and maintenance work. If a system can combine batch and streaming data behind one application interface, teams can keep only recent data in the stream and move older history to batch storage.
Managed services can be cheaper when people and operations count
Rohit says managed services often look expensive when compared only with raw cloud infrastructure. Self-managed systems also require on-call work, maintenance, specialist knowledge, and staffing. He mentions services from Confluent, including Tableflow for moving stream data into Iceberg, along with offerings from Databricks and Snowflake. Managed services can also affect product velocity. If a team expects to launch more models and features, operating a complicated data stack may delay that work or require another team to add resources. The decision should include the operational cost and the effect on how quickly products reach customers.
Iceberg could become a shared storage layer across compute vendors
Rohit compares Iceberg to GitHub for data. In his view, organizations could store data in a common open format and let different vendors read from and write to it. A compute job might use Databricks, Snowflake, BigQuery, or another engine while leaving the data in Iceberg. That would make it easier to use different systems for different workloads and reduce the difficulty of migrating between vendors. He says this trend is already visible as companies adopt Iceberg and vendors align around shared storage and catalog standards. He expects the same pattern to reach structured data and embedding data before long.
Rohit does not expect every workload to move to streaming. Streaming is good at recent data and continuous state, while batch systems handle large historical datasets. He compares the choice to transactional databases and analytical databases, which have different strengths. Hybrid systems can perform both jobs, but Rohit says they may be mediocre at each compared with specialized systems. He expects simpler tools to gain ground against systems such as Spark and Flink when workloads do not need distributed processing at large scale. Streaming has a harder developer experience because it must maintain state continuously, so its DuckDB equivalent has not yet emerged.
BYOC keeps sensitive data in the customer's cloud account
Rohit describes bring your own cloud as a growing deployment model for data infrastructure. Instead of moving sensitive data into a vendor's account, the vendor deploys its system inside the customer's cloud account. He mentions Confluent after the Warpstream acquisition, Redpanda, and Databricks as examples of companies offering BYOC approaches. The model fits the reality that data is difficult to move because of its size and sensitivity. Moving compute to the data can be easier than moving the data to a vendor. Rohit expects both streaming and batch vendors to adopt this model more widely.
"The simplest solution we see is people have a Kafka stream and they would have a stream processor which could either be like a Spark or a Flink and then they basically connect that to storage."Rohit Agrawal05:11
Who should watch
You are designing a real-time ML pipeline and need to understand where infrastructure complexity and operational costs come from.
Your team is choosing between self-managed streaming components and managed services, and you need to include staffing and maintenance in the comparison.
You are deciding how to divide data between streaming and batch storage or considering Iceberg and BYOC deployment models.
