Confluent, an IBM company, today announced new capabilities for building a Streamhouse, a new type of data architecture. The Streamhouse architecture keeps business context current for AI agents, applications, and analytics. As a result, these systems make decisions and take action on accurate information.
Lightning Tables is now generally available on AWS. With this release, Confluent completes its five-pillar Data Streaming Platform. The platform now unites Stream, Connect, Process, Govern, and Query capabilities for building a Streamhouse. Confluent also sits at the center of IBM’s data strategy. It combines real-time data streaming with IBM’s integration, governance, and AI capabilities across hybrid environments.
Why Enterprises Need a Streamhouse
Data lakes, warehouses, and lakehouses were built primarily for reporting and other batch jobs. However, AI agents now take on more work across the enterprise. They must make decisions using information spread across systems that update at different times. When those systems hold different views of the business, agents can act on outdated or conflicting information. Moreover, those actions can carry into other workflows.
A Streamhouse addresses this architectural problem. It continuously captures, processes, governs, and serves current business context across agents, applications, and analytics. Consider an airline managing a canceled flight. Today, different systems hold conflicting views of the same operation. Flight operations track where the aircraft is. Scheduling tracks when it will depart. Reservations track which seats are available.
Because those systems update at different times, an autonomous rebooking agent can assemble an itinerary that looks valid in each system. In practice, it fails. A seat is available, but the aircraft cannot arrive in time for the passenger’s connection. Then customer-service teams and notification systems act on the same conflicting views. Consequently, the disruption spreads.
With a Streamhouse, each change updates a shared picture of the airline’s operations. When an aircraft’s location changes, a departure is delayed, or a seat is booked, the source system sends that update into a continuous flow of events. The Streamhouse combines those updates into the current business context. It then makes that context available to rebooking agents, customer-service teams, and applications as they work. Instead of piecing together separately updated information, they check aircraft location, departure time, and seat availability together before offering a passenger a new itinerary.
Enterprises already see results. Accenture uses a Streamhouse architecture powered by Confluent to connect data across its global business.
“Operating across more than 120 countries, our global enterprise was previously constrained by legacy batch and point-to-point architectures where duplicative data extractions drove up operational overhead and analytics data lagged. Moving to a Streamhouse architecture powered by Confluent decouples data producers from consumers, transforming high-volume event streams into governed, reusable real-time data products that flow seamlessly across operational and analytical systems. With Confluent handling infrastructure automation and providing seamless enterprise connectivity, our developers operate in a standardized environment without the friction of building custom integrations or micro-managing data movement. This continuous data foundation has dropped our time-to-insight from months to days and slashed reporting times, empowering our teams to focus on delivering high-value AI for our 800,000 employees.”
— Thomas Mahaffey, SAI Chief Data Strategist, Accenture.
What Makes a Streamhouse Different
Three architectural principles define a Streamhouse: current, mission-critical, and decentralized. It maintains continuously updated business context. Engineers build it to production service levels. In addition, it meets data where it lives, without requiring consolidation first. Furthermore, it works alongside existing data warehouses and lakehouses. It simply adds a real-time operational layer for decisions and actions.
“As an AI-native commerce platform delivering personalized shopper experience and operational and financial advantage to merchants, Extend requires exceptional adaptability across rapid merchant onboarding, real-time pricing adjustments, and instant fraud detection. We initially struggled with traditional message queues and static data lakes that lacked event replay capabilities, persistence, and strict schema enforcement at scale. Implementing a 100% serverless Streamhouse architecture anchored by Confluent Cloud turns Kafka into our operational source of truth, enabling downstream microservices to asynchronously consume and query real-time data without breaking contracts. This inherently decoupled streaming foundation directly provides the flexibility and responsiveness required to scale our operations, adapt to live market signals, and power real-time agentic AI workflows.”
— Jonathan Kropp, Director of AI and Architecture, Extend
Confluent’s Data Streaming Platform Now Offers a Complete Streamhouse
Confluent’s Data Streaming Platform (DSP) provides a complete, unified foundation for building a Streamhouse. At its core sits Apache Kafka®, which captures and streams business events. On its own, however, Kafka leaves teams to assemble everything else they need to turn those events into current business context. The DSP brings that together in one platform. It now adds Query as its fifth pillar alongside Stream, Connect, Process, and Govern.
- Stream: Built on Apache Kafka and the Kora engine, it captures and streams business events across systems with sub-millisecond latency in Confluent internal testing. Actual latency varies by configuration and workload.
- Connect: More than 150 fully managed and custom connectors bring data from enterprise systems into Kafka and deliver it downstream. Hybrid connector management also extends how teams manage those connections.
- Process: Apache Flink combines, enriches, and derives business context close to the source as data changes. Additional batch execution capabilities help initialize streaming workloads from existing data.
- Govern: Teams discover, classify, and secure data through lineage, catalog, quality, encryption, schema, and access-control capabilities.
- Query: This pillar serves current business context to analytics, agents, and applications through the interface that fits each job.
Within Query, Confluent Tableflow makes streaming data available to lakehouses and warehouses as open table formats. Meanwhile, the Real-Time Context Engine serves current business context to AI agents through the Model Context Protocol (MCP). Lightning Tables gives applications low-latency SQL and API access to the current state of the business, so they respond as conditions change.
Lightning Tables: Fast Access to the Current State of the Business
Lightning Tables on Confluent Cloud is generally available on AWS. It works as a read-only serving layer. It durably stores Apache Kafka data, materializes it into tables, and keeps them current as changes arise. Teams can then run low-latency queries on the current state of the business. Key features include:
- Data freshness and query performance: In Confluent internal testing, data moves from creation to serving in approximately 100 milliseconds. Query latency runs 10-100 milliseconds, with throughput of 1,000+ queries per second.
- Lower cost and complexity: A fully managed serving layer runs continuously, without separate ETL pipelines, databases, or caches.
- Multiple query interfaces: Users query through SQL and a direct API.
- Unified governance: Teams apply schemas, lineage, and access policies from data capture through serving.
- Schema evolution: Teams manage schema changes to help prevent breaks in downstream applications.
- Feature store: Teams compute features in Flink and store them for machine learning use cases, including real-time personalization.
“The next era of enterprise AI will be won by the agents that know what’s happening in the business the moment they act,” said Shaun Clowes, General Manager, Confluent and Data, IBM. “That’s what a Streamhouse can deliver. With Lightning Tables, our Data Streaming Platform gives each agent and application a live, governed view of the business. This means companies can act on fraud signals sooner, serve customers in the moment and run operations on what’s true now versus what was true yesterday.”
Explore IT Tech News for the latest advancements in Information Technology & insightful updates from industry experts!
News Source: Businesswire.com
