Latest Insights on Data Streaming, AI, and Enterprise Architecture
Explore Kai Waehner’s latest articles on Apache Kafka, Apache Flink, real-time data streaming, data integration, process intelligence, agentic AI, and modern enterprise architecture. The blog covers practical use cases, technology trends, conference insights, and hands-on perspectives for teams building scalable, future-ready data platforms.
dbt Meets Apache Flink: One Workflow for Data Engineers on Snowflake, BigQuery, Databricks, and Confluent
Two toolchains, two skill sets, two CI/CD pipelines — that has been the reality for data engineers working across batch and streaming. dbt extending to Apache Flink changes that equation. One workflow, one tool, one engineering team…
The Shift Left Architecture 2.0: Operational, Analytical and AI Interfaces for Real-Time Data Products
The Shift Left Architecture moves data integration logic into an event-driven architecture where governed data products are built once and served to multiple consumers. The original pattern covered two interfaces: operational via…
UFC VIP Experience Worth the Price? Fan Review. Business Perspective. Tech Vision.
The Ultimate Fighting Championship (UFC) held Fight Night London on March 21, 2026, at The O2 Arena. It was a great event inside the octagon. But a 1,200 USD VIP ticket raised real questions about premium customer experience, and about…
Dashboards and Queries for Apache Kafka: Operational, Explorative, and the Role of the Context Engine
Dashboards are a popular way to make streaming data visible and useful, but they are not always the right solution. This blog post explains when dashboards make sense for Apache Kafka data and when other approaches like automation,…
Data Streaming at MWC 2026: How Apache Kafka, Flink and Agentic AI Power Telecom Trends
Mobile World Congress (MWC) 2026 highlights the shift from batch systems to real time data streaming in telecom. AI and agentic automation, network APIs, sovereign cloud, autonomous networks, and 5G monetization all depend on…
From Takeoff to Touchdown: Real-Time Aviation with Data Streaming at Qantas
This blog post explores how data streaming transforms airline operations by enabling real-time visibility, faster decision-making, and improved customer experience. Using Qantas as a leading example, it highlights how a modern data…
The Ultimate Data Streaming Guide is Back – Second Edition of the Book and Industry Editions Now Available
The second edition of The Ultimate Data Streaming Guide is now available as a free eBook. It includes over 70 use cases, over 20 customer stories, a detailed use case / customer matrix, and a stronger focus on AI topics like GenAI and…
When (Not) to Use Queues for Kafka?
Apache Kafka has long been the foundation for real-time data streaming. With the release of Queues for Kafka (QfK) in Apache Kafka 4.2, it now also supports native queuing, eliminating the need for separate message queue systems for…
Diskless Kafka at FinTech Robinhood for Cost-Efficient Log Analytics and Observability
Diskless Kafka is transforming how fintech and financial services organizations handle observability and log analytics. By using the Kafka protocol with cloud-native object storage, companies like Robinhood reduce infrastructure costs…
Shift Left in Automotive: Real-Time Intelligence from Vehicle Telemetry with Data Streaming at Rivian
Rivian and Volkswagen, through their joint venture RV Tech, process high-frequency telemetry from connected vehicles using Apache Kafka and Apache Flink. With a shift left architecture and smart filtering, they reduced data volume by…
Etihad Airways Makes Airline Operations Real-Time with Data Streaming
Airlines face constant pressure to deliver reliable service while managing complex operations and rising customer expectations. This blog post explores how Etihad Airways uses real-time data streaming with Apache Kafka and Flink to…
Stream Processing on the Mainframe with Apache Flink: Genius or a Glitch in the Matrix?
Running Apache Flink on a mainframe may sound surprising, but it is already happening and for good reason. As modern mainframes like IBM z17 evolve to support Linux, Kubernetes, and AI workloads, they are becoming a powerful platform…
10 FinTech Predictions That Depend on Real Time Data Streaming
Financial services companies are moving from batch processing to real time data flow. A data streaming platform enables financial institutions to connect systems, process events instantly, and power AI, fraud prevention, and customer…
Top Trends for Data Streaming with Apache Kafka and Flink in 2026
Each year brings new momentum to the data streaming space. In 2026, six key trends stand out. Platforms and vendors are consolidating. Diskless Kafka and Apache Iceberg are reshaping storage. Real-time analytics is moving into the…
The Data Streaming Landscape 2026
Data streaming is now a core software category in modern data architecture. It powers real-time use cases like fraud prevention, personalization, supply chain optimization, and AI automation. What started with open source Apache Kafka…
Life as a Lufthansa HON Circle Member: Inside the Ultimate Frequent Flyer Status
Reaching Lufthansa HON Circle status was both a personal milestone and a significant financial investment. Since my employer covers only Economy and Premium Economy, I pay the difference to fly in Business Class and qualify. This post…
CARIAD’s Unified Data Platform: A Data Streaming Automotive Success Story Behind Volkswagen’s Software-Defined Vehicles
The automotive industry transforms rapidly. Cars are now software-defined vehicles (SDVs) that demand constant, real-time data flow. This post highlights the CARIAD success story inside the Volkswagen Group. CARIAD tackled data…
Data Streaming Meets Lakehouse: Apache Iceberg for Unified Real-Time and Batch Analytics
Apache Iceberg is gaining momentum as the open table format of choice for modern data architectures. In this blog post, the key takeaways from my talk at Open Source Data Summit are shared, along with the full video and downloadable…
Data Streaming in Retail: Social Commerce from Influencers to Inventory
Social commerce is reshaping retail by merging entertainment, influencer marketing, and instant purchasing into one real-time experience. Platforms like TikTok and Instagram have become active digital storefronts where discovery and…
Kafka Proxy Demystified: Use Cases, Benefits, and Trade-offs
A Kafka proxy adds centralized security and governance for Apache Kafka. Solutions like Kroxylicious, Conduktor, and Confluent enable encryption, access control, and compliance without modifying clients or brokers. This article…
How Stablecoins Use Blockchain and Data Streaming to Power Digital Money
Stablecoins are reshaping digital money by linking traditional finance with blockchain technology. Built for stability and speed, they enable real time payments, settlements, and programmable financial services. To operate reliably at…
Cybersecurity with a Digital Twin: Why Real-Time Data Streaming Matters
Cyberattacks on critical infrastructure and manufacturing are growing, with ransomware and manipulated sensor data creating severe risks. Digital twins combined with data streaming provide real-time visibility, continuous monitoring,…
How Siemens, SAP, and Confluent Shape the Future of AI Ready Integration – Highlights from the Rojo Event in Amsterdam
Many enterprises want to become AI ready but are limited by slow, batch based integration platforms that prevent real time insight and automation. The Rojo “Future of Integration” event in Amsterdam addressed this challenge by bringing…
Scaling Kafka Consumers: Proxy vs. Client Library for High-Throughput Architectures
Apache Kafka’s pull-based model and decoupled architecture offer unmatched flexibility for event-driven systems. But as data volumes and consumer applications grow, new challenges emerge; from head-of-line blocking and rising…
Square, SumUp, Shopify: Real-Time Point-of-Sale (POS) in the Age of Data Streaming
Point-of-Sale systems are evolving into real-time, connected platforms that go far beyond payments. Mobile solutions from Square, SumUp, and Shopify give even the smallest merchants access to integrated sales channels, inventory…
Online Feature Store for AI and Machine Learning with Apache Kafka and Flink
Real-time personalization requires more than just smart models. It demands fresh data, fast processing, and scalable infrastructure. This blog post explores how Wix.com rebuilt its online feature store using Apache Kafka and Flink,…
How Data Streaming Powers AI and Autonomous Networks in Telecom – Insights from TM Forum Innovate Americas
AI and autonomous networks took center stage at TM Forum Innovate Americas 2025 in Dallas. Leaders from AT&T, Verizon, and other telcos showed how large-scale AI is transforming customer experience, security, and network operations,…
Telecom OSS Modernization with Data Streaming: From Legacy Burden to Cloud-Native Agility
OSS is critical for service delivery in telecom, yet legacy platforms have become rigid and costly. They slow innovation just as 5G, cloud native networks and OTT partnerships demand agility. This article explores how a data streaming…