Explore Kai Waehner’s latest articles on Apache Kafka, Apache Flink, real-time data streaming, data integration, process intelligence, agentic AI, and modern enterprise architecture. The blog covers practical use cases, technology trends, conference insights, and hands-on perspectives for teams building scalable, future-ready data platforms.
Two toolchains, two skill sets, two CI/CD pipelines — that has been the reality for data engineers working across batch and streaming. dbt extending to Apache Flink changes that equation. One workflow, one tool, one engineering team…
The Shift Left Architecture moves data integration logic into an event-driven architecture where governed data products are built once and served to multiple consumers. The original pattern covered two interfaces: operational via…
The Ultimate Fighting Championship (UFC) held Fight Night London on March 21, 2026, at The O2 Arena. It was a great event inside the octagon. But a 1,200 USD VIP ticket raised real questions about premium customer experience, and about…
Dashboards are a popular way to make streaming data visible and useful, but they are not always the right solution. This blog post explains when dashboards make sense for Apache Kafka data and when other approaches like automation,…
Mobile World Congress (MWC) 2026 highlights the shift from batch systems to real time data streaming in telecom. AI and agentic automation, network APIs, sovereign cloud, autonomous networks, and 5G monetization all depend on…
This blog post explores how data streaming transforms airline operations by enabling real-time visibility, faster decision-making, and improved customer experience. Using Qantas as a leading example, it highlights how a modern data…
The second edition of The Ultimate Data Streaming Guide is now available as a free eBook. It includes over 70 use cases, over 20 customer stories, a detailed use case / customer matrix, and a stronger focus on AI topics like GenAI and…
Apache Kafka has long been the foundation for real-time data streaming. With the release of Queues for Kafka (QfK) in Apache Kafka 4.2, it now also supports native queuing, eliminating the need for separate message queue systems for…
Diskless Kafka is transforming how fintech and financial services organizations handle observability and log analytics. By using the Kafka protocol with cloud-native object storage, companies like Robinhood reduce infrastructure costs…
Rivian and Volkswagen, through their joint venture RV Tech, process high-frequency telemetry from connected vehicles using Apache Kafka and Apache Flink. With a shift left architecture and smart filtering, they reduced data volume by…
Airlines face constant pressure to deliver reliable service while managing complex operations and rising customer expectations. This blog post explores how Etihad Airways uses real-time data streaming with Apache Kafka and Flink to…
Running Apache Flink on a mainframe may sound surprising, but it is already happening and for good reason. As modern mainframes like IBM z17 evolve to support Linux, Kubernetes, and AI workloads, they are becoming a powerful platform…
Financial services companies are moving from batch processing to real time data flow. A data streaming platform enables financial institutions to connect systems, process events instantly, and power AI, fraud prevention, and customer…
Each year brings new momentum to the data streaming space. In 2026, six key trends stand out. Platforms and vendors are consolidating. Diskless Kafka and Apache Iceberg are reshaping storage. Real-time analytics is moving into the…
Data streaming is now a core software category in modern data architecture. It powers real-time use cases like fraud prevention, personalization, supply chain optimization, and AI automation. What started with open source Apache Kafka…
Reaching Lufthansa HON Circle status was both a personal milestone and a significant financial investment. Since my employer covers only Economy and Premium Economy, I pay the difference to fly in Business Class and qualify. This post…
The automotive industry transforms rapidly. Cars are now software-defined vehicles (SDVs) that demand constant, real-time data flow. This post highlights the CARIAD success story inside the Volkswagen Group. CARIAD tackled data…
Apache Iceberg is gaining momentum as the open table format of choice for modern data architectures. In this blog post, the key takeaways from my talk at Open Source Data Summit are shared, along with the full video and downloadable…
Social commerce is reshaping retail by merging entertainment, influencer marketing, and instant purchasing into one real-time experience. Platforms like TikTok and Instagram have become active digital storefronts where discovery and…
A Kafka proxy adds centralized security and governance for Apache Kafka. Solutions like Kroxylicious, Conduktor, and Confluent enable encryption, access control, and compliance without modifying clients or brokers. This article…
Stablecoins are reshaping digital money by linking traditional finance with blockchain technology. Built for stability and speed, they enable real time payments, settlements, and programmable financial services. To operate reliably at…
Cyberattacks on critical infrastructure and manufacturing are growing, with ransomware and manipulated sensor data creating severe risks. Digital twins combined with data streaming provide real-time visibility, continuous monitoring,…
Many enterprises want to become AI ready but are limited by slow, batch based integration platforms that prevent real time insight and automation. The Rojo “Future of Integration” event in Amsterdam addressed this challenge by bringing…
Apache Kafka’s pull-based model and decoupled architecture offer unmatched flexibility for event-driven systems. But as data volumes and consumer applications grow, new challenges emerge; from head-of-line blocking and rising…
Point-of-Sale systems are evolving into real-time, connected platforms that go far beyond payments. Mobile solutions from Square, SumUp, and Shopify give even the smallest merchants access to integrated sales channels, inventory…
Real-time personalization requires more than just smart models. It demands fresh data, fast processing, and scalable infrastructure. This blog post explores how Wix.com rebuilt its online feature store using Apache Kafka and Flink,…
AI and autonomous networks took center stage at TM Forum Innovate Americas 2025 in Dallas. Leaders from AT&T, Verizon, and other telcos showed how large-scale AI is transforming customer experience, security, and network operations,…
OSS is critical for service delivery in telecom, yet legacy platforms have become rigid and costly. They slow innovation just as 5G, cloud native networks and OTT partnerships demand agility. This article explores how a data streaming…