Kafka in Banking: A Bridge Between Worlds – for Long-term Economical Projects
Core banking systems handle the most important processes in banking. The problem: These inflexible giants rarely harmonize with the wishes of today's customers. Systems are needed that connect the old with the new. Many banks rely on Apache Kafka for this. Why?
By Anatoly Zelenin
Some time ago, a report by Süddeutsche Zeitung made headlines. It was about a group of NASA retirees who, in 2014, 37 years after the launch of the world-famous Voyager missions, were still maintaining the space probes. The reason? These old veterans are among the few who still fully understand the engineering of that era.
The case in Houston sounds romantic, but it highlights a problem: The expertise of "all-knowing" individuals becomes a threatening scenario when these experts retire from professional life.
In theory, there could also be one or more groups of seasoned veterans in the banking system. However, this is anything but amusing.
Many banks still use various mainframes to handle their core business. The software for these massive computers was written decades ago. Many of the development teams from that time have long since retired or moved to other companies.
Modifying these systems is considered extremely difficult because there are hardly any people left who are really familiar with these archaic structures and programming languages like COBOL. A lot of knowledge went into the development that has since been forgotten outside of the program code.
Simply throwing away these systems is also not an option: too many current and vital processes depend on them.
Solution Approach: "Liberating" Data from Mainframes through Apache Kafka
Figure 1. With Kafka, we can cost-effectively make data from mainframes or core banking systems available to other services
An approach that many banks want to use to solve this dilemma is "liberating" the data from the mainframes. These mainframes must continue to exist. The only difference is that authorized individuals and services can access the information at any time, enabling them to meet the future requirements of banking. Apache Kafka is one of the most effective solutions for this connection between the past and the modern.
For example, some of my clients from the financial sector are introducing new, user-friendly frontends for their banking customers and employees. I’m talking about apps or improved websites. Banks often proceed as follows:
-
First, they need to transfer the data from the mainframe to Kafka. This is not trivial! Extracting data from core banking systems is fraught with great difficulties. The old systems were not designed to output the diverse information in the quantities needed today when they were initially integrated. Banks are forced to make a decision. They can develop individual solutions or use existing connectors as interfaces. Both solutions require considerable development effort. The good news: Modern systems have a direct connection to Kafka.
-
Then the data must be enriched and transformed. This is best done in real-time. Only then can the data generated in the mainframe reach other systems as quickly as possible. Stream processing frameworks like Kafka Streams or Apache Flink are ideal for this.
-
For many of these transformations, we can even avoid complex code and instead rely on SQL, for example, based on Apache Flink.
-
But now it gets really exciting and expensive: A normal-sized bank has millions of customer records. Even if each consists of only 1000 account transactions, a low estimate, that’s many billions of data points. Loading this data from the mainframe costs a lot of money. Especially since it should be done quickly.
But there’s good news: Apache Kafka doesn’t forget. Banks can’t avoid this expensive process. But with Kafka, they ensure it remains a one-time effort.
Even if months or years later, a new application is added and original mainframe data is needed again, the information is already available. The new application can get this directly from Kafka. After a (preferably agile) approval process, the data is immediately ready. Customers can access information within days instead of months. This is what I mean when I talk about "liberating" data.
Summary: What Kafka Brings to Banks
With Kafka, banks "liberate" data from mainframes and core banking systems. They can distribute information within the company more flexibly and according to needs. Core banking systems no longer need to be directly accessed by every new app. This significantly reduces costs and accelerates product development. Teams can access the required data in real-time and much more easily than before.
Apache Kafka is thus not only an instrument with which we can efficiently retrieve data from old systems at any time. Kafka also ensures that the information remains on the surface for the long term with quick access. Even when a project is supposedly completed.
This is how smart companies reduce the costs for numerous further and new developments with a one-time investment.
This blog will soon be expanded with a case interview from the industry. Don’t want to miss it? Follow me on LinkedIn!
Anatoly Zelenin teaches Apache Kafka to hundreds of participants in interactive training sessions. His clients from the DAX environment and German mid-sized companies have valued his expertise and inspiring approach for over a decade. In addition to being an IT consultant and trainer, he also explores our planet as an adventurer.
We believe in a world where companies make better decisions with real-time data, act sustainably, and actively shape their digital future.
Copyright 2025, All Right Reserved
Contact
Address
Commit to Flow GmbH
Wiener Platz 11
01069 Dresden