Setting up a Kafka Test Environment

Learn how to set up a multi-node Kafka test environment on a single machine for development and testing purposes. This guide shows you how to configure three Kafka brokers with proper controller setup.

By Arockia Nirmal Amala Doss

This blog post provides a practical guide for setting up a Kafka test environment. If you want to learn more about Kafka, please read the following articles:

In this blog post, we’ll provide a brief overview of how to set up a Kafka test environment on a single machine, such as a personal laptop. This configuration is intended strictly for testing and development purposes and should never be used in production. Unlike many tutorials that focus on single-node installations, we will demonstrate how to configure a multi-node Kafka environment on a single machine. This approach allows you to observe how Kafka handles partial failures and better understand its distributed nature.

Test Environment Overview

Kafka Cluster
Figure 1. Our test environment consists of three Kafka brokers in dual-mode.

In the figure above, we display the test environment, which consists of three Kafka brokers. Broker 1 listens on port 9092, Broker 2 on 9093, and Broker 3 on 9094. To simplify the setup, we will run the brokers in dual-mode, meaning that the three brokers will act both as Controllers and normal Brokers. The controllers require additional listeners, so the controller of Broker 1 will listen on port 9192, Broker 2’s controller on 9193, and Broker 3’s controller on 9194.

Operating System

Apache Kafka runs on the Java Virtual Machine (JVM) and thus supports all major operating systems. We have tested this environment on Ubuntu 24.04 LTS and MacOS. On Windows systems we recommend using the Windows Subsystem for Linux (WSL).

Downloading Kafka

You can always find the most up-to-date version of Kafka at https://kafka.apache.org/downloads. At the time of writing, the latest version is 3.9.0. As some parts of Kafka are still written in Scala, you will find downloads of the form kafka_2.13-3.9.0.tgz where 2.13 is the Scala version and 3.9.0 is the Kafka version. We recommend always using the latest Scala version. Let us download Kafka and extract it to the directory ~/kafka:

wget "https://downloads.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz"
tar xfz kafka_2.13-3.9.0.tgz
rm kafka_2.13-3.9.0.tgz
mv kafka_2.13-3.9.0/ ~/kafka

You will find the following subdirectories in the ~/kafka directory:

  • LICENSE, NOTICE: Files containing license information

  • bin: Command-line tools to interact with Kafka

  • config: Configuration files

  • libs: Java library files (*.jar)

  • site-docs: Documentation

Prerequisites

First, make sure that Java is installed on your machine:

java -version

Configuring Kafka

Let us configure Broker 1 first:

~/kafka/config/kafka1.properties
# The ID of the broker (here 1)
broker.id=1
# The directory where the broker should store the files. Please adapt the directory.
log.dirs=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/data/kafka1
# Use port 9092 for normal broker communication and port 9192 for the controller.
# You can call the listeners as you want
listeners=PLAINTEXT://:9092,CONTROLLER://:9192
# Running the broker in dual-mode
process.roles=broker,controller
# Providing the connection string for all our controllers
controller.quorum.voters=1@localhost:9192,2@localhost:9193,3@localhost:9194
# Which listener should be used for the controller?
controller.listener.names=CONTROLLER
# Do not use TLS for our test environment
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT

Let us also configure Broker 2:

~/kafka/config/kafka2.properties
broker.id=2
log.dirs=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/data/kafka2
listeners=PLAINTEXT://:9093,CONTROLLER://:9193
process.roles=broker,controller
controller.quorum.voters=1@localhost:9192,2@localhost:9193,3@localhost:9194
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT

Finally, configure Broker 3:

~/kafka/config/kafka3.properties
broker.id=3
log.dirs=<PATH-TO-YOUR-USER-DIRECTORY>/kafka/data/kafka3
listeners=PLAINTEXT://:9094,CONTROLLER://:9194
process.roles=broker,controller
controller.quorum.voters=1@localhost:9192,2@localhost:9193,3@localhost:9194
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT

Preparing the Data Directories

First, we will create the directories where the brokers should store the data. Afterwards, we need to initialize (format) the directories so that we can use them in our cluster. For this, we need to create a cluster ID. This way it is impossible to use a directory with another cluster, which limits the mistakes you can make.

mkdir -p ~/kafka/data/kafka1 ~/kafka/data/kafka2 ~/kafka/data/kafka3
export KAFKA_CLUSTER_ID="$(kafka-storage.sh random-uuid)"
kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c ~/kafka/config/kafka1.properties
kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c ~/kafka/config/kafka2.properties
kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c ~/kafka/config/kafka3.properties

Starting Kafka

Now it’s time to start our brokers. First, start the brokers in the foreground so we can see any possible errors, and if everything works, start the brokers in the background as a Daemon. Let’s start the first broker:

~/kafka/bin/kafka-server-start.sh ~/kafka/config/kafka1.properties

If this command crashes immediately, the broker is misconfigured. Please check the configuration and try again.

If you did everything correctly, the Kafka broker should complain about Node 2 and 3 being unreachable:

WARN [RaftManager nodeId=1] Connection to node 2 (localhost/127.0.0.1:9193) could not be established.
Broker may not be available. (org.apache.kafka.clients.NetworkClient)

This makes sense because we have not started the other brokers yet. Open a new terminal window and start the second broker:

~/kafka/bin/kafka-server-start.sh ~/kafka/config/kafka2.properties

You should see a lot of log messages. The brokers will still complain about the missing Node 3, but among the log messages, you should find the following line:

[KafkaRaftServer nodeId=2] Kafka Server started

Finally, start the remaining broker:

~/kafka/bin/kafka-server-start.sh ~/kafka/config/kafka3.properties

After that, you should see the following line in the logs of broker 3, and the brokers should have stopped complaining about not being able to connect to the other nodes:

[KafkaRaftServer nodeId=3] Kafka Server started

Of course, you can leave it this way, but you can also stop the brokers one after another and replace them with brokers in daemon mode. This way you will not be able to see the logs, but you also won’t need to have three terminal windows open all the time:

Stop Broker 1 with CTRL+C and execute the command below to start broker 1 in the background:

~/kafka/bin/kafka-server-start.sh -daemon ~/kafka/config/kafka1.properties

Stop Broker 2 with CTRL+C and execute the command below to start broker 2 in the background:

~/kafka/bin/kafka-server-start.sh -daemon ~/kafka/config/kafka2.properties

Stop Broker 3 with CTRL+C and execute the command below to start broker 3 in the background:

~/kafka/bin/kafka-server-start.sh -daemon ~/kafka/config/kafka3.properties

To check whether everything is still running, you can use the following command:

~/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

On the one hand, it outputs the API versions each broker supports, but more importantly, it shows which brokers are online:

localhost:9092 (id: 1 rack: null) -> (
# A lot of text
)
localhost:9093 (id: 2 rack: null) -> (
# A lot of text
)
localhost:9094 (id: 3 rack: null) -> (
# A lot of text
)

Stopping Kafka

To stop all Kafka brokers, simply execute the following command:

~/kafka/bin/kafka-server-stop.sh

But this command stops all Kafka brokers. To stop a single broker, you can create a new script.

~/kafka/bin/kafka-broker-stop.sh
#!/bin/bash
BROKER_ID="$1"
if [ -z "$BROKER_ID" ]; then
    echo "usage ./kafka-broker-stop.sh [BROKER-ID]"
    exit 1
fi
PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep "kafka${BROKER_ID}.properties" | grep -v grep | awk '{print $1}')
if [ -z "$PIDS" ]; then
    echo "No kafka server to stop"
    exit 1
else
    kill -s TERM $PIDS
fi

Run the command below to make the script executable:

chmod +x ~/kafka/bin/kafka-broker-stop.sh

Instead of stopping all brokers, it will stop only the one for which the broker ID matches. To stop broker 1, for example, you can use the following command:

~/kafka/bin/kafka-broker-stop.sh 1

You can check with the kafka-broker-api-versions.sh script whether this worked:

~/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
WARN [LegacyAdminClient clientId=admin-1]
Connection to node -3 (localhost/127.0.0.1:9094) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker localhost:9094 (id: -3 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
WARN [LegacyAdminClient clientId=admin-1]
Connection to node -2 (localhost/127.0.0.1:9093) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker localhost:9093 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Kafka is now stopped.

We wish you happy streaming, and have fun exploring Kafka.

Arockia Nirmal Amala Doss is an experienced Data Engineer with over 10 years of expertise in data architectures. As a certified Apache Kafka Developer and AWS Data Engineer, he specializes in scalable event streaming pipelines and cloud-native data solutions. His clients from the DAX environment value his practical experience with Amazon MSK, ETL/ELT processes, and developing robust data infrastructures.

logo

We believe in a world where companies make better decisions with real-time data, act sustainably, and actively shape their digital future.

Copyright 2025, All Right Reserved

Contact

Address

Commit to Flow GmbH
Wiener Platz 11
01069 Dresden

E-Mail Address

kontakt@dataflow.academy

Links