Kraft: How to Install Kafka without zookeeper?

It is now possible to run Kafka without Apache zookeeper and is called as Kafka Raft metadata mode. It is currently in preview and should not be used in production environment. Its available for testing in Kafka release 2.8 and in this tutorial we are using Release 3.1 Kafka.

When the Kafka cluster is in KRaft mode, it does not store its metadata in zookeeper. In fact, you do not have to run zookeeper at all, because it stores its metadata in a KRaft quorum of controller nodes.

There may be bugs, including serious ones. You should assume that your data could be lost at any time if you try the preview release of KRaft mode.

Here are the steps to install and run Kafka without zookeeper :

1- Install WSL 2 (Windows subsystem for Linux)
2- Install Java JDK version 11
3- Download Apache Kafka from https://kafka.apache.org/downloads
4- Generate a cluster ID and format the storage using kafka-storage.sh
5- Start Kafka using the binaries in another process in WSL 2

1- Install WSL 2

WSL 2 is Windows Subsystem for Linux 2 and provides a Linux environment for your Windows computer that does not require a virtual machine.

To install WSL 2, make sure you’re on Windows 10 version 2004 and higher or window 11, you can check your windows version by running the command winver in Command Prompt

You can now install everything you need to run Windows Subsystem for Linux (WSL) by entering this command in an administrator PowerShell or Windows Command Prompt and then restarting your machine.

wsl --install -d Ubuntu

WSL command Installed Ubuntu

After running this command , it will take some time to download and install WSL on your system. After Installing it will ask for setting up the username and password, you can write any username/password, this is not necessary to match it up with your current windows password.

2- Install Java JDK 11 on WSL

• wget -O- https://apt.corretto.aws/corretto.key | sudo apt-key add —
• sudo add-apt-repository ‘deb https://apt.corretto.aws stable main’
• sudo apt-get update; sudo apt-get install -y java-11-amazon-corretto-jd

After running these commands you can run the java -version to check the installed version.

3- Download Apache Kafka

wget https://dlcdn.apache.org/kafka/3.1.0/kafka-3.1.0-src.tgz
tar -xzf kafka-3.1.0-src.tgz

Kafka download will be expanded to kafka-3.1.0-src directory .

4- Generate a cluster ID

You need to generate an ID for your new cluster, using the Kafka storage tool:

 cd kafka-3.1.0-src/bin
./kafka-storage.sh random-uuid

This returns a UUID for example , eOwRQ-yMThaVuEslgP4CAA

You might encounter the following error while running above command to get the cluster ID.

Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVersion=2.13.6'

To fix this error we need to run the following commands —

cd kafka-3.1.0-src
./gradlew jar -PscalaVersion=2.13.6

Next format your storage directory by the following command-

./kafka-storage.sh format -t <<REPLACE_WITH_UUID>> -c ../config/kraft/server.properties

5- Start Kafka

cd kafka-3.1.0-src/bin
./kafka-server-start.sh ../config/kraft/server.properties

Don’t close this window as it will shutdown the broker. ( Default port is 9092)

Congratulation, the Kafka broker is now running without zookeeper in Kraft mode!