How to start Apache Kafka

Apache Kafka is an open-source for distributed streaming system. We can use it as a Messaging System, a Storage System or Stream Processing. So in the tutorial, JavaSampleApproach will show you the first step to quick start with Apache Kafka.

Related Articles:
– How to start Spring Kafka Application with Spring Boot
– How to start Spring Apache Kafka Application with SpringBoot Auto-Configuration

1. Download Apache Kafka

Go to the download page, download file Scala 2.12 – kafka_2.12-0.10.2.1.tgz (asc, md5)

Apache kafka Quick start- download source

We will get a .tar file, then un-tar it, we have folder kafka_2.12-0.10.2.1:

Apache kafka folder

cd to .\kafka_2.12-0.10.2.1\bin:

Apache kafka Quick start- unix bin

All files .sh under folder .\kafka_2.12-0.10.2.1\bin is used to setup Apache Kafka in Unix-based environment.

cd to .\kafka_2.12-0.10.2.1\bin\windows:

Apache kafka Quick start- windows

All files .bat under folder .\kafka_2.12-0.10.2.1\bin\windows is used to setup Apache Kafka in Windows platforms.

2. Start a Kafka server

Apache Kafka uses ZooKeeper as a centralized service, so we need to start a ZooKeeper.

-> Open a cmd, cd to .\kafka_2.12-0.10.2.1:

– Unix-based

> bin/zookeeper-server-start.sh config/zookeeper.properties

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

-> Logs:
[2017-06-06 17:59:52,636] INFO Reading configuration from: .\config\zookeeper.properties
...

Start the Apache Kafka server.

-> Open a new cmd, cd to .\kafka_2.12-0.10.2.1:

– Unix-based

> bin/kafka-server-start.sh config/server.properties

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-server-start.bat .\config\server.properties

-> Logs:
[2017-06-06 18:06:53,365] INFO starting (kafka.server.KafkaServer)
[2017-06-06 18:06:53,367] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2017-06-06 18:06:53,379] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
...

3. Create a Kafka Topic

Setup a topic with name jsa-test, that has only one partition & one replica.

-> Open a new cmd and cd to .\kafka_2.12-0.10.2.1:

– Unix-based

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic jsa-test

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic jsa-test

-> Logs:
Created topic "jsa-test".

List out the topics:

– Unix-based

> bin/kafka-topics.sh --list --zookeeper localhost:2181
jsa-test

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
jsa-test

4. Use Kafka Producer to send messages

Use Kafka command line client to send messages to Kafka topic.

– Unix-based

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic jsa-test
Hello World!
This is a message to jsa-test topic!

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic jsa-test
Hello World!
This is a message to jsa-test topic!

5. Start a Kafka consumer

Use Kafka command line consumer to print out messages on standard output.

– Unix-based

> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic jsa-test --from-beginning

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic jsa-test --from-beginning

-> Results:

Hello World!
This is a message to jsa-test topic!

6. Create a multi-broker cluster

Up to now, we had setup the Kafka cluster with single node. Now we go to next step: setup 2 new nodes for the Kafka cluster.

6.1 Create a config file for new brokers

– Unix-based

> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties

Edit new files as below:

config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dirs=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dirs=/tmp/kafka-logs-2

– Windows

C:\kafka_2.12-0.10.2.1>copy .\config\server.properties .\config\server-1.properties
        1 file(s) copied.

C:\kafka_2.12-0.10.2.1>copy .\config\server.properties .\config\server-2.properties
        1 file(s) copied.

Edit new files {server-1.properties, server-2.properties} as below:

config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dirs=kafka-logs-1

config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dirs=kafka-logs-2
6.2 Start new nodes

– Unix-based

> bin/kafka-server-start.sh config/server-1.properties &
...
> bin/kafka-server-start.sh config/server-2.properties &
...

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-server-start.bat .\config\server-1.properties &
...

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-server-start.bat .\config\server-2.properties &
...
6.3 Create a new topic for replication

– Unix-based

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic jsa-replicated-topic

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic jsa-replicated-topic
Created topic "jsa-replicated-topic".

Check the replicated-topic by describe topics command:

– Unix-based

> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic jsa-replicated-topic

– Windows

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-topics.bat --describe --zookeeper localhost:2181 --topic jsa-replicated-topic

-> Results:

Topic:jsa-replicated-topic      PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: jsa-replicated-topic     Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1

The first line gives a summary of all the partitions. Each additional line gives information about one partition.

– leader is the node responsible for all reads and writes for the given partition.
– replicas is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
– isr is the set of “in-sync” replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.

6.4 Create a Producer and Consumer to replicated-topic

– Unix-based

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic jsa-replicated-topic
Message 1
Message 2

> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic jsa-replicated-topic
Message 1
Message 2

– Window

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic jsa-replicated-topic
Message 1
Message 2

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --from-beginning --topic jsa-replicated-topic
Message 1
Message 2
6.5 Fault-Tolerance

See again the topic description:

Topic:jsa-replicated-topic      PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: jsa-replicated-topic     Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1

Broker 2 is the leader. -> We will kill it by commandline:

– Unix-based

> ps aux | grep server-1.properties
435 ttys002    0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java...
> kill -9 435

– Windows

> wmic process get processid,caption,commandline | find "java.exe" | find "server-2.properties"
java.exe   ... config\server-1.properties    422
> taskkill /pid 422 /f

Or We can use Ctr + C on cmd of Broker 2 to kill it.

See shutdown logs:

...

[2017-06-06 19:16:40,807] INFO Truncating log jsa-replicated-topic-0 to offset 2. (kafka.log.Log)
[2017-06-06 19:16:40,836] INFO [Kafka Server 2], Controlled shutdown succeeded (kafka.server.KafkaServer)
[2017-06-06 19:16:40,838] INFO [Socket Server on Broker 2], Shutting down (kafka.network.SocketServer)
[2017-06-06 19:16:40,844] INFO [Socket Server on Broker 2], Shutdown completed (kafka.network.SocketServer)
[2017-06-06 19:16:40,847] INFO [Kafka Request Handler on Broker 2], shutting down (kafka.server.KafkaRequestHandlerPool)
[2017-06-06 19:16:40,873] INFO [Kafka Request Handler on Broker 2], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2017-06-06 19:16:40,877] INFO [ThrottledRequestReaper-Fetch], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-06-06 19:16:41,864] INFO [ThrottledRequestReaper-Fetch], Stopped  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-06-06 19:16:41,864] INFO [ThrottledRequestReaper-Fetch], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-06-06 19:16:41,866] INFO [ThrottledRequestReaper-Produce], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-06-06 19:16:41,896] INFO [ThrottledRequestReaper-Produce], Stopped  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-06-06 19:16:41,896] INFO [ThrottledRequestReaper-Produce], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-06-06 19:16:41,898] INFO [KafkaApi-2] Shutdown complete. (kafka.server.KafkaApis)

...

Again check description of replicated-topic:

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-topics.bat --describe --zookeeper localhost:2181 --topic jsa-replicated-topic
Topic:jsa-replicated-topic      PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: jsa-replicated-topic     Partition: 0    Leader: 0       Replicas: 2,0,1 Isr: 0,1

-> Now the Leader is Broker 0. Broker 2 had been shutdown so in-sync list, so We just have 2 brokers {0, 1}.

Make a consumer for checking the available messages:

C:\kafka_2.12-0.10.2.1>.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --from-beginning --topic jsa-replicated-topic
Message 1
Message 2

-> Kafka Fault-Tolerance works right! Now you can start development with Apache Kafka!

0 0 votes
Article Rating
Subscribe
Notify of
guest
1.9K Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments