Kafka Commands

Starting Zookeeper & Kafka

[npntraining ~]$ cd \opt\kafka\zookeeper-3.5.6-bin\bin
[npntraining ~]$ zkserver.sh

Starting Kafka

[npntraining ~]$ cd \opt\kafka\kafka_2.12-2.4.0\bin\
[npntraining ~]$ kafka-server-start.sh \opt\kafka\kafka_2.12-2.4.0\config\server.properties

Kafka Topic

You can create a new Kafka topic named simple-producer-consumer as follows:

[npntraining ~]$ cd \opt\kafka\kafka_2.12-2.4.0\bin\
[npntraining ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --topic simple-producer-consumer

List all topics

You can verify that the simple-producer-consumer topic was successfully created by listing all available topics:

[npntraining ~]$ kafka-topics.sh --list --zookeeper localhost:2181

Deleting Topic

You can delete a topic named simple-producer-consumer as follows:

[npntraining ~]$ kafka-topics.sh --delete --zookeeper localhost:2181 --topic simple-producer-consumer

[npntraining ~]$ kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic simple-producer-consumer

Describing Topic

You can find more details about a topic named simple-producer-consumer

[npntraining ~]$ kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic <topic-name> 

Creating topic with multiple partitions

You can add more partitions as follows:

[npntraining ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic simple-producer-consumer

Under-replicated partitions

You can see the under-replicated partitions for all topics as follows:

[npntraining ~]$ kafka-topics.sh --zookeeper localhost:2181/kafka-cluster --describe --under-replicated-partitions

Producing and Consuming Messages

Starting Kafka Console Producer

You can produce messages from standard input as follows:

[npntraining ~]$ kafka-console-producer.sh --broker-list localhost:9092 --topic simple-producer-consumer
Message 01

Starting Kafka Console Consumer

[npntraining ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic simple-producer-consumer
Message01

You can begin a consumer from the beginning of the log as follows:

[npntraining ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic simple-producer-consumer --from-beginning

Running JAR from Command Line

java -cp kafka-producer-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.npntraining.SimpleProducer first-topic
java -cp kafka-producer-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.npntraining.SimpleConsumer my-first-topic

Producers

You can begin a consumer from the beginning of the log as follows:

kafka-console-producer --broker-list localhost:9092 --topic test < messages.txt

You can produce Avro messages as follows:

kafka-avro-console-producer --broker-list localhost:9092 --topic my.Topic --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"name","type":"string"}]}' --property schema.registry.url=http://localhost:8081

You can enter a few new values from the console as follows:

{"name": "Naveen"}

Consumers

You can consume a single message as follows:

kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic  --max-messages 1

You can consume 10 Avro messages from a topic named position-reports as follows:

kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081 --max-messages 10

You can consume all existing Avro messages from a topic named position-reports as follows:

kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081