Useful Apache Kafka Commands

Useful Apache Kafka Commands

Kafka developed at LinkedIn and then open source -tool for building real time data pipe lines. kafka is distributed streaming platform and similar to enterprise messaging system. Let’s see more about Apache Kafka Commands.

ZooKeeper : open source project of Hadoop. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group service.

 

Kafka components:

connectors : to import databases into kafka and kafka to databases

producers: an application that sends message to kafka

consumers: an application that receive/reads  message from kafka

stream processors:

broker:  kafka server

Topic :  A unique name for kafka stream

Partitions : Part of topic

Offset : Unique id for message within a partition

consumers groups : A group of consumers acting as a single logical unit

 

Apache Kafka Commands :

To List All Kafka Topics :

Cd ${PATH_TO_KAFKA_BIN}

kafka-topics.sh —list –zookeeper localhost:2181

To Create Topics:

Cd ${PATH_TO_KAFKA_BIN}

./kafka-topics.sh —create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic TopicName

./kafka-topics.sh —create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic error. TopicName

To Alter Topics:

Cd ${PATH_TO_KAFKA_BIN}

./kafka-topics.sh –zookeeper localhost:2181 —alter –topic TopicName –partitions 10

To delete Topics:

Cd ${PATH_TO_KAFKA_BIN}

kafka-topics.sh –zookeeper localhost:2181 —delete –topic test_topic_name

To Describe Topics:

Cd ${PATH_TO_KAFKA_BIN}

./kafka-topics.sh —describe –zookeeper localhost:2181 –topic topic_name

./kafka-topics.sh —describe –zookeeper localhost:2181 –topic TopicName

./kafka-topics.sh —describe –zookeeper localhost:2181 –topic error.TopicName

Check partitions are NOT Under Replicated:

${PATH_TO_KAFKA_BIN}/kafka-topics.sh –zookeeper localhost:2181 —describe –under-replicated-partitions

To Start Kafka:

Start ZooKeeper : kafka server –> bin –>zookeeper-server-start.sh config/zookeeper.properties —> Looks for port 2181

${PATH_TO_KAFKA_BIN}/zkprs_start.sh

start Kafka Broker : bin/kafka-server-start.sh config/server.properties  –> Look for “started”

${PATH_TO_KAFKA_BIN}/kafka_start.sh

To Stop Kafka:

Stop kafka broker/server:

${PATH_TO_KAFKA_BIN}/kafka-server-stop.sh

Stop zookeeper/controller:

${PATH_TO_KAFKA_BIN}/zookeeper-server-stop.sh

 

Important Kafka Topic Properties

 

Review the following settings in the Advanced kafka-broker category, and modify as needed:

auto.create.topics.enable

Enable automatic creation of topics on the server. If this property is set to true, then attempts to produce, consume, or fetch metadata for a nonexistent topic automatically create the topic with the default replication factor and number of partitions.

The default is enabled. The auto.create.topics.enable property, when set to true, automatically creates topics when applications attempt to produce, consume, or fetch metadata for a nonexistent topic.

 

default.replication.factor

Specifies default replication factors for automatically created topics. For high availability production systems, you should set this value to at least 3.

 

num.partitions:

Specifies the default number of log partitions per topic, for automatically created topics. The default value is 1. Change this setting based on the requirements related to your topic and partition design.

 

delete.topic.enable:

Allows users to delete a topic from Kafka using the admin tool, for Kafka versions 0.9 and later. Deleting a topic through the admin tool will have no effect if this setting is turned off.

Related Posts:

Apache Kafka Tutorial

Difference between Apache Kafka and JMS

Difference between Apache Kafka and Flume

Apache Kafka Vs Apache Spark: Know the Differences

Kafka vs Storm: Feature Wise Comparison of Kafka & Storm

Kafka Vs Redis (Difference Between Kafka and Redis)

Apache Kafka Tutorial

Top 15 Apache Kafka Alternatives Popular In 2022

Apache Kafka Tutorial