Kafka developed at LinkedIn and then open source -tool for building real time data pipe lines. kafka is distributed streaming platform and similar to enterprise messaging system. Let’s see more about Apache Kafka Commands.
ZooKeeper : open source project of Hadoop. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group service.
Kafka components:
connectors : to import databases into kafka and kafka to databases
producers: an application that sends message to kafka
consumers: an application that receive/reads message from kafka
stream processors:
broker: kafka server
Topic : A unique name for kafka stream
Partitions : Part of topic
Offset : Unique id for message within a partition
consumers groups : A group of consumers acting as a single logical unit
Apache Kafka Commands :
To List All Kafka Topics :
Cd ${PATH_TO_KAFKA_BIN}
kafka-topics.sh —list –zookeeper localhost:2181
To Create Topics:
Cd ${PATH_TO_KAFKA_BIN}
./kafka-topics.sh —create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic TopicName
./kafka-topics.sh —create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic error. TopicName
To Alter Topics:
Cd ${PATH_TO_KAFKA_BIN}
./kafka-topics.sh –zookeeper localhost:2181 —alter –topic TopicName –partitions 10
To delete Topics:
Cd ${PATH_TO_KAFKA_BIN}
kafka-topics.sh –zookeeper localhost:2181 —delete –topic test_topic_name
To Describe Topics:
Cd ${PATH_TO_KAFKA_BIN}
./kafka-topics.sh —describe –zookeeper localhost:2181 –topic topic_name
./kafka-topics.sh —describe –zookeeper localhost:2181 –topic TopicName
./kafka-topics.sh —describe –zookeeper localhost:2181 –topic error.TopicName
Check partitions are NOT Under Replicated:
${PATH_TO_KAFKA_BIN}/kafka-topics.sh –zookeeper localhost:2181 —describe –under-replicated-partitions
To Start Kafka:
Start ZooKeeper : kafka server –> bin –>zookeeper-server-start.sh config/zookeeper.properties —> Looks for port 2181
${PATH_TO_KAFKA_BIN}/zkprs_start.sh
start Kafka Broker : bin/kafka-server-start.sh config/server.properties –> Look for “started”
${PATH_TO_KAFKA_BIN}/kafka_start.sh
To Stop Kafka:
Stop kafka broker/server:
${PATH_TO_KAFKA_BIN}/kafka-server-stop.sh
Stop zookeeper/controller:
${PATH_TO_KAFKA_BIN}/zookeeper-server-stop.sh
Important Kafka Topic Properties
Review the following settings in the Advanced kafka-broker category, and modify as needed:
auto.create.topics.enable
Enable automatic creation of topics on the server. If this property is set to true, then attempts to produce, consume, or fetch metadata for a nonexistent topic automatically create the topic with the default replication factor and number of partitions.
The default is enabled. The auto.create.topics.enable property, when set to true, automatically creates topics when applications attempt to produce, consume, or fetch metadata for a nonexistent topic.
default.replication.factor
Specifies default replication factors for automatically created topics. For high availability production systems, you should set this value to at least 3.
num.partitions:
Specifies the default number of log partitions per topic, for automatically created topics. The default value is 1. Change this setting based on the requirements related to your topic and partition design.
delete.topic.enable:
Allows users to delete a topic from Kafka using the admin tool, for Kafka versions 0.9 and later. Deleting a topic through the admin tool will have no effect if this setting is turned off.
Related Posts:
Difference between Apache Kafka and JMS
Difference between Apache Kafka and Flume
Apache Kafka Vs Apache Spark: Know the Differences
Kafka vs Storm: Feature Wise Comparison of Kafka & Storm
Kafka Vs Redis (Difference Between Kafka and Redis)
Top 15 Apache Kafka Alternatives Popular In 2022