Kafka Common configuration

Auto start

For Ambari managed Kafka, configure autostart from Console

http://[ambari-host]:8080/#/main/admin/serviceAutoStart

By default, the Kafka and zookeeper “autostart” function provided by Ambari is stopped. For any env other than DEV this is be modified to enabled.

Change Kafka configuration

To change kakfa configuration,

For Ambari managed Kafka Cluster,
- Modify from Ambari Console
For manually configured Kafka Cluster
modify the kafka.properties file.

Configurations

Allow Topic auto created

auto.create.topics.enable

Default is true. Should disable in PROD environment

Allow Topic Deletion

delete.topic.enable

Default is false. Should disable in DEV environment.

Useful Commands

Assuming kafka is installed to default path by Hortonworks Ambari Pack.

Create a topic

/usr/hdf/current/kafka-broker/bin/kafka-topics.sh --create \
--zookeeper zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181 \
--replication-factor 3 \
--partitions 1 \
--topic my-new-topic

List all existing Topics

1 2	/usr/hdf/current/kafka-broker/bin/kafka-topics.sh --list \ --zookeeper zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181

Delete a topic

1 2	/usr/hdf/current/kafka-broker/bin/kafka-topics.sh --delete \ --zookeeper zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181 \ --topic my-new-topic

Describe a topic

1
2
3

/usr/hdf/current/kafka-broker/bin/kafka-topics.sh \
--zookeeper zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181 \
--describe --topic my-new-topic

Publish message to a topic

1
2
3

kafka-console-producer.sh \
--broker-list kafka-broker1:6667,kafka-broker2:6667,kafka-broker3:6667 \
--sync --topic my-new-topic

Consume a topic

1
2
3

/usr/hdf/current/kafka-broker/bin/kafka-console-consumer.sh \
--bootstrap-server kafka-broker1:6667,kafka-broker2:6667,kafka-broker3:6667 \
--topic my-new-topic --from-beginning

Change topic message retention time

/usr/hdf/current/kafka-broker/bin/kafka-configs.sh \
--zookeeper zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181 \
--alter --entity-type topics \
--entity-name my-new-topic \
--add-config retention.ms=86400000

Kafka Monitring Via Ambari

Definition of metrics

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-user-guide/content/grafana_kfka_hosts.html

Bytes In