How to set an Apache Kafka multi node – multi broker cluster

Adrian Citu

Set a multi node Apache ZooKeeper cluster

  • On every node of the cluster add the following lines to the file kafka/config/zookeeper.properties
server.1=zNode01:2888:3888
server.2=zNode02:2888:3888
server.3=zNode03:2888:3888
#add here more servers if you want
initLimit=5
syncLimit=2

For more informations about the meaning of the parameters please read Running Replicated ZooKeeper.

  • On every node of the cluster create a file called myid in the folder represented by the dataDir property (by default the folder is /tmp/zookeeper ). The myid file should only contains the id of the znode (‘1’ for zNode01, ‘2’ for ZNode02, etc… )

Set a multi broker Apache Kafka cluster

  • On every node of the cluster modify modify the property zookeeper.connect from the file  kafka/config/server.properties:
zookeeper.connect=zNode01:2181,zNode02:2181,zNode03:2181
  • On every node of the cluster modify the property host.name from the file kafka/config/server.properties:
host.name=zNode0x
  • On every node of the cluster modify the property broker.id from the file kafka/config/server.properties (every broker in the cluster should have a unique id)