For the installation of kafka, Zookeeper must be installed. On this system, hduser is the hadoop user. Let us follow the following below steps to install kafka.

Step2: Download kafka:

Download the required stable release of kafka from the official website. Here, you will get several mirror sites. Just go through any of the mirror site and save the file as shown in the image given below:

Step 3: Extract and Move

Here, you are going to log in through hduser (hduser is hadoop user on this system), because you want to install kafka under hduser.

    $ su - hduser
    Password:

Now you need to explore your root user’s Downloads directory. You need to put your root username at the place of “user” in the command given below:

    $ cd /home/user/Downloads

You can take help from the image given below. In this image, gopal is the root user on this system.

Extract the downloaded file using the command given below:

    $ sudo tar -zxvf kafka_2.10-0.10.1.1.tgz 

Move the extracted file into installation directory using the command given below:

    $ sudo mv kafka_2.10-0.10.1.1 /usr/local/kafka

Step 4: Start Server

Start Zookeeper Server:

  $ cd /usr/local/kafka
  $ bin/zookeeper-server-start.sh config/zookeeper.properties

You will get the output like the image given below:

Another way to start zookeeper server:

    $ cd /usr/local/zookeeper/bin
    $ sudo ./zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED

Connect to localhost:2181

    $ ./zkCli.sh -server 127.0.0.1:2181
    ........................
    ........................
    [zk: localhost:2181(CONNECTED) 0] 

You will get the starting server which looks as the image below:

    $ su - hduser
    Password:
    $ cd /usr/local/kafka
    $ bin/kafka-server-start.sh config/server.properties

Step 5: Create a topic

Open another terminal (Say terminal-3) and run the following commands:

    $ su - hduser
    Password:
    $ cd /usr/local/kafka
    $ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Hey-Kafka
    Created topic "Hey Kafka!"

You can now see that topic, if we run the list topic command:

    $ bin/kafka-topics.sh --list --zookeeper localhost:2181

You can take help from the image given below:

Step 6: Send some messages

Run the producer and then type a few messages into the console to send to the server.

    $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hey-Kafka

    This is a message       < Enter >
    This is another message < Enter >

You can take help from the image given below:

Step 7: Start a consumer

Open a new terminal (say terminal-4) and start the consumer using the following command:

    $ su - hduser
    Password:
    $ cd /usr/local/kafka/
    $ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Hey-Kafka --from-beginning

Now what you type at terminal-3 i.e., producer produces, you get the same thing at terminal-4 i.e., consumer consumes.

You can take help from the image given below: