Monday, 15 June 2015

Installing Apache Kafka 0.8.2.1 on Windows 7

JDK and JRE environment variable Setup

1. Set the JAVA_HOME environment variable entry in user variable section 
JAVA_HOME = C:\Program Files (x86)\Java\jdk1.7.0_75 
(Modify the path and jdk version according to your usage)

2. Append JRE server's bin directory in PATH system variable section 
My Path entry looks like following : 

Path = C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.7.0_75\bin;C:\Program Files (x86)\scala\bin;C:\Program Files (x86)\sbt\\bin;C:\Program Files (x86)\Java\jre7\bin;


Download and Setup of Kafka

1. Download Apache Kafka from http://kafka.apache.org/downloads.html (I downloaded kafka_2.10-0.8.2.1.tgz (asc, md5))

2. Unpack kafka_2.10-0.8.2.1.tgz using 7-Zip 
(You can download 7-zip from http://www.7-zip.org/download.html)

3. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
After copying, your directory structure would look like as follows : 
     
C:\
kafka_2.10-0.8.2.1\
bin\
config\
libs\
LICENSE (file)
NOTISE  (file)

4. Download Kafka080WinBatFiles.zip file from https://github.com/HCanber/kafka/releases. Extract the zip and copy the extracted files to C:\kafka_2.10-0.8.2.1\bin\windows (i.e. overwrite the existing files)
Note : I have followed this blog (http://blog.canberger.se/2014/01/kafka-080-on-windows.html) to get the concern configuration files.

5. Open C:\kafka_2.10-0.8.2.1\config\server.properties and update existing log.dir entry as mentioned below : 

log.dirs=c:/kafka_2.10-0.8.2.1/kafka-logs 

6. Open C:\kafka_2.10-0.8.2.1\config\zookeeper.properties and update existing dataDir entry as mentioned below : 

dataDir=c:/kafka_2.10-0.8.2.1/zookeeper-data

(Note : Make sure you use forward slash as depicted for step-5 and step-6)

We are done with all the necessary configuration, Let's start the zookeeper and kafka server as mentioned below.

7. Starting Zookeeper and Kafka server 

Open a command prompt and start Zookeeper server using following command : 

cd c:\kafka_2.10-0.8.2.1
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Note : If you get an error saying "Unrecognized VM option '+UseCompressedOops'
Could not create the Java virtual machine". Here is the solution.

Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-run-class.bat file. 
Find and remove '-XX:+UseCompressedOops' entry


Before removing '-XX:+UseCompressedOops'

IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true

)

After removing '-XX:+UseCompressedOops'

IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true

)


Now open another command prompt and start Kafka server using following command : 
.\bin\windows\kafka-server-start.bat .\config\server.properties

I hope both Zookeeper and Kafka server are executed successfully for you.


8. Now let's create Topic using following command

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test

9. List the topics using following command

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --list  --zookeeper localhost:2181

10. Send message using Kafka console producer

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test

Note : If you get an error 'Error: Could not find or load main class kafka.producer.ConsoleProducer'. Here is the solution.

Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.bat
Change kafka.producer.ConsoleProducer to kafka.tools.ConsoleProducer 

Now you should be able to successfully send the message after fixing above issue. Here is my command prompt output for your reference : 

C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit



11. Consume the above message using Kafka console consumer

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

Note : If you get an error 'Error: Could not find or load main class kafka.consumer.ConsoleConsumer'. Here is the solution.

Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat. 
Change kafka.consumer.ConsoleConsumer to kafka.tools.ConsoleConsumer

My command prompt executes the command successfully after fixing above issue : 

C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit


I hope you should be able to successfully install Apache Kafka.

17 comments:

  1. Hi Rashmit Rathod,

    When Im trying to send the messages using the command '.\bin\windows\kafka-console-producer.bat -–broker-list localhost:9092 -–topic test', I'm getting the below error.Please help me out of these error.

    Exception in thread "main" joptsimple.UnrecognizedOptionException: '?' is not a
    recognized option
    at joptsimple.OptionException.unrecognizedOption(OptionException.java:93
    )
    at joptsimple.OptionParser.validateOptionCharacters(OptionParser.java:47
    0)
    at joptsimple.OptionParser.handleShortOptionCluster(OptionParser.java:42
    0)
    at joptsimple.OptionParser.handleShortOptionToken(OptionParser.java:415)

    at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:
    57)
    at joptsimple.OptionParser.parse(OptionParser.java:392)
    at kafka.tools.ConsoleProducer$ProducerConfig.(ConsoleProducer.sca
    la:216)
    at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:35)
    at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)


    Thanks & Regards,
    VamshiKrishna.CH



    ReplyDelete
  2. Hi Rashmit,
    I'm trying to read from particular offset is that possible ??

    Thanks and Regards,
    Santhosh N

    ReplyDelete
  3. First post that actually worked for me. Thanks!

    ReplyDelete
  4. Great!!
    Its really working fine for me in one go.

    Thanks,
    Krishan

    ReplyDelete
  5. Thank you all for your comments.

    Good to see the post has served it purpose of making Apache Kafka installation easy and pain free :-)

    Happy Learning Guys..

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. was able to flawlessly implement the same in my win 7 machine. Thanks a lot

    ReplyDelete
  8. Works like a charm, thank you.

    ReplyDelete
  9. Hi
    i am facing following issue while starting kafka server
    Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
    java.io.IOException: The system cannot find the path specified

    ReplyDelete
  10. awesome man, u rocks, works perfectly for me.
    but after first run , i am facing issues like

    : Unable to read additional data from client sessionid 0x154bd7f77640000, likely client has closed socket

    Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)

    Please suggest !

    ReplyDelete
  11. First post that has such a proper description for errors that may be faced while configuring kafka.Thank u Rashmi.

    ReplyDelete
  12. I really appreciate information shared above. It’s of great help. If someone wants to learn Online (Virtual) instructor lead live training in APACHE KAFKA, kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training On APACHE KAFKA. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Saurabh Srivastava
    MaxMunus
    E-mail: saurabh@maxmunus.com
    Skype id: saurabhmaxmunus
    Ph:+91 8553576305 / 080 - 41103383
    http://www.maxmunus.com/

    ReplyDelete
  13. while typing cd c:\kafka_2.10-0.8.2.1
    .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties on cmd, it says The system cannot find the path specified. how can i pass it?

    ReplyDelete
  14. Thank You and I have a keen present: What House Renovations Can You Claim On Tax simple home renovations

    ReplyDelete