Installing Apache Kafka 0.8.2.1 on Windows 7

JDK and JRE environment variable Setup

1. Set the JAVA_HOME environment variable entry in user variable section 
JAVA_HOME = C:\Program Files (x86)\Java\jdk1.7.0_75 
(Modify the path and jdk version according to your usage)

2. Append JRE server's bin directory in PATH system variable section 
My Path entry looks like following : 

Path = C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.7.0_75\bin;C:\Program Files (x86)\scala\bin;C:\Program Files (x86)\sbt\\bin;C:\Program Files (x86)\Java\jre7\bin;


Download and Setup of Kafka

1. Download Apache Kafka from http://kafka.apache.org/downloads.html (I downloaded kafka_2.10-0.8.2.1.tgz (asc, md5))

2. Unpack kafka_2.10-0.8.2.1.tgz using 7-Zip 
(You can download 7-zip from http://www.7-zip.org/download.html)

3. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
After copying, your directory structure would look like as follows : 
     
C:\
kafka_2.10-0.8.2.1\
bin\
config\
libs\
LICENSE (file)
NOTISE  (file)

4. Download Kafka080WinBatFiles.zip file from https://github.com/HCanber/kafka/releases. Extract the zip and copy the extracted files to C:\kafka_2.10-0.8.2.1\bin\windows (i.e. overwrite the existing files)
Note : I have followed this blog (http://blog.canberger.se/2014/01/kafka-080-on-windows.html) to get the concern configuration files.

5. Open C:\kafka_2.10-0.8.2.1\config\server.properties and update existing log.dir entry as mentioned below : 

log.dirs=c:/kafka_2.10-0.8.2.1/kafka-logs 

6. Open C:\kafka_2.10-0.8.2.1\config\zookeeper.properties and update existing dataDir entry as mentioned below : 

dataDir=c:/kafka_2.10-0.8.2.1/zookeeper-data

(Note : Make sure you use forward slash as depicted for step-5 and step-6)

We are done with all the necessary configuration, Let's start the zookeeper and kafka server as mentioned below.

7. Starting Zookeeper and Kafka server 

Open a command prompt and start Zookeeper server using following command : 

cd c:\kafka_2.10-0.8.2.1
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Note : If you get an error saying "Unrecognized VM option '+UseCompressedOops'
Could not create the Java virtual machine". Here is the solution.

Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-run-class.bat file. 
Find and remove '-XX:+UseCompressedOops' entry


Before removing '-XX:+UseCompressedOops'

IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true

)

After removing '-XX:+UseCompressedOops'

IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true

)

Now open another command prompt and start Kafka server using following command : 
.\bin\windows\kafka-server-start.bat .\config\server.properties

I hope both Zookeeper and Kafka server are executed successfully for you.


8. Now let's create Topic using following command

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test

9. List the topics using following command

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --list  --zookeeper localhost:2181

10. Send message using Kafka console producer

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test

Note : If you get an error 'Error: Could not find or load main class kafka.producer.ConsoleProducer'. Here is the solution.

Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.bat
Change kafka.producer.ConsoleProducer to kafka.tools.ConsoleProducer 

Now you should be able to successfully send the message after fixing above issue. Here is my command prompt output for your reference : 

C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit



11. Consume the above message using Kafka console consumer

cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

Note : If you get an error 'Error: Could not find or load main class kafka.consumer.ConsoleConsumer'. Here is the solution.

Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat. 
Change kafka.consumer.ConsoleConsumer to kafka.tools.ConsoleConsumer

My command prompt executes the command successfully after fixing above issue : 

C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit


I hope you should be able to successfully install Apache Kafka.