JDK and JRE environment variable Setup
1. Set the JAVA_HOME environment variable entry in user variable section
JAVA_HOME = C:\Program Files (x86)\Java\jdk1.7.0_75
(Modify the path and jdk version according to your usage)
2. Append JRE server's bin directory in PATH system variable section
My Path entry looks like following :
Path = C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.7.0_75\bin;C:\Program Files (x86)\scala\bin;C:\Program Files (x86)\sbt\\bin;C:\Program Files (x86)\Java\jre7\bin;
Download and Setup of Kafka
1. Download Apache Kafka from http://kafka.apache.org/downloads.html (I downloaded kafka_2.10-0.8.2.1.tgz (asc, md5))
2. Unpack kafka_2.10-0.8.2.1.tgz using 7-Zip
(You can download 7-zip from http://www.7-zip.org/download.html)
3. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
After copying, your directory structure would look like as follows :
C:\
kafka_2.10-0.8.2.1\
bin\
config\
libs\
LICENSE (file)
NOTISE (file)
4. Download Kafka080WinBatFiles.zip file from https://github.com/HCanber/kafka/releases. Extract the zip and copy the extracted files to C:\kafka_2.10-0.8.2.1\bin\windows (i.e. overwrite the existing files)
Note : I have followed this blog (http://blog.canberger.se/2014/01/kafka-080-on-windows.html) to get the concern configuration files.
5. Open C:\kafka_2.10-0.8.2.1\config\server.properties and update existing log.dir entry as mentioned below :
log.dirs=c:/kafka_2.10-0.8.2.1/kafka-logs
6. Open C:\kafka_2.10-0.8.2.1\config\zookeeper.properties and update existing dataDir entry as mentioned below :
dataDir=c:/kafka_2.10-0.8.2.1/zookeeper-data
(Note : Make sure you use forward slash as depicted for step-5 and step-6)
We are done with all the necessary configuration, Let's start the zookeeper and kafka server as mentioned below.
7. Starting Zookeeper and Kafka server
Open a command prompt and start Zookeeper server using following command :
cd c:\kafka_2.10-0.8.2.1
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Note : If you get an error saying "Unrecognized VM option '+UseCompressedOops'
Could not create the Java virtual machine". Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-run-class.bat file.
Find and remove '-XX:+UseCompressedOops' entry
Before removing '-XX:+UseCompressedOops'
IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
After removing '-XX:+UseCompressedOops'
Now open another command prompt and start Kafka server using following command :
.\bin\windows\kafka-server-start.bat .\config\server.properties
I hope both Zookeeper and Kafka server are executed successfully for you.
8. Now let's create Topic using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
9. List the topics using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
10. Send message using Kafka console producer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
Note : If you get an error 'Error: Could not find or load main class kafka.producer.ConsoleProducer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.bat
Change kafka.producer.ConsoleProducer to kafka.tools.ConsoleProducer
Now you should be able to successfully send the message after fixing above issue. Here is my command prompt output for your reference :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
11. Consume the above message using Kafka console consumer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning
Note : If you get an error 'Error: Could not find or load main class kafka.consumer.ConsoleConsumer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat.
Change kafka.consumer.ConsoleConsumer to kafka.tools.ConsoleConsumer
My command prompt executes the command successfully after fixing above issue :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
I hope you should be able to successfully install Apache Kafka.
1. Set the JAVA_HOME environment variable entry in user variable section
JAVA_HOME = C:\Program Files (x86)\Java\jdk1.7.0_75
(Modify the path and jdk version according to your usage)
2. Append JRE server's bin directory in PATH system variable section
My Path entry looks like following :
Path = C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.7.0_75\bin;C:\Program Files (x86)\scala\bin;C:\Program Files (x86)\sbt\\bin;C:\Program Files (x86)\Java\jre7\bin;
Download and Setup of Kafka
1. Download Apache Kafka from http://kafka.apache.org/downloads.html (I downloaded kafka_2.10-0.8.2.1.tgz (asc, md5))
2. Unpack kafka_2.10-0.8.2.1.tgz using 7-Zip
(You can download 7-zip from http://www.7-zip.org/download.html)
3. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
After copying, your directory structure would look like as follows :
C:\
kafka_2.10-0.8.2.1\
bin\
config\
libs\
LICENSE (file)
NOTISE (file)
4. Download Kafka080WinBatFiles.zip file from https://github.com/HCanber/kafka/releases. Extract the zip and copy the extracted files to C:\kafka_2.10-0.8.2.1\bin\windows (i.e. overwrite the existing files)
Note : I have followed this blog (http://blog.canberger.se/2014/01/kafka-080-on-windows.html) to get the concern configuration files.
5. Open C:\kafka_2.10-0.8.2.1\config\server.properties and update existing log.dir entry as mentioned below :
log.dirs=c:/kafka_2.10-0.8.2.1/kafka-logs
6. Open C:\kafka_2.10-0.8.2.1\config\zookeeper.properties and update existing dataDir entry as mentioned below :
dataDir=c:/kafka_2.10-0.8.2.1/zookeeper-data
(Note : Make sure you use forward slash as depicted for step-5 and step-6)
We are done with all the necessary configuration, Let's start the zookeeper and kafka server as mentioned below.
7. Starting Zookeeper and Kafka server
Open a command prompt and start Zookeeper server using following command :
cd c:\kafka_2.10-0.8.2.1
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Note : If you get an error saying "Unrecognized VM option '+UseCompressedOops'
Could not create the Java virtual machine". Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-run-class.bat file.
Find and remove '-XX:+UseCompressedOops' entry
Before removing '-XX:+UseCompressedOops'
IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
After removing '-XX:+UseCompressedOops'
IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
Now open another command prompt and start Kafka server using following command :
.\bin\windows\kafka-server-start.bat .\config\server.properties
I hope both Zookeeper and Kafka server are executed successfully for you.
8. Now let's create Topic using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
9. List the topics using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
10. Send message using Kafka console producer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
Note : If you get an error 'Error: Could not find or load main class kafka.producer.ConsoleProducer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.bat
Change kafka.producer.ConsoleProducer to kafka.tools.ConsoleProducer
Now you should be able to successfully send the message after fixing above issue. Here is my command prompt output for your reference :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
11. Consume the above message using Kafka console consumer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning
Note : If you get an error 'Error: Could not find or load main class kafka.consumer.ConsoleConsumer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat.
Change kafka.consumer.ConsoleConsumer to kafka.tools.ConsoleConsumer
My command prompt executes the command successfully after fixing above issue :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
I hope you should be able to successfully install Apache Kafka.
