JDK and JRE environment variable Setup
1. Set the JAVA_HOME environment variable entry in user variable section
JAVA_HOME = C:\Program Files (x86)\Java\jdk1.7.0_75
(Modify the path and jdk version according to your usage)
2. Append JRE server's bin directory in PATH system variable section
My Path entry looks like following :
Path = C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.7.0_75\bin;C:\Program Files (x86)\scala\bin;C:\Program Files (x86)\sbt\\bin;C:\Program Files (x86)\Java\jre7\bin;
Download and Setup of Kafka
1. Download Apache Kafka from http://kafka.apache.org/downloads.html (I downloaded kafka_2.10-0.8.2.1.tgz (asc, md5))
2. Unpack kafka_2.10-0.8.2.1.tgz using 7-Zip
(You can download 7-zip from http://www.7-zip.org/download.html)
3. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
After copying, your directory structure would look like as follows :
C:\
kafka_2.10-0.8.2.1\
bin\
config\
libs\
LICENSE (file)
NOTISE (file)
4. Download Kafka080WinBatFiles.zip file from https://github.com/HCanber/kafka/releases. Extract the zip and copy the extracted files to C:\kafka_2.10-0.8.2.1\bin\windows (i.e. overwrite the existing files)
Note : I have followed this blog (http://blog.canberger.se/2014/01/kafka-080-on-windows.html) to get the concern configuration files.
5. Open C:\kafka_2.10-0.8.2.1\config\server.properties and update existing log.dir entry as mentioned below :
log.dirs=c:/kafka_2.10-0.8.2.1/kafka-logs
6. Open C:\kafka_2.10-0.8.2.1\config\zookeeper.properties and update existing dataDir entry as mentioned below :
dataDir=c:/kafka_2.10-0.8.2.1/zookeeper-data
(Note : Make sure you use forward slash as depicted for step-5 and step-6)
We are done with all the necessary configuration, Let's start the zookeeper and kafka server as mentioned below.
7. Starting Zookeeper and Kafka server
Open a command prompt and start Zookeeper server using following command :
cd c:\kafka_2.10-0.8.2.1
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Note : If you get an error saying "Unrecognized VM option '+UseCompressedOops'
Could not create the Java virtual machine". Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-run-class.bat file.
Find and remove '-XX:+UseCompressedOops' entry
Before removing '-XX:+UseCompressedOops'
IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
After removing '-XX:+UseCompressedOops'
Now open another command prompt and start Kafka server using following command :
.\bin\windows\kafka-server-start.bat .\config\server.properties
I hope both Zookeeper and Kafka server are executed successfully for you.
8. Now let's create Topic using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
9. List the topics using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
10. Send message using Kafka console producer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
Note : If you get an error 'Error: Could not find or load main class kafka.producer.ConsoleProducer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.bat
Change kafka.producer.ConsoleProducer to kafka.tools.ConsoleProducer
Now you should be able to successfully send the message after fixing above issue. Here is my command prompt output for your reference :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
11. Consume the above message using Kafka console consumer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning
Note : If you get an error 'Error: Could not find or load main class kafka.consumer.ConsoleConsumer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat.
Change kafka.consumer.ConsoleConsumer to kafka.tools.ConsoleConsumer
My command prompt executes the command successfully after fixing above issue :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
I hope you should be able to successfully install Apache Kafka.
1. Set the JAVA_HOME environment variable entry in user variable section
JAVA_HOME = C:\Program Files (x86)\Java\jdk1.7.0_75
(Modify the path and jdk version according to your usage)
2. Append JRE server's bin directory in PATH system variable section
My Path entry looks like following :
Path = C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.7.0_75\bin;C:\Program Files (x86)\scala\bin;C:\Program Files (x86)\sbt\\bin;C:\Program Files (x86)\Java\jre7\bin;
Download and Setup of Kafka
1. Download Apache Kafka from http://kafka.apache.org/downloads.html (I downloaded kafka_2.10-0.8.2.1.tgz (asc, md5))
2. Unpack kafka_2.10-0.8.2.1.tgz using 7-Zip
(You can download 7-zip from http://www.7-zip.org/download.html)
3. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
After copying, your directory structure would look like as follows :
C:\
kafka_2.10-0.8.2.1\
bin\
config\
libs\
LICENSE (file)
NOTISE (file)
4. Download Kafka080WinBatFiles.zip file from https://github.com/HCanber/kafka/releases. Extract the zip and copy the extracted files to C:\kafka_2.10-0.8.2.1\bin\windows (i.e. overwrite the existing files)
Note : I have followed this blog (http://blog.canberger.se/2014/01/kafka-080-on-windows.html) to get the concern configuration files.
5. Open C:\kafka_2.10-0.8.2.1\config\server.properties and update existing log.dir entry as mentioned below :
log.dirs=c:/kafka_2.10-0.8.2.1/kafka-logs
6. Open C:\kafka_2.10-0.8.2.1\config\zookeeper.properties and update existing dataDir entry as mentioned below :
dataDir=c:/kafka_2.10-0.8.2.1/zookeeper-data
(Note : Make sure you use forward slash as depicted for step-5 and step-6)
We are done with all the necessary configuration, Let's start the zookeeper and kafka server as mentioned below.
7. Starting Zookeeper and Kafka server
Open a command prompt and start Zookeeper server using following command :
cd c:\kafka_2.10-0.8.2.1
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Note : If you get an error saying "Unrecognized VM option '+UseCompressedOops'
Could not create the Java virtual machine". Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-run-class.bat file.
Find and remove '-XX:+UseCompressedOops' entry
Before removing '-XX:+UseCompressedOops'
IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
After removing '-XX:+UseCompressedOops'
IF ["%KAFKA_JVM_PERFORMANCE_OPTS%"] EQU [""] (
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
set KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
)
Now open another command prompt and start Kafka server using following command :
.\bin\windows\kafka-server-start.bat .\config\server.properties
I hope both Zookeeper and Kafka server are executed successfully for you.
8. Now let's create Topic using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
9. List the topics using following command
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181
10. Send message using Kafka console producer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
Note : If you get an error 'Error: Could not find or load main class kafka.producer.ConsoleProducer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-producer.bat
Change kafka.producer.ConsoleProducer to kafka.tools.ConsoleProducer
Now you should be able to successfully send the message after fixing above issue. Here is my command prompt output for your reference :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
11. Consume the above message using Kafka console consumer
cd C:\kafka_2.10-0.8.2.1
.\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning
Note : If you get an error 'Error: Could not find or load main class kafka.consumer.ConsoleConsumer'. Here is the solution.
Open C:\kafka_2.10-0.8.2.1\bin\windows\kafka-console-consumer.bat.
Change kafka.consumer.ConsoleConsumer to kafka.tools.ConsoleConsumer
My command prompt executes the command successfully after fixing above issue :
C:\kafka_2.10-0.8.2.1>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
[2015-06-01 14:08:20,196] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
1. Hello Kafka from Rashmit
2. Good bye Kafka from Rashmit
I hope you should be able to successfully install Apache Kafka.
Hi Rashmit Rathod,
ReplyDeleteWhen Im trying to send the messages using the command '.\bin\windows\kafka-console-producer.bat -–broker-list localhost:9092 -–topic test', I'm getting the below error.Please help me out of these error.
Exception in thread "main" joptsimple.UnrecognizedOptionException: '?' is not a
recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:93
)
at joptsimple.OptionParser.validateOptionCharacters(OptionParser.java:47
0)
at joptsimple.OptionParser.handleShortOptionCluster(OptionParser.java:42
0)
at joptsimple.OptionParser.handleShortOptionToken(OptionParser.java:415)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:
57)
at joptsimple.OptionParser.parse(OptionParser.java:392)
at kafka.tools.ConsoleProducer$ProducerConfig.(ConsoleProducer.sca
la:216)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:35)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Thanks & Regards,
VamshiKrishna.CH
Hi Rashmit,
ReplyDeleteI'm trying to read from particular offset is that possible ??
Thanks and Regards,
Santhosh N
Thanks for the start !
ReplyDeleteFirst post that actually worked for me. Thanks!
ReplyDeleteGreat!!
ReplyDeleteIts really working fine for me in one go.
Thanks,
Krishan
Really great!!
ReplyDeleteThank you all for your comments.
ReplyDeleteGood to see the post has served it purpose of making Apache Kafka installation easy and pain free :-)
Happy Learning Guys..
This comment has been removed by the author.
ReplyDeletewas able to flawlessly implement the same in my win 7 machine. Thanks a lot
ReplyDeleteWorks like a charm, thank you.
ReplyDeleteHi
ReplyDeletei am facing following issue while starting kafka server
Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.IOException: The system cannot find the path specified
i am facing the same issue in cmd .
Deleteawesome man, u rocks, works perfectly for me.
ReplyDeletebut after first run , i am facing issues like
: Unable to read additional data from client sessionid 0x154bd7f77640000, likely client has closed socket
Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
Please suggest !
First post that has such a proper description for errors that may be faced while configuring kafka.Thank u Rashmi.
ReplyDeleteI really appreciate information shared above. It’s of great help. If someone wants to learn Online (Virtual) instructor lead live training in APACHE KAFKA, kindly contact us http://www.maxmunus.com/contact
ReplyDeleteMaxMunus Offer World Class Virtual Instructor led training On APACHE KAFKA. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Saurabh Srivastava
MaxMunus
E-mail: saurabh@maxmunus.com
Skype id: saurabhmaxmunus
Ph:+91 8553576305 / 080 - 41103383
http://www.maxmunus.com/
while typing cd c:\kafka_2.10-0.8.2.1
ReplyDelete.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties on cmd, it says The system cannot find the path specified. how can i pass it?
Thank You and I have a keen present: What House Renovations Can You Claim On Tax simple home renovations
ReplyDelete