Getting started with Apache Kafka, a high-throughput, distributed, publish-subscribe messaging system

Here’s a blog entry on how to get started using Apache Kafka, a high-throughput, distributed, publish-subscribe messaging system like JMS or ActiveMQ. Very simple to use interface. Sample code is easy to run.

Only caveat is that your /etc/hosts file (or comparable) will need to have an entry for your host name. If you’ve named your machine foo, then you’ll need an entry in your hosts file that reads:

127.0.0.1 foo

Here’s the article:

https://www.mapr.com/blog/getting-started-sample-programs-apache-kafka-09

Here’s my summary of the article:

Getting Started with Kafka
==========================

Found at:

https://www.mapr.com/blog/getting-started-sample-programs-apache-kafka-09

To start the server, first you have to start the included configuration manager,
ZooKeeper, and then run Kafka itself using the ZooKeeper managed configuration.
(If you get errors about an unresolved host name, ensure that the /etc/hosts
file contains a record that specifies the current machine’s host name. For
example:

127.0.0.1 chocolatemilk

Setup Step 0:

cd [kafka installation directory]

Setup Step 1:

bin/zookeeper-server-start.sh config/zookeeper.properties &

Setup Step 2:

bin/kafka-server-start.sh config/server.properties

Setup Step 3 -- create some topics:


bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic summary-markers

Setup Step 4 -- list the topics, because you can:

bin/kafka-topics.sh --list --zookeeper localhost:2181

Now you can run your app.

Demo Step 0:

You can run some example programs. This example has a simple producer and consumer.


git clone https://github.com/mapr-demos/kafka-sample-programs.git
cd kafka-sample-programs/
mvn clean package

Demo Step 1:

Start the consumer:

target/kafka-example consumer

Demo Step 2:

Then run the producer as often as you want to see all the fun:

target/kafka-example producer

Writing a consumer or producer requires the following Maven dependency:


org.apache.kafka
kafka-clients
0.9.0.0

The consumer has a dependency on a histogram library to produce some interesting output.


org.hdrhistogram
HdrHistogram
2.1.8

com.fasterxml.jackson.core
jackson-databind
2.5.1

That's about all you need to run and try out Kafka.