Message Queues Demo

Apache Kafka

StartStart zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

Start brokers:

bin/kafka-server-start.sh config/server.properties

bin/kafka-server-start.sh config/server1.properties

bin/kafka-server-start.sh config/server2.properties

Create a topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic demo

bin/kafka-topics.sh --list --zookeeper localhost:2181

Create a producer

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo

Create a consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic demo

Kill a brokerbin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic demo

Then kill a leader broker…

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic demo

Check available messages…bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic demo

Kafka Streams for data processingLet’s create a file…

echo -e "all streams lead to kafka\nhello kafka streams\njoin kafka summit" > file-input.txt

...and then create a topic…

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic streams-file-input

...and publish data to this topic…

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic streams-file-input < file-input.txt

Kafka Streams for data processingLet’s run an analytics…

bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo

And see results in output topic:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic streams-wordcount-output --from-beginning --property print.key=true --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer

Kafka Streams for data processingWordCountDemo:

KTable wordCounts = textLines // Split each text line, by whitespace, into words. .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+")))

// Ensure the words are available as record keys for the next aggregate operation. .map((key, value) -> new KeyValue<>(value, value))

// Count the occurrences of each word (record key) and store the results into a table named "Counts". .countByKey("Counts")

RabbitMQ

Startsbin/rabbitmq-server -detached

nano etc/rabbitmq/rabbitmq.config

sbin/rabbitmqctl status

Web UI:sbin/rabbitmq-plugins enable rabbitmq_management

http://localhost:15672/

Management HTTP API:http://localhost:15672/api/

Sending data

Receive data

What happened

Routing

Exchange typesExchanges - entities where messages are sent.

They take a message and route it into zero or more queues. The routing algorithm used depends on the exchange type and rules called bindings.

Types:

Direct

Fanout

Topic

Headers

Direct exchange

Fanout exchange

Topic Exchange

Clients

Q&A