Lecstor's Blog

Kafka/Protobuf Cheatsheet

January 30, 2019

Tools for working with Kafka and Protobufs - all about dumping messages to the console at this point.




$ brew install kafkacat --with-yajl

list topics

$ kafkacat -L -b localhost:9092


$ kafkacat -P -b localhost:9092 -t awesome-topic
...type or paste message...


$ kafkacat -C -b localhost:9092 -t awesome-topic
$ kafkacat -C -b localhost:9092 -t awesome-topic -p




$ brew install protobuf

KafkaCat & Protoc


The solution in the above issue didn’t quite work for me, I needed to add the -e option to get kafkacat to exit before protoc would dump the messages.

This means that you don’t get the desired tail/follow functionality but it’s still usable enough with a little extra effort to set the offset.

decode any Type of protobuf message and dump raw values

$ kafkacat -C -b localhost:9092 -t awesome-topic -o beginning -D "" -e | protoc --decode_raw
% Reached end of topic awesome-topic [0] at offset 201: exiting
$ kafkacat -C -b localhost:9092 -t awesome-topic -o 201 -D "" -e | protoc --decode_raw
% Reached end of topic awesome-topic [0] at offset 321: exiting
...new messages...

decode specifc Types of protobuf messages and dump pretty values

(bugga, it only dumps the last mesage..)

$ kafkacat -C -b localhost -t awesome-topic -D "" -e | protoc --decode=my.message.Type \
--proto_path=./path/to/proto/files My.proto

Jason Galea

Written by Jason Galea from Cairns, Australia.

Full Stack developer specialising in React and NodeJS.

LinkedIn | Github | StackOverflow | Twitter