January 30, 2019
Tools for working with Kafka and Protobufs - all about dumping messages to the console at this point.
https://medium.com/@coderunner/debugging-with-kafkacat-df7851d21968
$ brew install kafkacat --with-yajl
$ kafkacat -L -b localhost:9092
$ kafkacat -P -b localhost:9092 -t awesome-topic
...type or paste message...
$ kafkacat -C -b localhost:9092 -t awesome-topic
$ kafkacat -C -b localhost:9092 -t awesome-topic -p
http://google.github.io/proto-lens/installing-protoc.html
$ brew install protobuf
https://github.com/edenhill/kafkacat/issues/72
The solution in the above issue didn’t quite work for me, I needed to add the -e
option to get kafkacat to exit before protoc would dump the messages.
This means that you don’t get the desired tail/follow functionality but it’s still usable enough with a little extra effort to set the offset.
$ kafkacat -C -b localhost:9092 -t awesome-topic -o beginning -D "" -e | protoc --decode_raw
% Reached end of topic awesome-topic [0] at offset 201: exiting
...messages...
$ kafkacat -C -b localhost:9092 -t awesome-topic -o 201 -D "" -e | protoc --decode_raw
% Reached end of topic awesome-topic [0] at offset 321: exiting
...new messages...
(bugga, it only dumps the last mesage..)
$ kafkacat -C -b localhost -t awesome-topic -D "" -e | protoc --decode=my.message.Type \
--proto_path=./path/to/proto/files My.proto
Written by Jason Galea from Cairns, Australia.
Full Stack developer specialising in React and NodeJS.
LinkedIn | Github | StackOverflow | Twitter