Skip to content

Instantly share code, notes, and snippets.

Last active Jan 16, 2022
What would you like to do?

Install protobuf compiler

brew install protobuf

protoc compiler is responsible to generate language-specific stubs based on the protobuf schema.

Stubs are necessary to encode and decode data. Therefore, encoding always requires language runtime environment and protobuf schema. Stubs implement the Protobuf API to read messages and write messages.

To use the Protobuf API you need the language-specific library. For Python, this is protobuf library.

pip install protobuf

Now you can generate the stub in Python. This will crete

protoc --python-out=. foo.proto

Inspect a protobuf message

$ protoc --decode foo.BarMsg foo.proto < foo.message

JSON-protobuf bridge

To convert a json file into protobuf message, you need the proto file and related stub.

cat sample.json | pbgen > sample.message

Convert json to protobuf message and publish to kafka

cat sample.json | pbgen | kafkacat -Pt foo

process substitution

kafkacat -Plt < <(cat braze_sync_request.json | ./pbgen)
kafkacat -Pt braze_sync_request.message

Consume from kafka and convert to json

kafkacat -C -u -q -f "%R%s" -t foo7 | pq --protofile todolist.proto --msgtype protoblog.TodoList --stream i32be


Install pq

cargo install pq

Convert protobuf message to json

cat foo.message | pq --protofile foo.proto --msgtype foo.BarMsg


  • what is proto extentsion?
  • what is compiled file?


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment