Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Protocol Buffers & gRPC Tutorial: Python & Go

Protocol Buffers Basics: Python & Go

Complete tutorial commands and its results and a message exchange between Python and Go as an additional example.

Install packages

$ brew install protobuf
$ pip install protobuf
$ go get -u github.com/golang/protobuf/protoc-gen-go
$ git clone https://github.com/protocolbuffers/protobuf.git

Compile protocol buffers

Schema Definition

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;

  google.protobuf.Timestamp last_updated = 5;
}

message AddressBook {
  repeated Person people = 1;
}

Python

$ cd protobuf/examples
$ protoc --python_out=. ./addressbook.proto # addressbook_pb2.py

Go

$ cd protocolbuffers/protobuf/examples
$ mkdir $GOPATH/src/github.com/protocolbuffers/protobuf/examples/tutorial
$ protoc --go_out=$GOPATH/src/github.com/protocolbuffers/protobuf/examples/tutorial ./addressbook.proto
$ make go
go build -o add_person_go add_person.go
go build -o list_people_go list_people.go

Exchange a message between Python and Go

Write a message in Python

$ ./add_person.py addressbook.data
addressbook.data: File not found.  Creating a new file.
Enter person ID number: 1234
Enter name: John Doe
Enter email address (blank for none): jdoe@example.com
Enter a phone number (or leave blank to finish): 555-4321
Is this a mobile, home, or work phone? home
Enter a phone number (or leave blank to finish):

Read a message in Go

$ ./list_people_go addressbook.data
Person ID: 1234
  Name: John Doe
  E-mail address: jdoe@example.com
  Home phone #: 555-4321

Write a message in Go

$ ./add_person_go addressbook.data
Enter person ID number: 5678
Enter name: foobar
Enter email address (blank for none): foo@bar.com
Enter a phone number (or leave blank to finish): 111-2345
Is this a mobile, home, or work phone? work
Enter a phone number (or leave blank to finish):

Read a message in Python

$ ./list_people.py addressbook.data
Person ID: 1234
  Name: John Doe
  E-mail address: jdoe@example.com
  Home phone #: 555-4321
Person ID: 5678
  Name: foobar
  E-mail address: foo@bar.com
  Work phone #: 111-2345

Make sure the data in addressbook.data

$ cat addressbook.data | protoc --decode_raw
1 {
  1: "John Doe"
  2: 1234
  3: "jdoe@example.com"
  4 {
    1: "555-4321"
    2: 1
  }
}
1 {
  1: "foobar"
  2: 5678
  3: "foo@bar.com"
  4 {
    1: "111-2345"
    2: 2
  }
}
$ hexdump addressbook.data
0000000 0a 2d 0a 08 4a 6f 68 6e 20 44 6f 65 10 d2 09 1a
0000010 10 6a 64 6f 65 40 65 78 61 6d 70 6c 65 2e 63 6f
0000020 6d 22 0c 0a 08 35 35 35 2d 34 33 32 31 10 01 0a
0000030 26 0a 06 66 6f 6f 62 61 72 10 ae 2c 1a 0b 66 6f
0000040 6f 40 62 61 72 2e 63 6f 6d 22 0c 0a 08 31 31 31
0000050 2d 32 33 34 35 10 02
0000057
$ hexdump -c addressbook.data
0000000  \n   -  \n  \b   J   o   h   n       D   o   e 020   �  \t 032
0000010 020   j   d   o   e   @   e   x   a   m   p   l   e   .   c   o
0000020   m   "  \f  \n  \b   5   5   5   -   4   3   2   1 020 001  \n
0000030   &  \n 006   f   o   o   b   a   r 020   �   , 032  \v   f   o
0000040   o   @   b   a   r   .   c   o   m   "  \f  \n  \b   1   1   1
0000050   -   2   3   4   5 020 002
0000057
$ xxd addressbook.data
00000000: 0a2d 0a08 4a6f 686e 2044 6f65 10d2 091a  .-..John Doe....
00000010: 106a 646f 6540 6578 616d 706c 652e 636f  .jdoe@example.co
00000020: 6d22 0c0a 0835 3535 2d34 3332 3110 010a  m"...555-4321...
00000030: 260a 0666 6f6f 6261 7210 ae2c 1a0b 666f  &..foobar..,..fo
00000040: 6f40 6261 722e 636f 6d22 0c0a 0831 3131  o@bar.com"...111
00000050: 2d32 3334 3510 02                        -2345..
$ xxd -bits addressbook.data
00000000: 00001010 00101101 00001010 00001000 01001010 01101111  .-..Jo
00000006: 01101000 01101110 00100000 01000100 01101111 01100101  hn Doe
0000000c: 00010000 11010010 00001001 00011010 00010000 01101010  .....j
00000012: 01100100 01101111 01100101 01000000 01100101 01111000  doe@ex
00000018: 01100001 01101101 01110000 01101100 01100101 00101110  ample.
0000001e: 01100011 01101111 01101101 00100010 00001100 00001010  com"..
00000024: 00001000 00110101 00110101 00110101 00101101 00110100  .555-4
0000002a: 00110011 00110010 00110001 00010000 00000001 00001010  321...
00000030: 00100110 00001010 00000110 01100110 01101111 01101111  &..foo
00000036: 01100010 01100001 01110010 00010000 10101110 00101100  bar..,
0000003c: 00011010 00001011 01100110 01101111 01101111 01000000  ..foo@
00000042: 01100010 01100001 01110010 00101110 01100011 01101111  bar.co
00000048: 01101101 00100010 00001100 00001010 00001000 00110001  m"...1
0000004e: 00110001 00110001 00101101 00110010 00110011 00110100  11-234
00000054: 00110101 00010000 00000010                             5..

How does Protocol buffer look like in Python?

$ python
Python 3.7.1 (default, Nov 11 2018, 16:01:00)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import addressbook_pb2
>>> person = addressbook_pb2.Person()
>>> person.id = 1234
>>> person.name = "John Doe"
>>> person.email = "jdoe@example.com"
>>> phone = person.phones.add()
>>> phone.number = "555-4321"
>>> phone.type = addressbook_pb2.Person.HOME
>>> person
name: "John Doe"
id: 1234
email: "jdoe@example.com"
phones {
  number: "555-4321"
  type: HOME
}

>>> phone
number: "555-4321"
type: HOME

>>> person.no_such_field = 1  # raises AttributeError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: Assignment not allowed (no field "no_such_field" in protocol message object).
>>> person.id = "1234"        # raises TypeError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '1234' has type str, but expected one of: int, long
>>> print(person)
name: "John Doe"
id: 1234
email: "jdoe@example.com"
phones {
  number: "555-4321"
  type: HOME
}

>>> str(person)
'name: "John Doe"\nid: 1234\nemail: "jdoe@example.com"\nphones {\n  number: "555-4321"\n  type: HOME\n}\n'
>>> person2 = addressbook_pb2.Person()
>>> person2

>>> person2.CopyFrom(person)
>>> person2
name: "John Doe"
id: 1234
email: "jdoe@example.com"
phones {
  number: "555-4321"
  type: HOME
}

>>> person.IsInitialized()
True
>>> person.Clear()
>>> person.IsInitialized()
True
>>> from google.protobuf import json_format
>>> json_format.MessageToJson(person)
'{\n  "name": "John Doe",\n  "id": 1234,\n  "email": "jdoe@example.com",\n  "phones": [\n    {\n      "number": "555-4321"\n    }\n  ]\n}'
>>> print(json_format.MessageToJson(person))
{
  "name": "John Doe",
  "id": 1234,
  "email": "jdoe@example.com",
  "phones": [
    {
      "number": "555-4321"
    }
  ]
}
>>> person_dict = json.loads(json_format.MessageToJson(person))
>>> person_json['name']
'John Doe'

Protocol buffer classes are basically dumb data holders (like structs in C). Wrap generated classes if you want richer behaviour to a generated class.

Extending a Protocol Buffer

https://developers.google.com/protocol-buffers/docs/pythontutorial#extending-a-protocol-buffer

However, keep in mind that new optional fields will not be present in old messages, so you will need to either check explicitly whether they're set with has_, or provide a reasonable default value in your .proto file with [default = value] after the tag number.

Note also that if you added a new repeated field, your new code will not be able to tell whether it was left empty (by new code) or never set at all (by old code) since there is no has_ flag for it.

gRPC

https://grpc.io/

Overview

https://grpc.io/docs/guides/index.html

gRPC Concept

https://grpc.io/docs/guides/concepts.html

Python Quickstart

https://grpc.io/docs/quickstart/python.html

$ pip install grpcio
$ pip install grpcio-tools googleapis-common-protos
$ git clone -b v1.15.0 https://github.com/grpc/grpc
$ cd grpc/examples/python/helloworld
$ cd grpc/examples/python/helloworld
$ python greeter_server.py # In a terminal
$ python greeter_client.py # In another terminal
Greeter client received: Hello, you!
$ vi ../../protos/helloworld.proto # See https://grpc.io/docs/quickstart/python.html#update-a-grpc-service
$ python -m grpc_tools.protoc -I../../protos --python_out=. --grpc_python_out=. ../../protos/helloworld.proto
$ vi greeter_server.py # See https://grpc.io/docs/quickstart/python.html#update-the-server
$ vi greeter_client.py # See https://grpc.io/docs/quickstart/python.html#update-the-client
$ python greeter_server.py # In a terminal
$ python greeter_client.py # In another terminal
Greeter client received: Hello, you!
Greeter client received: Hello again, you! I'm a server in Python!

Go Quickstart

https://grpc.io/docs/quickstart/go.html

$ go get -u google.golang.org/grpc
$ cd $GOPATH/src/google.golang.org/grpc/examples/helloworld
$ go run greeter_server/main.go # In a terminal
$ go run greeter_client/main.go # In another terminal
2018/11/16 09:36:45 Greeting: Hello world
$ vi helloworld/helloworld.proto # See https://grpc.io/docs/quickstart/go.html#update-a-grpc-service
$ protoc -I helloworld/ helloworld/helloworld.proto --go_out=plugins=grpc:helloworld
$ vi greeter_server/main.go # See https://grpc.io/docs/quickstart/go.html#update-the-server
$ vi greeter_client/main.go # See https://grpc.io/docs/quickstart/go.html#update-the-client
$ go run greeter_server/main.go # In a terminal
$ go run greeter_client/main.go
2018/11/16 14:20:49 Greeting: Hello world
2018/11/16 14:20:49 Greeting: Hello again world! I'm a server in Go!

Exchange a message between Go and Python

  • Old version: codes before you made above changes (Original code of tutorial)
  • New version: codes after you added 'SayHelloAgain' update as above
syntax = "proto3";

option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";

package helloworld;

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
  // Sends another greeting
  rpc SayHelloAgain (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

Go and Python can communicate each other even a client is still old version.

$ go run greeter_server/main.go # Old version in a terminal 
$ go run greeter_client/main.go # Old version in another terminal
2018/11/16 09:36:45 Greeting: Hello world
$ python greeter_client.py # Old version in another terminal
Greeter client received: Hello you
$ go run greeter_server/main.go # New version in a terminal
$ python greeter_client.py # Old version in another terminal
Greeter client received: Hello you
$ go run greeter_client/main.go # New version in another terminal
2018/11/16 14:27:36 Greeting: Hello world
2018/11/16 14:27:36 Greeting: Hello again world! I'm a server in Go!
$ python greeter_client.py # New version in another terminal
Greeter client received: Hello you
Greeter client received: Hello again you! I'm a server in Go!

If client version is newer than server, the request fails because of UNIMPLEMENTED (Of course).

$ python greeter_client.py # New version while go server is still old version
Greeter client received: Hello you
Traceback (most recent call last):
  File "greeter_client.py", line 37, in <module>
    run()
  File "greeter_client.py", line 32, in run
    response = stub.SayHelloAgain(helloworld_pb2.HelloRequest(name="you"))
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 533, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNIMPLEMENTED
	details = "unknown service helloworld.Greeter"
	debug_error_string = "{"created":"@1542331683.060293000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"unknown service helloworld.Greeter","grpc_status":12}"

They can communicate vice versa (Server: Python, Client: Go).

$ cd grpc/examples/python/helloworld
$ python greeter_server.py # New version in a terminal
$ go run greeter_client/main.go # New version in another terminal
2018/11/16 14:29:44 Greeting: Hello, world!
2018/11/16 14:29:44 Greeting: Hello again, world! I'm a server in Python!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment