In this tutorial:
- Start with a blank Ubuntu
- Install requirements
- Download code & pre-tranied inception-v3 model from Google
- Export the downloaded "checkpoint" format to a TF Graph that is servable with Tensorflow Model Serving
- Query the server with images of a cat and a dog
- Compiling Tensorflow Serving from source (on docker with their official instructions) produces an internal gcc error (probably specific to tensorflow commit / gcc version used when writing this:
tf serving commit: c1ec43508ee57a5d6269116aba82d2a16d383c8a
) - If you need to debug or write code, get yourself a decent IDE environment such as PyCharm on the local machine that is able to execute the code.
- You have an existing model that can be loaded via Tensorflow
- Work on *nix (Specific instructions are for Ubuntu). Tensorflow Serving is available as a pre-compiled binary using apt-get, this is a big advantage (but should probably work on RHEL / CentOS as well.
- Get tensorflow source (this includes python)
cd ~
git clone --recurse-submodules https://github.com/tensorflow/serving
- Install Python 2.7
- Install
tensorflow-serving-api
- to run the python client for TF Serving
# Note: May need to 'sudo bash' before running this,
# depending on your python installation
pip install tensorflow-serving-api
- Install Tensorflow model server via apt-get TF Model Server
# Add TensorFlow Serving distribution URI as a package source (one time setup)
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
# Install and update TensorFlow ModelServer
sudo apt-get update && sudo apt-get install tensorflow-model-server
# It is now available via global CLI: tensorflow_model_server
Test that the official Inception-v3 works with TF Serving. Full steps are available at: Tensorflow Serving - Inception
- Download & Extract a checkpoint of the model.
root@c97d8e820ced:~/serving$ curl -O http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz
root@c97d8e820ced:~/serving$ tar xzf inception-v3-2016-03-01.tar.gz
root@c97d8e820ced:~/serving$ ls inception-v3
- Convert checkpoint to a model that TF Serving can handle (Wrap with input & output signature) - instructions
# Fix module inception is not on python path (only for current process)
export PYTHONPATH=~/serving/tf_models/research/inception/
# Fix packages - Python files won't run because of missing __init__.py files:
echo > tf_models/research/inception/inception/__init__.py
echo > tf_models/research/inception/inception/slim/__init__.py
# Export inception from checkpoint to servable model
python tensorflow_serving/example/inception_saved_model.py --checkpoint_dir=inception-v3 --output_dir=/tmp/inception-export
- Start TF Serving Model (this will block the current bash)
tensorflow_model_server --port=9000 --model_name=inception --model_base_path=/tmp/inception-export
- Get some samples for the query:
# Download images to /tmp/images/
mkdir /tmp/images
curl -o /tmp/images/cat-sphynx.jpg https://www.pets4homes.co.uk/images/breeds/23/large/3514efe61d990b82bbc37bed00eea52a.jpg
curl -o /tmp/images/dog-husky.jpg https://www.pets4homes.co.uk/images/articles/3994/large/10-reasons-why-everyone-loves-the-siberian-husky-58cf9f7a1137e.jpg
- Start TF Inception Client - instructions
root@c97d8e820ced:~/serving$ python tensorflow_serving/example/inception_client.py --server=localhost:9000 --image=/tmp/images/cat-sphynx.jpg
outputs {
key: "classes"
value {
dtype: DT_STRING
tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
string_val: "Mexican hairless"
string_val: "toy terrier"
string_val: "Italian greyhound"
string_val: "Siamese cat, Siamese"
string_val: "Egyptian cat"
}
}
root@c97d8e820ced:~/serving$ python tensorflow_serving/example/inception_client.py --server=localhost:9000 --image=/tmp/images/dog-husky.jpg
outputs {
key: "classes"
value {
dtype: DT_STRING
tensor_shape {
dim {
size: 1
}
dim {
size: 5
}
}
string_val: "Siberian husky"
string_val: "Eskimo dog, husky"
string_val: "malamute, malemute, Alaskan malamute"
string_val: "timber wolf, grey wolf, gray wolf, Canis lupus"
string_val: "dogsled, dog sled, dog sleigh"
}
}