After spending minutes or hours playing around with all the wonderful examples available for instance on the Google AI hub, one may wants to deploy one model or another online.
This article presents a fast, optimal and neat way of doing it with Tensorflow Serving and Heroku.
| #!/bin/bash | |
| tensorflow_model_server --port=8500 --rest_api_port="${PORT}" --model_name="${MODEL_NAME}" --model_base_path="${MODEL_BASE_PATH}"/"${MODEL_NAME}" "$@" |
| FROM tensorflow/serving | |
| ENV MODEL_BASE_PATH /models | |
| ENV MODEL_NAME classifier | |
| COPY models/classifier /models/classifier | |
| # Fix because base tf_serving_entrypoint.sh does not take $PORT env variable while $PORT is set by Heroku | |
| # CMD is required to run on Heroku | |
| COPY tf_serving_entrypoint.sh /usr/bin/tf_serving_entrypoint.sh |
| response = requests.post( | |
| "http://localhost:8501/v1/models/classifier:predict", | |
| json={ | |
| "signature_name": "serving_default", # can be omitted | |
| "inputs": { | |
| "image_bytes": [image.numpy().decode("utf-8") for image in image_bytes][:2], # batch request | |
| }, | |
| }, | |
| ) |
| tf.saved_model.save( | |
| classifier, | |
| export_dir="classifier/1", | |
| signatures={ | |
| "serving_default": decode_and_serve, | |
| "preprocessing": preprocessing, | |
| }, | |
| ) |
| @tf.function(input_signature=(tf.TensorSpec(shape=[None], dtype=tf.string),)) | |
| def decode(image_bytes): | |
| """ | |
| Takes a base64 encoded image and returns the preprocessed input tensor ready for the inference | |
| """ | |
| # not working on GPU if tf.__version__ < 2.3, see https://github.com/tensorflow/tensorflow/issues/28007 | |
| with tf.device("/cpu:0"): | |
| input_tensor = tf.map_fn( | |
| lambda x: preprocessing(tf.io.decode_jpeg(contents=tf.io.decode_base64(x), channels=3))["output_0"], | |
| image_bytes, |
| @tf.function(input_signature=(tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8),)) | |
| def preprocessing(input_tensor): | |
| output_tensor = tf.cast(input_tensor, dtype=tf.float32) | |
| output_tensor = tf.image.resize_with_pad(output_tensor, target_height=224, target_width=224) | |
| output_tensor = keras_applications.mobilenet.preprocess_input(output_tensor, data_format="channels_last") | |
| return output_tensor |
and why Jupyter Notebook is not a good tool to do it.
At Sicara, we build machine learning based products for our customers:
Jupyter notebook has been heavily reported as the perfect prototyping tool for data scientist. Its main features are:
This overall flexibility has made it a preferred tool compared to the more rustic iPython command line. However it should not be forgotten that this is no more than an REPL where you can navigate efficiently throughout the history.