model_serving_preprocess.md

Preprocessing before model serving

1. Get the input and output tensor name from the model
2. Create a tensorflow model builder

   # Create SavedModelBuilder class
   # defines where the model will be exported
   export_path_base = MODEL_PATH
   export_path = os.path.join(
       tf.compat.as_bytes(export_path_base),
       tf.compat.as_bytes(str(VERSION_NUMBER)))
   print('Exporting trained model to', export_path)
   builder = tf.saved_model.builder.SavedModelBuilder(export_path)

3. Set signature definition

    # Example
    # Creates the TensorInfo protobuf objects that encapsulates the input/output tensors
    input_images = tf.placeholder(tf.float32, shape=[None, None, None, 3], name='input_images')
    tensor_info_input = tf.saved_model.utils.build_tensor_info(input_images) 
    tensor_info_height = tf.saved_model.utils.build_tensor_info(tf.constant('height'))
    tensor_info_width = tf.saved_model.utils.build_tensor_info(tf.constant('width'))
    model_output = get_model()
    # output tensor info
    tensor_info_output = tf.saved_model.utils.build_tensor_info(model_output)
   
    # Defines the DeepLab signatures, uses the TF Predict API
    # It receives an image and its dimensions and output the segmentation mask
    prediction_signature = (
        tf.saved_model.signature_def_utils.build_signature_def(
            inputs={'images': tensor_info_input, 'height': tensor_info_height, 'width': tensor_info_width},
            outputs={'segmentation_map': tensor_info_output},
            method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))

4. Export

    builder.add_meta_graph_and_variables(
    sess, [tf.saved_model.tag_constants.SERVING],
    signature_def_map={
        'predict_images':
            prediction_signature,
    })
   
    # export the model
    builder.save(as_text=True)
    print('Done exporting!')

5. Check if signature has been set for the model using 'saved_model_cli' (preinstalled with tensorflow)

   saved_model_cli show --dir PATH_TO_MODEL --all

Reference

1. Serving tensorflow models
2. Model Signature