tf-serving-build-instructions

Build Instructions:
===================

We follow the "Building with Docker" instructions (with modifications) as described here: 
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/building_with_docker.md#building-a-container-from-a-dockerfile

Modfification: 
---------------
Add xla_gpu_jit to the dependencies in the BUILD file and modify the dockerfile to use the change:

Example Diff:
-*******
diff --git a/tensorflow_serving/model_servers/BUILD b/tensorflow_serving/model_servers/BUILD
index 548c5719..12ef8976 100644
--- a/tensorflow_serving/model_servers/BUILD
+++ b/tensorflow_serving/model_servers/BUILD
@@ -391,7 +391,7 @@ cc_library(
     deps = [
         ":server_lib",
         "@org_tensorflow//tensorflow/c:c_api",
-        "@org_tensorflow//tensorflow/compiler/jit:xla_cpu_jit",
+        "@org_tensorflow//tensorflow/compiler/jit:xla_gpu_jit",
         "@org_tensorflow//tensorflow/core:lib",
         "@org_tensorflow//tensorflow/core/platform/cloud:gcs_file_system",
     ] + if_libtpu([
diff --git a/tensorflow_serving/tools/docker/Dockerfile.devel-gpu b/tensorflow_serving/tools/docker/Dockerfile.devel-gpu
index 980ec382..60e5712d 100644
--- a/tensorflow_serving/tools/docker/Dockerfile.devel-gpu
+++ b/tensorflow_serving/tools/docker/Dockerfile.devel-gpu
@@ -145,7 +145,8 @@ ENV TMP="/tmp"
 
 # Download TF Serving sources (optionally at specific commit).
 WORKDIR /tensorflow-serving
-RUN curl -sSL --retry 5 https://github.com/tensorflow/serving/tarball/${TF_SERVING_VERSION_GIT_COMMIT} | tar --strip-components=1 -xzf -
+RUN curl -sSL --retry 5 https://github.com/ultrons/serving/tarball/${TF_SERVING_VERSION_GIT_COMMIT} | tar --strip-components=1 -xzf -



-*******

Once the dockerfiler is updated, you can build using:

docker build --pull -t $USER/tensorflow-serving-gpu -f  Dockerfile.devel-gpu  .

Run Instructions:
=================

Start a tf serving container:
-----------------------------

Start the container with model dir mount and port forwarding:

docker run -v "/tmp/mean:/models/mean" -it -p 8501:8501  -p 8500:8500 --gpus all sivaibhav/tensorflow-serving-gpu 

In this example /tmp/mean contains the model.
Port 8501 is the REST API port, this the where prediction requests will be sent.
Port 8500 is a gRPC port, it's optional, used here to run profiler.

Not that we are using using the development config here. 
i.e. no default ENTRYPOINT or COMMAND is launching the tensorflow server. 
And therefore once the container is up, we will connect to the container using docker exec -it and then start the server.

Alternatively you can add/edit the ENTRYPOINT/CMD to start the server.

Modification:
-------------

Include "--xla_cpu_compilation_enabled=true" in the tensorflow serving launch command:

For example:
tensorflow_model_server --enable_profiler=true  --xla_cpu_compilation_enabled=true --port=8500 --rest_api_port=8501 --model_name=mean --model_base_path=/models/mean


Profiling:
==========

Once the tensorflow server is running, you can also capture serving profile using tensorboard "capture profile" on localhost:8500
Ref: https://www.tensorflow.org/tfx/serving/tensorboard