This gist summarises the tensorflow related steps I took to get the above combo working.
When its all tested in c++ I'll update the instructions fully (and add anything I forgot below).
But in summary ( with the current master dd06643cf098ed362212ce0f76ee746951466e81 ):
I have uploaded the pip wheel which I believe should work if you have the same setup but no promises (built for compute capability 3.5, 5.2, 6.0 and named tensorflow-gpu).
Install with (not sure dropbox allows this direct linking):