I am writing this gist because I spent 6 hours navigating links trying to get Theano to work with CUDA on windows 10. Hopefully, you wouldn't have to. Once theano is setup and running, you can install pymc3 and it all works. I had tensorflow-gpu setup and running on windows 10; it isn't as simple as pip install theano.
I will list down the instructions, with the links where I found them. Hope this helps someone.
This set of instructions depend on anaconda. Also, this is sort of hacky in the end.
Ok let's begin.