answer to Schematic link between Python, C, Theano, CUDA, Windows 7, Machine Code, CPU, GPU and many more?
There is a very good paper published by James Bergstra et al, Proceedings of the 9th PYTHON IN SCIENCE CONF. (SCIPY 2010) describing the idea of Theano and how it works behind the scenes.
In fact, it's abstract is already quite informative. Let's go through it step by step:
Abstract — Theano is a compiler for mathematical expressions in Python that combines the convenience of NumPy’s syntax with the speed of optimized native machine language.
That's a very important relevation: Theano is not a library that computes math expressions by itself, it is a compiler that translates expressions into some form of executable code.
The user composes mathematical expressions in a high-level description that mimics NumPy’s syntax and semantics, while being statically typed and functional (as opposed to imperative). These expressions allow Theano to provide symbolic differentiation.
So this is how Theano works: the user (you) writes a program by creating Theano expressions. An expression is just some data structure in memory, that Theano understans, and that the Theano code knows to optimize.
It's a bit like when you write the same expressions on a piece of paper: The paper is just the medium to capture your formulae, it can't execute them. To execute the formulae someone has to read them, understand them and then apply the painstaking process of working through them step by step to finally come up with a solution.
So in essence your Theano Python program is just the piece of paper capturing expressions, and Theano is the "mind" that reads the paper and upon your requests performs the computation.
Before performing computation, Theano optimizes the choice of expressions, translates them into C++ (or CUDA for GPU), compiles them into dynamically loaded Python modules, all automatically.
In other words, the Theano expressions like tensors and scalars, do not directly run on the CPU or GPU. Rather, your Python code produces an abstract representation of these expressions, which Theano then translates into C++ code or uses other Python library such as Numpy to execute.