Pycon Talk: Cython as a Game Changer for Efficiency
You should use Cython for cheap, (surprisingly) easy to write performance improvements.
Cython is a superset of python with additional typing and metadata which allows you to get C-like performance with code which is mostly written in Python.
Instagram changed the django url dispatcher to use cython which with pretty little work gave them a 3x performance boost and dropped cpu usage. (mentioned in the talk)
Because its still basically python, it is very easy to utilize without knowing too much about how it works. Basically free optimization magic.
Logistically, the added work needed to build your code is also luckily, pretty light.
One way to build with cython is via tweaks to a distutils setup.py so including it in the packaging flow can actually be pretty easy if you're using a setup.py already. For more details see the cpython compilation page.
Pycon Talk: Grok the GIL Related Article I knew about the GIL in a general way, but coming from languages where this was not an issue, I didn't fully understand it.
The GIL works akin to time-slicing on a mainframe with a single processor. Basically, the GIL sucks, threads on CPython aren't actually parallel.
The GIL basically provides time-slicing for threads on a process to run using 2 task switching strategies:
- Cooperative: expect an action on a thread to give up the GIL when it is done. An example of this is the CPython socket code which will drop the GIL while it runs the internal C thread-safe "establish connection" code. Tidbit: CPython drops and re-acquires the GIL using an "allow threads"/"disallow threads" macro.
- Pre-emptive: if an thread doesn't give up the GIL within 1000 bytecode instructions (note this has changed in newer versions to 15ms), cut-off the thread and let another one acquire the GIL.
To get real parrallel processing with python, you need to use multiprocessing.
Itss not hard to do with standard python libraries:
A bit of os.pipe (to get reader/writer ends of a pipe), a bit of os.fork (forks a child process) and write your output from the child process to the pipe and read it from the parent and you're cooking with multiprocessing!
And, like with most python things, there are multiple libaries to let you do this easily including the std multiprocessing library and many, many more.
The GIL exists on Cython, other flavors, like Jython may not have it. There are also efforts to remove the GIL from it from Cython, but don't hold your breath as the GIL is a core part of CPython.
-Mark Davidoff