Today I read about openflow usage in google. The summary is that when failures are detected in a network, the nodes inform the central controller. The CC computes new topologies and distributes to all relevant nodes. So recovery is very fast.
I also started reading about Google's datacenter architecture. The basic ideas are fascinating. The arch is throughtput oriented and not latency oriented. But split-and-parallelize is used to improve latency. It helps that the web search problem is inherently splittable and parallelizable. Failures are handled in software.
I also started reading graphlab examples. The code seems to flow easily and running it is done using mpiexec. So it is a layer built on MPI. The basic operations to be implemented are gather_edges, gather, apply, and scatter_edges. http://docs.graphlab.org/using_graphlab.html