ipython map，list comprehensions，与python自带的map对比的性能测试

ipython_test

在macbook pro 4核上的测试结果，python自带的map表现比起ipython 的map，list comprehensions都要好。但是有优化的multi_mcpi还是不错。
$ ipcluster start --n=4

print mcpi(100000000)
#print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
#print multi_mcpi(dview, 100000000)

$ time py multi_mcpi.py 
importing random from random on engine(s)
3.14141568

real  0m34.134s
user	0m33.633s
sys	0m0.252s
(此时间应该更少一些，ipython client的启动是没必要的，在此种情况下)


#print mcpi(100000000)
#print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
print multi_mcpi(dview, 100000000)

$ time py multi_mcpi.py 
importing random from random on engine(s)
3.1414756

real  0m12.279s
user	0m1.128s
sys	0m0.415s

剩下ipython相关的用法multi_mcpi_by_map，multi_mcpi_by_list_comp几分钟都出不了结果


在16核服务器上的测试
$ ipcluster start --n=32

print mcpi(100000000)
#print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
#print multi_mcpi(dview, 100000000)

$ time python multi_mcpi.py 
importing random from random on engine(s)
3.141699

real    0m36.687s
user    0m36.486s
sys     0m0.116s

#print mcpi(100000000)
print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
#print multi_mcpi(dview, 100000000)

$ time python multi_mcpi.py 
importing random from random on engine(s)
3.14190892

real    0m5.545s
user    0m0.868s
sys     0m0.156s

#print mcpi(100000000)
#print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
print multi_mcpi(dview, 100000000)

$ time python multi_mcpi.py 
importing random from random on engine(s)
3.14149612

real    0m5.615s
user    0m0.748s
sys     0m0.168s

multi_mcpi_by_list_comp的用法在几分钟之内出不了结果

#print mcpi(100000000)
print multi_mcpi_by_map(dview, 1000000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
#print multi_mcpi(dview, 100000000)

$ time python multi_mcpi.py                                                                                                                                                        
importing random from random on engine(s)
3.141635904

real    0m49.666s
user    0m5.560s
sys     0m0.460s

#print mcpi(100000000)
#print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
print multi_mcpi(dview, 1000000000)

$ time python multi_mcpi.py                                                                                                                                                        
importing random from random on engine(s)
3.141651244

real    0m50.820s
user    0m4.696s
sys     0m0.440s

所以在16核服务器上的表现是非常不错的，人海战术的multi_mcpi_by_map远远超过了python自带的map，更重要的是还超过了算法更好的multi_mcpi(由于分布式产生的传递信息量更少),不过list comprehensions的用法估计由于实现的问题，没有任何优势。

所以我的想法是ipython在机器资源很充裕的情况下，在很多场景下多优化优化，还是有用武之地。

multi_mcpi.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from IPython import parallel as p


rc = p.Client()
dview = rc[:]
dview.block = True

with dview.sync_imports():
    from random import random


def mcpi(nsamples):
    s = 0
    for i in xrange(nsamples):
        x = random()
        y = random()
        if x * x + y * y <= 1:
            s += 1
    return 4. * s / nsamples


def mcpi_base(nsamples):
    s = 0
    for i in xrange(nsamples):
        x = random()
        y = random()
        if x * x + y * y <= 1:
            s += 1
    return s


def multi_mcpi_by_map(dview, nsamples):
    p = len(dview.targets)
    if nsamples % p:
        # ensure even divisibility
        nsamples += p - (nsamples % p)

    subsamples = nsamples / p

    parallel_result = dview.map(mcpi_base, [subsamples] * p)
    return sum(parallel_result) * 4. / nsamples


def multi_mcpi_by_list_comp(dview, nsamples):
    dview.scatter('x', range(nsamples))
    dview.execute('y = [(lambda _: 1 if random() ** 2 + random() ** 2 <= 1 else 0)(i) for i in x]')
    y = dview.gather('y')
    return sum(y) * 4. / nsamples


def multi_mcpi(view, nsamples):
    p = len(view.targets)
    if nsamples % p:
        # ensure even divisibility
        nsamples += p - (nsamples % p)

    subsamples = nsamples / p

    ar = view.apply(mcpi, subsamples)
    return sum(ar) / p


#print mcpi(100000000)
print multi_mcpi_by_map(dview, 100000000)  # 传递太耗时
#print multi_mcpi_by_list_comp(dview, 100000000)
#print multi_mcpi(dview, 100000000)

good job :)

good job