Skip to content

Instantly share code, notes, and snippets.

@wting
Last active November 17, 2016 12:21
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wting/5819331 to your computer and use it in GitHub Desktop.
Save wting/5819331 to your computer and use it in GitHub Desktop.
Test Python intersection speeds.
In [4]: dis.dis(try0)
 19           0 LOAD_GLOBAL              0 (set)
              3 LOAD_FAST                0 (x)
              6 LOAD_ATTR                1 (keys)
              9 CALL_FUNCTION            0
             12 CALL_FUNCTION            1
             15 LOAD_ATTR                2 (intersection)
             18 LOAD_FAST                1 (y)
             21 LOAD_ATTR                1 (keys)
             24 CALL_FUNCTION            0
             27 CALL_FUNCTION            1
             30 RETURN_VALUE

In [5]: dis.dis(try1)
 22           0 LOAD_GLOBAL              0 (set)
              3 LOAD_FAST                0 (x)
              6 CALL_FUNCTION            1
              9 LOAD_ATTR                1 (intersection)
             12 LOAD_FAST                1 (y)
             15 CALL_FUNCTION            1
             18 RETURN_VALUE

In [6]: dis.dis(try2)
 25           0 BUILD_LIST               0
              3 LOAD_FAST                0 (x)
              6 GET_ITER
        >>    7 FOR_ITER                24 (to 34)
             10 STORE_FAST               2 (k)
             13 LOAD_FAST                2 (k)
             16 LOAD_FAST                1 (y)
             19 COMPARE_OP               6 (in)
             22 POP_JUMP_IF_FALSE        7
             25 LOAD_FAST                2 (k)
             28 LIST_APPEND              2
             31 JUMP_ABSOLUTE            7
        >>   34 RETURN_VALUE

In [7]: dis.dis(try3)
 28           0 LOAD_GLOBAL              0 (filter)
              3 LOAD_FAST                0 (x)
              6 LOAD_ATTR                1 (has_key)
              9 LOAD_FAST                1 (y)
             12 LOAD_ATTR                2 (keys)
             15 CALL_FUNCTION            0
             18 CALL_FUNCTION            2
             21 RETURN_VALUE

In [8]: dis.dis(try4)
 31           0 LOAD_GLOBAL              0 (filter)
              3 LOAD_FAST                0 (x)
              6 LOAD_ATTR                1 (has_key)
              9 LOAD_FAST                1 (y)
             12 LOAD_ATTR                2 (iterkeys)
             15 CALL_FUNCTION            0
             18 CALL_FUNCTION            2
             21 RETURN_VALUE
#!/usr/bin/env python
# encoding: utf-8
import cProfile
import random
import sys
def create_dict(size):
d = {}
while len(d) < size:
n = random.randint(0, size * 10)
if n not in d:
d[n] = True
return d
def try0(x, y):
return set(x.keys()).intersection(y.keys())
def try1(x, y):
return set(x).intersection(y)
def try2(x, y):
return [k for k in x if k in y]
def try3(x, y):
return filter(x.has_key, y.keys())
def try4(x, y):
return filter(x.has_key, y.iterkeys())
def main(argv=None):
if not argv:
argv = sys.argv
if len(argv) > 1:
size = int(argv[1])
else:
size = 1000
x = create_dict(size)
y = create_dict(size)
assert len(try0(x, y)) \
== len(try1(x, y)) \
== len(try2(x, y)) \
== len(try3(x, y)) \
== len(try4(x, y))
if __name__ == "__main__":
sys.exit(main())

Relevant Lines:

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    4.990    4.990    9.913    9.913 intersection.py:18(try0)
        1    3.489    3.489    6.359    6.359 intersection.py:21(try1)
        1    0.369    0.369    5.233    5.233 intersection.py:27(try3)
        1    4.216    4.216    4.216    4.216 intersection.py:24(try2)
        1    0.000    0.000    4.023    4.023 intersection.py:30(try4)

Full Output:

$  python -m cProfile -s cumulative intersection.py 10000000 > results.txt

         84288048 function calls in 95.562 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.075    2.075   95.562   95.562 intersection.py:4(<module>)
        1    0.286    0.286   93.487   93.487 intersection.py:33(main)
        2   24.045   12.022   63.457   31.728 intersection.py:8(create_dict)
 21071993   10.723    0.000   37.764    0.000 random.py:237(randint)
 21071993   25.071    0.000   27.041    0.000 random.py:173(randrange)
        1    4.990    4.990    9.913    9.913 intersection.py:18(try0)
        2    8.046    4.023    8.046    4.023 {filter}
        1    3.489    3.489    6.359    6.359 intersection.py:21(try1)
        2    6.128    3.064    6.128    3.064 {method 'intersection' of 'set' objects}
        1    0.369    0.369    5.233    5.233 intersection.py:27(try3)
        1    4.216    4.216    4.216    4.216 intersection.py:24(try2)
        1    0.000    0.000    4.023    4.023 intersection.py:30(try4)
        3    2.507    0.836    2.507    0.836 {method 'keys' of 'dict' objects}
 21071993    1.970    0.000    1.970    0.000 {method 'random' of '_random.Random' objects}
 21072001    1.648    0.000    1.648    0.000 {len}
        1    0.000    0.000    0.001    0.001 random.py:40(<module>)
        1    0.000    0.000    0.000    0.000 hashlib.py:55(<module>)
        1    0.000    0.000    0.000    0.000 random.py:91(__init__)
        1    0.000    0.000    0.000    0.000 random.py:100(seed)
        1    0.000    0.000    0.000    0.000 cProfile.py:5(<module>)
        6    0.000    0.000    0.000    0.000 hashlib.py:94(__get_openssl_constructor)
        1    0.000    0.000    0.000    0.000 {posix.urandom}
        1    0.000    0.000    0.000    0.000 {function seed at 0x23e6320}
        1    0.000    0.000    0.000    0.000 __future__.py:48(<module>)
        1    0.000    0.000    0.000    0.000 {math.exp}
        1    0.000    0.000    0.000    0.000 random.py:72(Random)
        2    0.000    0.000    0.000    0.000 {math.log}
        1    0.000    0.000    0.000    0.000 random.py:651(WichmannHill)
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_md5}
        1    0.000    0.000    0.000    0.000 {math.sqrt}
        1    0.000    0.000    0.000    0.000 __future__.py:74(_Feature)
        1    0.000    0.000    0.000    0.000 {sys.exit}
        1    0.000    0.000    0.000    0.000 cProfile.py:66(Profile)
        1    0.000    0.000    0.000    0.000 random.py:801(SystemRandom)
        1    0.000    0.000    0.000    0.000 {method 'iterkeys' of 'dict' objects}
        7    0.000    0.000    0.000    0.000 __future__.py:75(__init__)
        1    0.000    0.000    0.000    0.000 {binascii.hexlify}
        6    0.000    0.000    0.000    0.000 {getattr}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha224}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha512}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha384}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha1}
        1    0.000    0.000    0.000    0.000 {_hashlib.openssl_sha256}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        6    0.000    0.000    0.000    0.000 {globals}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment