Skip to content

Instantly share code, notes, and snippets.

@laserson

laserson/gist:4478737

Created January 7, 2013 21:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save laserson/4478737 to your computer and use it in GitHub Desktop.
Save laserson/4478737 to your computer and use it in GitHub Desktop.
Download ZIP
Quick mrjob profiling
Raw
gistfile1.txt
In [3]: cProfile.runctx("job.run_mapper()", globals(), locals())
142667920 function calls (142667912 primitive calls) in 134.090 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 134.090 134.090 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 UserDict.py:18(__getitem__)
6266740 7.745 0.000 53.110 0.000 __init__.py:245(dumps)
6266740 11.868 0.000 45.365 0.000 encoder.py:215(encode)
6266740 27.665 0.000 28.389 0.000 encoder.py:245(iterencode)
2 0.000 0.000 0.000 0.000 job.py:300(steps)
10 0.000 0.000 0.000 0.000 job.py:318(<genexpr>)
2 0.000 0.000 0.000 0.000 job.py:334(mr)
1 12.350 12.350 134.090 134.090 job.py:526(run_mapper)
1 0.000 0.000 0.000 0.000 job.py:668(_steps_desc)
8000001 2.269 0.000 4.885 0.000 job.py:682(_read_input)
1 0.000 0.000 0.000 0.000 job.py:696(_wrap_protocols)
8000001 9.492 0.000 18.593 0.000 job.py:715(read_lines)
3133370 6.441 0.000 67.958 0.000 job.py:727(write_line)
3 0.000 0.000 0.000 0.000 job.py:739(_step_key)
1 0.000 0.000 0.000 0.000 job.py:742(_script_step_mapping)
1 0.000 0.000 0.000 0.000 job.py:776(_pick_protocol_instances)
1 0.000 0.000 0.000 0.000 job.py:821(pick_protocols)
1 0.000 0.000 0.000 0.000 job.py:911(input_protocol)
1 0.000 0.000 0.000 0.000 job.py:921(internal_protocol)
1 0.000 0.000 0.000 0.000 ngrams.py:15(mapper_init)
11133370 23.599 0.000 35.188 0.000 ngrams.py:20(mapper)
1 0.000 0.000 0.000 0.000 posixpath.py:110(basename)
8000000 1.585 0.000 1.585 0.000 protocol.py:173(read)
3133370 4.555 0.000 61.518 0.000 protocol.py:69(write)
6266740 3.853 0.000 56.963 0.000 protocol.py:92(dump_to_string)
1 0.000 0.000 0.000 0.000 re.py:169(findall)
1 0.000 0.000 0.000 0.000 re.py:226(_compile)
1 0.000 0.000 0.000 0.000 sre_compile.py:178(_compile_charset)
1 0.000 0.000 0.000 0.000 sre_compile.py:207(_optimize_charset)
3/1 0.000 0.000 0.000 0.000 sre_compile.py:32(_compile)
1 0.000 0.000 0.000 0.000 sre_compile.py:354(_simple)
1 0.000 0.000 0.000 0.000 sre_compile.py:361(_compile_info)
2 0.000 0.000 0.000 0.000 sre_compile.py:474(isstring)
1 0.000 0.000 0.000 0.000 sre_compile.py:480(_code)
1 0.000 0.000 0.000 0.000 sre_compile.py:495(compile)
6 0.000 0.000 0.000 0.000 sre_parse.py:126(__len__)
10 0.000 0.000 0.000 0.000 sre_parse.py:130(__getitem__)
1 0.000 0.000 0.000 0.000 sre_parse.py:134(__setitem__)
6 0.000 0.000 0.000 0.000 sre_parse.py:138(append)
4/2 0.000 0.000 0.000 0.000 sre_parse.py:140(getwidth)
1 0.000 0.000 0.000 0.000 sre_parse.py:178(__init__)
13 0.000 0.000 0.000 0.000 sre_parse.py:182(__next)
8 0.000 0.000 0.000 0.000 sre_parse.py:195(match)
11 0.000 0.000 0.000 0.000 sre_parse.py:201(get)
1 0.000 0.000 0.000 0.000 sre_parse.py:225(_class_escape)
2/1 0.000 0.000 0.000 0.000 sre_parse.py:301(_parse_sub)
2/1 0.000 0.000 0.000 0.000 sre_parse.py:379(_parse)
1 0.000 0.000 0.000 0.000 sre_parse.py:663(parse)
1 0.000 0.000 0.000 0.000 sre_parse.py:67(__init__)
1 0.000 0.000 0.000 0.000 sre_parse.py:72(opengroup)
1 0.000 0.000 0.000 0.000 sre_parse.py:83(closegroup)
3 0.000 0.000 0.000 0.000 sre_parse.py:90(__init__)
3 0.000 0.000 0.000 0.000 step.py:100(__getitem__)
3 0.000 0.000 0.000 0.000 step.py:118(_render_substep)
1 0.000 0.000 0.000 0.000 step.py:134(render_mapper)
1 0.000 0.000 0.000 0.000 step.py:137(render_combiner)
1 0.000 0.000 0.000 0.000 step.py:140(render_reducer)
1 0.000 0.000 0.000 0.000 step.py:143(description)
2 0.000 0.000 0.000 0.000 step.py:52(__init__)
4 0.000 0.000 0.000 0.000 step.py:63(<genexpr>)
4 0.000 0.000 0.000 0.000 step.py:66(<genexpr>)
4 0.000 0.000 0.000 0.000 step.py:69(<genexpr>)
34 0.000 0.000 0.000 0.000 step.py:71(<genexpr>)
6 0.000 0.000 0.000 0.000 step.py:75(_prefix_set)
14 0.000 0.000 0.000 0.000 step.py:76(<genexpr>)
6 0.000 0.000 0.000 0.000 step.py:78(_check_cmd)
8000001 2.616 0.000 2.616 0.000 util.py:342(read_input)
1 0.000 0.000 0.000 0.000 {_sre.compile}
6 0.000 0.000 0.000 0.000 {any}
72 0.000 0.000 0.000 0.000 {getattr}
12533495 3.691 0.000 3.691 0.000 {isinstance}
16000056/16000054 1.884 0.000 1.884 0.000 {len}
39 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects}
6266740 0.724 0.000 0.724 0.000 {method 'clear' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
8 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'findall' of '_sre.SRE_Pattern' objects}
3 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'iteritems' of 'dict' objects}
6266740 1.416 0.000 1.416 0.000 {method 'join' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'remove' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
8000000 2.631 0.000 2.631 0.000 {method 'rstrip' of 'str' objects}
16000000 6.810 0.000 6.810 0.000 {method 'split' of 'str' objects}
96 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}
4 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects}
6 0.000 0.000 0.000 0.000 {min}
4 0.000 0.000 0.000 0.000 {ord}
3133372 2.895 0.000 2.895 0.000 {sorted}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment