README.md

Talos analyze script

It's just a quick hack

Usage

• Get: http://mxr.mozilla.org/build/source/tools/buildbot-helpers/addon-tests/addons.txt
• Get tinderbox logs incl. baseline log
• Execute: talos_analyze.py *.gz

Limitations

Doesn't currently handle baseline from different OSes or runs (run baselines should be similar/identical anyway, as the same stuff is tested)

talos_analyze_w7_sample.log

Adblock_Plus
--> date:             2011-04-02 16:31:44
--> measurements:     [650, 660, 645, 648, 634, 643, 644, 651, 656, 646, 651, 639, 648, 724, 649, 650, 647, 655, 633]
--> talos_ts:         651.21 ms
--> my_ts:            651.21 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 109.32 ms
--> impact:           20.17 %
--> stddev:           18.38 ms
--> stddev_ts_per:    2.82 %
--> stddev_impact:    3.39 %
!!! stddev/ts impact > 2%

Adblock_Plus
--> date:             2011-04-03 01:57:37
--> measurements:     [838, 548, 530, 548, 634, 544, 535, 539, 546, 533, 543, 545, 540, 541, 550, 532, 541, 544, 625]
--> talos_ts:         566.11 ms
--> my_ts:            566.11 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 24.21 ms
--> impact:           4.47 %
--> stddev:           69.77 ms
--> stddev_ts_per:    12.32 %
--> stddev_impact:    12.88 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

AddThis
--> date:             2011-04-03 01:27:53
--> measurements:     [544, 552, 538, 549, 544, 547, 553, 618, 546, 552, 554, 554, 545, 552, 535, 550, 541, 550, 555]
--> talos_ts:         551.53 ms
--> my_ts:            551.53 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 9.63 ms
--> impact:           1.78 %
--> stddev:           16.59 ms
--> stddev_ts_per:    3.01 %
--> stddev_impact:    3.06 %
!!! stddev/ts impact > 2%

AniWeather
--> date:             2011-04-03 01:41:49
--> measurements:     [1573, 1564, 1560, 1574, 1561, 1565, 1568, 1577, 1582, 1565, 1577, 1563, 1570, 1556, 1565, 1572, 1574, 1561, 1583]
--> talos_ts:         1568.95 ms
--> my_ts:            1568.95 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 1027.05 ms
--> impact:           189.53 %
--> stddev:           7.46 ms
--> stddev_ts_per:    0.48 %
--> stddev_impact:    1.38 %

Ant.com_Video_Downloader_+_Video_Player
--> date:             2011-04-03 01:30:34
--> measurements:     [613, 599, 617, 608, 614, 636, 613, 620, 602, 618, 627, 615, 625, 614, 603, 615, 608, 609, 604]
--> talos_ts:         613.68 ms
--> my_ts:            613.68 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 71.79 ms
--> impact:           13.25 %
--> stddev:           8.96 ms
--> stddev_ts_per:    1.46 %
--> stddev_impact:    1.65 %

AutoPager
--> date:             2011-04-03 01:58:54
--> measurements:     [624, 615, 600, 623, 609, 611, 615, 621, 613, 604, 621, 609, 616, 613, 703, 629, 681, 618, 620]
--> talos_ts:         623.42 ms
--> my_ts:            623.42 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 81.53 ms
--> impact:           15.04 %
--> stddev:           24.75 ms
--> stddev_ts_per:    3.97 %
--> stddev_impact:    4.57 %
!!! stddev/ts impact > 2%

BetterPrivacy
--> date:             2011-04-03 01:28:36
!!! busted

ColorZilla
--> date:             2011-04-03 02:06:23
--> measurements:     [611, 619, 601, 611, 613, 624, 611, 616, 607, 602, 618, 602, 614, 616, 617, 608, 600, 610, 612]
--> talos_ts:         611.16 ms
--> my_ts:            611.16 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 69.26 ms
--> impact:           12.78 %
--> stddev:           6.45 ms
--> stddev_ts_per:    1.06 %
--> stddev_impact:    1.19 %

ColorfulTabs
--> date:             2011-04-03 01:46:00
!!! busted

Delicious_Bookmarks
--> date:             2011-04-03 01:54:15
--> measurements:     [546, 537, 538, 529, 529, 538, 538, 543, 535, 542, 530, 543, 536, 531, 538, 537, 543, 530, 545]
--> talos_ts:         537.26 ms
--> my_ts:            537.26 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: -4.63 ms
--> impact:           -0.85 %
--> stddev:           5.37 ms
--> stddev_ts_per:    1.00 %
--> stddev_impact:    0.99 %
!!! negative impact

DownThemAll!
--> date:             2011-04-03 01:52:06
--> measurements:     [619, 637, 641, 630, 615, 622, 630, 617, 623, 631, 620, 616, 627, 626, 624, 628, 623, 626, 611]
--> talos_ts:         624.53 ms
--> my_ts:            624.53 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 82.63 ms
--> impact:           15.25 %
--> stddev:           7.32 ms
--> stddev_ts_per:    1.17 %
--> stddev_impact:    1.35 %

Download_Flash_and_Video
--> date:             2011-04-03 01:42:54
--> measurements:     [539, 536, 549, 531, 541, 540, 528, 533, 536, 541, 541, 551, 639, 526, 541, 549, 535, 540, 539]
--> talos_ts:         543.95 ms
--> my_ts:            543.95 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 2.05 ms
--> impact:           0.38 %
--> stddev:           23.31 ms
--> stddev_ts_per:    4.28 %
--> stddev_impact:    4.30 %
!!! stddev/ts impact > 2%

Download_Manager_Tweak
--> date:             2011-04-03 01:50:16
--> measurements:     [554, 564, 553, 537, 552, 538, 546, 552, 546, 550, 539, 547, 565, 552, 633, 555, 554, 538, 546]
--> talos_ts:         553.74 ms
--> my_ts:            553.74 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 11.84 ms
--> impact:           2.19 %
--> stddev:           20.21 ms
--> stddev_ts_per:    3.65 %
--> stddev_impact:    3.73 %
!!! stddev/ts impact > 2%

Download_Statusbar
--> date:             2011-04-03 01:49:23
--> measurements:     [602, 631, 612, 623, 608, 657, 605, 709, 618, 628, 632, 622, 626, 611, 617, 621, 622, 626, 617]
--> talos_ts:         625.63 ms
--> my_ts:            625.63 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 83.74 ms
--> impact:           15.45 %
--> stddev:           22.90 ms
--> stddev_ts_per:    3.66 %
--> stddev_impact:    4.23 %
!!! stddev/ts impact > 2%

Download_YouTube_Videos_+
--> date:             2011-04-03 02:11:34
--> measurements:     [610, 602, 610, 600, 608, 594, 608, 597, 605, 615, 609, 609, 599, 609, 617, 607, 617, 611, 598]
--> talos_ts:         606.58 ms
--> my_ts:            606.58 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 64.68 ms
--> impact:           11.94 %
--> stddev:           6.49 ms
--> stddev_ts_per:    1.07 %
--> stddev_impact:    1.20 %

Easy_YouTube_Video_Downloader
--> date:             2011-04-03 01:46:36
--> measurements:     [565, 562, 545, 567, 557, 561, 548, 555, 562, 544, 559, 553, 548, 554, 552, 559, 546, 551, 558]
--> talos_ts:         555.05 ms
--> my_ts:            555.05 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 13.16 ms
--> impact:           2.43 %
--> stddev:           6.70 ms
--> stddev_ts_per:    1.21 %
--> stddev_impact:    1.24 %

Echofon_for_Twitter
--> date:             2011-04-03 01:58:45
--> measurements:     [554, 555, 537, 545, 538, 542, 534, 555, 547, 543, 541, 554, 558, 531, 554, 551, 550, 549, 542]
--> talos_ts:         546.32 ms
--> my_ts:            546.32 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 4.42 ms
--> impact:           0.82 %
--> stddev:           7.71 ms
--> stddev_ts_per:    1.41 %
--> stddev_impact:    1.42 %

FEBE
--> date:             2011-04-03 02:05:20
--> measurements:     [545, 555, 544, 552, 538, 543, 551, 547, 542, 559, 547, 542, 557, 544, 549, 554, 547, 550, 538]
--> talos_ts:         547.58 ms
--> my_ts:            547.58 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 5.68 ms
--> impact:           1.05 %
--> stddev:           5.87 ms
--> stddev_ts_per:    1.07 %
--> stddev_impact:    1.08 %

Facebook_Toolbar
--> date:             2011-04-03 02:08:20
--> measurements:     [582, 578, 572, 566, 574, 561, 557, 570, 563, 566, 585, 571, 584, 554, 572, 563, 575, 565, 608]
--> talos_ts:         571.89 ms
--> my_ts:            571.89 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 30.00 ms
--> impact:           5.54 %
--> stddev:           11.96 ms
--> stddev_ts_per:    2.09 %
--> stddev_impact:    2.21 %
!!! stddev/ts impact > 2%

Fast_Dial
--> date:             2011-04-03 01:47:06
--> measurements:     [552, 543, 548, 553, 534, 545, 554, 543, 548, 553, 557, 536, 556, 548, 554, 587, 542, 537, 543]
--> talos_ts:         549.11 ms
--> my_ts:            549.11 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 7.21 ms
--> impact:           1.33 %
--> stddev:           11.15 ms
--> stddev_ts_per:    2.03 %
--> stddev_impact:    2.06 %
!!! stddev/ts impact > 2%

Fast_Video_Download
--> date:             2011-04-03 01:40:12
--> measurements:     [613, 599, 610, 617, 605, 612, 613, 608, 600, 609, 607, 616, 622, 615, 616, 621, 611, 626, 612]
--> talos_ts:         612.21 ms
--> my_ts:            612.21 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 70.32 ms
--> impact:           12.98 %
--> stddev:           6.73 ms
--> stddev_ts_per:    1.10 %
--> stddev_impact:    1.24 %

Fast_YouTube_Download
--> date:             2011-04-03 01:37:53
--> measurements:     [545, 534, 545, 536, 531, 548, 537, 542, 531, 537, 548, 534, 546, 545, 539, 532, 590, 585, 591]
--> talos_ts:         547.16 ms
--> my_ts:            547.16 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 5.26 ms
--> impact:           0.97 %
--> stddev:           18.82 ms
--> stddev_ts_per:    3.44 %
--> stddev_impact:    3.47 %
!!! stddev/ts impact > 2%

Feedly
--> date:             2011-04-03 02:06:21
--> measurements:     [568, 560, 568, 566, 562, 565, 557, 550, 552, 558, 562, 553, 555, 594, 549, 571, 563, 556, 561]
--> talos_ts:         561.58 ms
--> my_ts:            561.58 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 19.68 ms
--> impact:           3.63 %
--> stddev:           9.80 ms
--> stddev_ts_per:    1.75 %
--> stddev_impact:    1.81 %

Firebug
--> date:             2011-04-03 01:53:16
--> measurements:     [917, 918, 927, 906, 924, 912, 928, 924, 909, 923, 906, 914, 919, 913, 916, 907, 1001, 904, 912]
--> talos_ts:         920.00 ms
--> my_ts:            920.00 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 378.11 ms
--> impact:           69.77 %
--> stddev:           20.39 ms
--> stddev_ts_per:    2.22 %
--> stddev_impact:    3.76 %
!!! stddev/ts impact > 2%

Flagfox
--> date:             2011-04-03 01:36:36
--> measurements:     [753, 784, 621, 611, 615, 598, 608, 618, 622, 606, 642, 655, 637, 789, 651, 777, 808, 882, 763]
--> talos_ts:         686.32 ms
--> my_ts:            686.32 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 144.42 ms
--> impact:           26.65 %
--> stddev:           86.62 ms
--> stddev_ts_per:    12.62 %
--> stddev_impact:    15.99 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

FlashGot
--> date:             2011-04-03 01:53:46
--> measurements:     [616, 610, 625, 612, 622, 622, 615, 617, 608, 616, 622, 621, 618, 622, 630, 605, 622, 612, 620]
--> talos_ts:         617.63 ms
--> my_ts:            617.63 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 75.74 ms
--> impact:           13.98 %
--> stddev:           6.10 ms
--> stddev_ts_per:    0.99 %
--> stddev_impact:    1.13 %

Flash_Video_Downloader
--> date:             2011-04-03 01:28:57
--> measurements:     [586, 583, 587, 599, 600, 582, 590, 584, 614, 604, 596, 589, 597, 582, 603, 577, 600, 605, 594]
--> talos_ts:         593.26 ms
--> my_ts:            593.26 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 51.37 ms
--> impact:           9.48 %
--> stddev:           9.61 ms
--> stddev_ts_per:    1.62 %
--> stddev_impact:    1.77 %

Flashblock
--> date:             2011-04-03 01:48:11
--> measurements:     [529, 536, 540, 533, 540, 531, 548, 529, 532, 537, 543, 536, 540, 528, 541, 538, 537, 535, 538]
--> talos_ts:         536.37 ms
--> my_ts:            536.37 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: -5.53 ms
--> impact:           -1.02 %
--> stddev:           5.06 ms
--> stddev_ts_per:    0.94 %
--> stddev_impact:    0.93 %
!!! negative impact

Forecastfox
--> date:             2011-04-03 01:26:10
--> measurements:     [705, 527, 544, 534, 543, 547, 539, 549, 529, 546, 534, 545, 533, 545, 544, 532, 542, 534, 543]
--> talos_ts:         548.16 ms
--> my_ts:            548.16 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 6.26 ms
--> impact:           1.16 %
--> stddev:           37.52 ms
--> stddev_ts_per:    6.84 %
--> stddev_impact:    6.92 %
!!! stddev/ts impact > 2%

FoxClocks
--> date:             2011-04-03 02:05:16
--> measurements:     [779, 760, 755, 773, 779, 771, 785, 772, 778, 769, 792, 778, 770, 823, 785, 776, 766, 789, 761]
--> talos_ts:         776.89 ms
--> my_ts:            776.89 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 235.00 ms
--> impact:           43.37 %
--> stddev:           14.44 ms
--> stddev_ts_per:    1.86 %
--> stddev_impact:    2.67 %
!!! stddev/ts impact > 2%

FoxTab
--> date:             2011-04-03 01:52:32
--> measurements:     [553, 564, 548, 559, 566, 556, 563, 554, 564, 551, 559, 562, 557, 562, 551, 559, 548, 555, 563]
--> talos_ts:         557.58 ms
--> my_ts:            557.58 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 15.68 ms
--> impact:           2.89 %
--> stddev:           5.48 ms
--> stddev_ts_per:    0.98 %
--> stddev_impact:    1.01 %

FoxyProxy_Standard
--> date:             2011-04-03 01:35:50
--> measurements:     [565, 553, 560, 560, 547, 558, 551, 559, 558, 576, 561, 543, 565, 590, 551, 554, 550, 555, 549]
--> talos_ts:         558.16 ms
--> my_ts:            558.16 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 16.26 ms
--> impact:           3.00 %
--> stddev:           10.52 ms
--> stddev_ts_per:    1.88 %
--> stddev_impact:    1.94 %

FoxyTunes
--> date:             2011-04-03 02:04:03
--> measurements:     [682, 669, 674, 683, 660, 663, 668, 677, 665, 675, 659, 672, 679, 674, 673, 666, 666, 660, 675]
--> talos_ts:         670.53 ms
--> my_ts:            670.53 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 128.63 ms
--> impact:           23.74 %
--> stddev:           7.13 ms
--> stddev_ts_per:    1.06 %
--> stddev_impact:    1.31 %

Glue
--> date:             2011-04-03 02:04:08
--> measurements:     [837, 547, 531, 540, 548, 553, 545, 550, 544, 529, 618, 543, 543, 553, 543, 532, 544, 549, 545]
--> talos_ts:         562.84 ms
--> my_ts:            562.84 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 20.95 ms
--> impact:           3.87 %
--> stddev:           67.04 ms
--> stddev_ts_per:    11.91 %
--> stddev_impact:    12.37 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

Gmail_Notifier
--> date:             2011-04-03 02:11:26
--> measurements:     [543, 524, 548, 533, 546, 537, 544, 531, 529, 531, 537, 546, 534, 548, 538, 528, 539, 546, 529]
--> talos_ts:         537.42 ms
--> my_ts:            537.42 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: -4.47 ms
--> impact:           -0.83 %
--> stddev:           7.42 ms
--> stddev_ts_per:    1.38 %
--> stddev_impact:    1.37 %
!!! negative impact

Google_Shortcuts
--> date:             2011-04-03 01:43:08
--> measurements:     [607, 614, 603, 616, 620, 623, 617, 612, 630, 603, 615, 610, 619, 622, 617, 608, 620, 616, 613]
--> talos_ts:         615.00 ms
--> my_ts:            615.00 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 73.11 ms
--> impact:           13.49 %
--> stddev:           6.70 ms
--> stddev_ts_per:    1.09 %
--> stddev_impact:    1.24 %

IE_Tab
--> date:             2011-04-03 01:51:07
!!! busted

Image_Zoom
--> date:             2011-04-03 02:10:30
--> measurements:     [541, 543, 535, 545, 541, 544, 543, 548, 537, 531, 546, 553, 543, 552, 549, 546, 544, 549, 544]
--> talos_ts:         543.89 ms
--> my_ts:            543.89 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 2.00 ms
--> impact:           0.37 %
--> stddev:           5.33 ms
--> stddev_ts_per:    0.98 %
--> stddev_impact:    0.98 %

Keep_Tube
--> date:             2011-04-03 02:03:09
--> measurements:     [850, 533, 545, 544, 531, 546, 539, 540, 536, 621, 531, 535, 545, 539, 543, 540, 551, 533, 554]
--> talos_ts:         560.84 ms
--> my_ts:            560.84 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 18.95 ms
--> impact:           3.50 %
--> stddev:           70.76 ms
--> stddev_ts_per:    12.62 %
--> stddev_impact:    13.06 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

MinimizeToTray_Plus
--> date:             2011-04-03 01:55:18
!!! busted

Multirow_Bookmarks_Toolbar
--> date:             2011-04-03 01:29:41
--> measurements:     [536, 543, 542, 542, 533, 561, 522, 542, 535, 537, 547, 538, 528, 535, 546, 536, 538, 537, 542]
--> talos_ts:         538.95 ms
--> my_ts:            538.95 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: -2.95 ms
--> impact:           -0.54 %
--> stddev:           7.80 ms
--> stddev_ts_per:    1.45 %
--> stddev_impact:    1.44 %
!!! negative impact

NoScript
--> date:             2011-04-03 01:22:26
--> measurements:     [628, 620, 622, 626, 629, 612, 623, 618, 627, 617, 624, 629, 613, 625, 621, 612, 620, 624, 609]
--> talos_ts:         621.00 ms
--> my_ts:            621.00 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 79.11 ms
--> impact:           14.60 %
--> stddev:           5.96 ms
--> stddev_ts_per:    0.96 %
--> stddev_impact:    1.10 %

Personas_Plus
--> date:             2011-04-03 01:25:42
--> measurements:     [604, 616, 604, 612, 619, 622, 610, 607, 606, 613, 620, 629, 703, 614, 604, 613, 606, 615, 601]
--> talos_ts:         616.74 ms
--> my_ts:            616.74 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 74.84 ms
--> impact:           13.81 %
--> stddev:           21.52 ms
--> stddev_ts_per:    3.49 %
--> stddev_impact:    3.97 %
!!! stddev/ts impact > 2%

Picnik
--> date:             2011-04-03 02:00:06
--> measurements:     [539, 543, 548, 544, 551, 548, 546, 531, 545, 553, 539, 550, 544, 545, 534, 543, 547, 539, 546]
--> talos_ts:         543.95 ms
--> my_ts:            543.95 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 2.05 ms
--> impact:           0.38 %
--> stddev:           5.44 ms
--> stddev_ts_per:    1.00 %
--> stddev_impact:    1.00 %

Provider_for_Google_Calendar
--> date:             2011-04-03 02:09:23
--> measurements:     [543, 533, 541, 543, 552, 538, 545, 553, 528, 547, 540, 544, 574, 542, 532, 548, 545, 548, 543]
--> talos_ts:         544.16 ms
--> my_ts:            544.16 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 2.26 ms
--> impact:           0.42 %
--> stddev:           9.39 ms
--> stddev_ts_per:    1.73 %
--> stddev_impact:    1.73 %

RSS_Ticker
--> date:             2011-04-03 01:57:53
--> measurements:     [620, 821, 615, 625, 621, 816, 614, 815, 629, 779, 631, 783, 628, 636, 609, 609, 876, 609, 821]
--> talos_ts:         692.47 ms
--> my_ts:            692.47 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 150.58 ms
--> impact:           27.79 %
--> stddev:           96.19 ms
--> stddev_ts_per:    13.89 %
--> stddev_impact:    17.75 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

Read_It_Later
--> date:             2011-04-03 01:54:52
--> measurements:     [552, 530, 541, 543, 535, 554, 549, 552, 535, 550, 603, 565, 598, 703, 720, 809, 753, 826, 718]
--> talos_ts:         617.68 ms
--> my_ts:            617.68 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 75.79 ms
--> impact:           13.99 %
--> stddev:           98.49 ms
--> stddev_ts_per:    15.94 %
--> stddev_impact:    18.17 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

ReminderFox
--> date:             2011-04-03 01:38:36
--> measurements:     [619, 613, 619, 602, 614, 621, 624, 618, 606, 613, 601, 609, 611, 616, 611, 617, 605, 608, 621]
--> talos_ts:         613.05 ms
--> my_ts:            613.05 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 71.16 ms
--> impact:           13.13 %
--> stddev:           6.48 ms
--> stddev_ts_per:    1.06 %
--> stddev_impact:    1.20 %

ScrapBook
--> date:             2011-04-03 01:30:00
--> measurements:     [619, 819, 610, 621, 617, 618, 608, 609, 615, 617, 606, 619, 612, 615, 600, 611, 616, 604, 612]
--> talos_ts:         623.58 ms
--> my_ts:            623.58 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 81.68 ms
--> impact:           15.07 %
--> stddev:           46.38 ms
--> stddev_ts_per:    7.44 %
--> stddev_impact:    8.56 %
!!! stddev/ts impact > 2%

SearchPreview
--> date:             2011-04-03 02:07:21
--> measurements:     [614, 616, 599, 612, 605, 615, 619, 610, 613, 600, 600, 612, 613, 612, 609, 598, 610, 619, 603]
--> talos_ts:         609.42 ms
--> my_ts:            609.42 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 67.53 ms
--> impact:           12.46 %
--> stddev:           6.50 ms
--> stddev_ts_per:    1.07 %
--> stddev_impact:    1.20 %

SimilarWeb
--> date:             2011-04-03 02:12:30
--> measurements:     [840, 603, 613, 598, 601, 592, 610, 607, 606, 598, 604, 606, 604, 637, 717, 652, 725, 777, 776]
--> talos_ts:         650.84 ms
--> my_ts:            650.84 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 108.95 ms
--> impact:           20.10 %
--> stddev:           74.23 ms
--> stddev_ts_per:    11.40 %
--> stddev_impact:    13.70 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

Speed_Dial
--> date:             2011-04-03 01:33:44
--> measurements:     [528, 536, 541, 543, 538, 547, 540, 523, 615, 540, 545, 535, 543, 591, 580, 580, 753, 690, 677]
--> talos_ts:         576.05 ms
--> my_ts:            576.05 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 34.16 ms
--> impact:           6.30 %
--> stddev:           62.38 ms
--> stddev_ts_per:    10.83 %
--> stddev_impact:    11.51 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

Stylish
--> date:             2011-04-03 01:41:18
--> measurements:     [577, 573, 572, 592, 598, 569, 558, 573, 580, 583, 576, 578, 571, 564, 589, 588, 574, 572, 585]
--> talos_ts:         577.47 ms
--> my_ts:            577.47 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 35.58 ms
--> impact:           6.57 %
--> stddev:           9.62 ms
--> stddev_ts_per:    1.67 %
--> stddev_impact:    1.78 %

Torbutton
--> date:             2011-04-03 01:59:47
--> measurements:     [551, 557, 544, 554, 559, 552, 633, 546, 551, 543, 561, 548, 559, 546, 546, 567, 563, 557, 554]
--> talos_ts:         557.42 ms
--> my_ts:            557.42 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 15.53 ms
--> impact:           2.87 %
--> stddev:           19.00 ms
--> stddev_ts_per:    3.41 %
--> stddev_impact:    3.51 %
!!! stddev/ts impact > 2%

Tree_Style_Tab
--> date:             2011-04-03 01:59:51
--> measurements:     [853, 549, 546, 556, 553, 556, 540, 557, 552, 551, 555, 554, 557, 558, 549, 555, 560, 545, 552]
--> talos_ts:         568.32 ms
--> my_ts:            568.32 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 26.42 ms
--> impact:           4.88 %
--> stddev:           67.28 ms
--> stddev_ts_per:    11.84 %
--> stddev_impact:    12.42 %
!!! stddev > 10% of the run
!!! stddev/ts impact > 2%

TwitterBar
--> date:             2011-04-03 02:07:26
--> measurements:     [734, 551, 556, 564, 545, 557, 552, 585, 565, 554, 544, 554, 548, 578, 556, 548, 559, 545, 553]
--> talos_ts:         565.68 ms
--> my_ts:            565.68 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 23.79 ms
--> impact:           4.39 %
--> stddev:           41.01 ms
--> stddev_ts_per:    7.25 %
--> stddev_impact:    7.57 %
!!! stddev/ts impact > 2%

Video_DownloadHelper
--> date:             2011-04-03 01:25:00
--> measurements:     [751, 723, 680, 724, 691, 735, 709, 763, 723, 676, 732, 707, 746, 728, 775, 732, 706, 743, 721]
--> talos_ts:         724.47 ms
--> my_ts:            724.47 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 182.58 ms
--> impact:           33.69 %
--> stddev:           25.27 ms
--> stddev_ts_per:    3.49 %
--> stddev_impact:    4.66 %
!!! stddev/ts impact > 2%

Weave_Browser_Sync
--> date:             2011-04-03 01:31:15
--> measurements:     [543, 636, 542, 552, 531, 541, 553, 538, 549, 538, 563, 535, 542, 546, 541, 560, 552, 535, 552]
--> talos_ts:         549.95 ms
--> my_ts:            549.95 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 8.05 ms
--> impact:           1.49 %
--> stddev:           21.93 ms
--> stddev_ts_per:    3.99 %
--> stddev_impact:    4.05 %
!!! stddev/ts impact > 2%

WebMail_Notifier
--> date:             2011-04-03 02:02:00
--> measurements:     [574, 585, 576, 559, 559, 565, 578, 565, 575, 574, 572, 563, 579, 569, 575, 588, 571, 574, 559]
--> talos_ts:         571.58 ms
--> my_ts:            571.58 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 29.68 ms
--> impact:           5.48 %
--> stddev:           8.11 ms
--> stddev_ts_per:    1.42 %
--> stddev_impact:    1.50 %

Web_Developer
--> date:             2011-04-03 01:43:23
--> measurements:     [617, 614, 616, 609, 627, 606, 605, 598, 608, 611, 608, 614, 622, 708, 601, 609, 609, 615, 608]
--> talos_ts:         616.05 ms
--> my_ts:            616.05 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 74.16 ms
--> impact:           13.68 %
--> stddev:           22.67 ms
--> stddev_ts_per:    3.68 %
--> stddev_impact:    4.18 %
!!! stddev/ts impact > 2%

Xmarks_Bookmark_and_Password_Sync
--> date:             2011-04-03 01:40:47
--> measurements:     [875, 699, 703, 688, 732, 660, 697, 705, 679, 705, 690, 701, 712, 702, 707, 693, 704, 690, 714]
--> talos_ts:         708.21 ms
--> my_ts:            708.21 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: 166.32 ms
--> impact:           30.69 %
--> stddev:           41.85 ms
--> stddev_ts_per:    5.91 %
--> stddev_impact:    7.72 %
!!! stddev/ts impact > 2%

YouTube_to_MP3
--> date:             2011-04-03 01:47:43
--> measurements:     [531, 542, 536, 537, 529, 522, 540, 535, 541, 532, 536, 607, 525, 538, 531, 543, 548, 539, 530]
--> talos_ts:         539.05 ms
--> my_ts:            539.05 ms
--> my_ts_baseline:   541.89 ms
--> ts_baseline_diff: -2.84 ms
--> impact:           -0.52 %
--> stddev:           17.19 ms
--> stddev_ts_per:    3.19 %
--> stddev_impact:    3.17 %
!!! negative impact
!!! stddev/ts impact > 2%

Bad: 14/62 (22.58%), ['Adblock_Plus', 'Delicious_Bookmarks', 'Flagfox', 'Flashblock', 'Glue', 'Gmail_Notifier', 'Keep_Tube', 'Multirow_Bookmarks_Toolbar', 'RSS_Ticker', 'Read_It_Later', 'SimilarWeb', 'Speed_Dial', 'Tree_Style_Tab', 'YouTube_to_MP3']

talosanalyse.py

# written by Nils Maier
# public domain

import os, sys, re
import gzip
import math

from datetime import datetime
from glob import glob

addons = {"baseline": "baseline"}
for line in open('addons.txt'):
  u,n = line.strip().split(' ')
  addons[u] = n

def avg(m):
  return float(sum(m)) / len(m)
  
def process_log(fn):
  run_start, run_end = map(lambda d: datetime.fromtimestamp(int(d)), fn.split('.')[:2])
  gp = filter(lambda l: len(l) > 0, map(lambda l: l.strip(), gzip.open(fn, 'rb').readlines()))
  for i in range(len(gp)):
    line = gp[i]
    if line.startswith("wget -O addon.xpi"):
      break
  try:
    addon = re.search('/en-US/.*$', line).group(0)
  except:
    addon = "baseline"
    i = 0
  for i in range(i+1, len(gp)):
    line = gp[i]
    if line.startswith("'python run_tests.py ...'"):
      break
  for i in range(len(gp)):
    line = gp[i]
    if line == "i,val":
      break

  measurements = []

  for i in range(i+1, len(gp)):
    line = gp[i]
    if line.startswith("RETURN:"):
      break
    measurements += int(line.split(",")[1]),
  try:
    talos_ts = float(line.split(': ')[2])
  except:
    return {
      'addon': addon,
      'name': addons[addon],
      'run_start': run_start,
      'run_end': run_end,
      'busted': True
      }
  
  mm = max(measurements)
  measurements = filter(lambda m: m != mm, measurements)
  my_ts = avg(measurements)
  ts_ok = abs(my_ts - talos_ts) < 1
  span = max(measurements) - min(measurements)
  stddev = math.sqrt(avg(map(lambda m: (m - my_ts)**2, measurements)))
  stddev_p = stddev * 100.0 / my_ts
  return {
    'addon': addon,
    'name': addons[addon],
    'run_start': run_start,
    'run_end': run_end,
    'busted': False,
    'measurements': measurements,
    'talos_ts': talos_ts,
    'my_ts': my_ts,
    'ts_ok': ts_ok,
    'span': span,
    'stddev': stddev,
    'stddev_p': stddev_p,
    }

baseline = None
results = []
for x in sys.argv[1:]:
  for f in glob(x):
    print >>sys.stderr, "processing", f
    r = process_log(f)
    if not r:
      continue
    if r['name'] == 'baseline':
      baseline = r
    else:
      results += r,
results.sort(key=lambda x: x['name'])

bad = []
for r in results:
  print
  print r['name']
  print "--> date:            ", r['run_start']
  if r['busted']:
    print '!!! busted'
    continue

  ts_diff = r['my_ts'] - baseline['my_ts']
  impact = (ts_diff) * 100.0 / baseline['my_ts']
  stddev_impact = r['stddev'] * 100.0 / baseline['my_ts']
   
  print "--> measurements:    ", str(r['measurements'])
  print "--> talos_ts:         %.2f ms" % r['talos_ts']
  print "--> my_ts:            %.2f ms" % r['my_ts']
  print "--> my_ts_baseline:   %.2f ms" % baseline['my_ts']
  print "--> ts_baseline_diff: %.2f ms" % ts_diff
  print "--> impact:           %.2f %%" % impact
  print "--> stddev:           %.2f ms" % r['stddev']
  print "--> stddev_ts_per:    %.2f %%" % r['stddev_p']
  print "--> stddev_impact:    %.2f %%" % stddev_impact

  good = True
  if impact < 0:
    print "!!! negative impact"
    good = False
  if r['stddev_p'] > 10:
    print '!!! stddev > 10% of the run'
    good = False
  if not good:
    bad += r['name'],
  if stddev_impact > 2:
    print "!!! stddev/ts impact > 2%"

print
print "Bad: %d/%d (%.2f%%), %s" % (len(bad), len(results), len(bad) * 100.0 / len(results), str(bad))