automated test stability

meh.txt

<stability%> (<fail>/<success>) platform

Report of all errors with analysis by platform

100% (0/30) identity.browserid.dev.mac.firefox.14
100% (0/30) identity.browserid.dev.windows.7.firefox.13
97%  (1/30) identity.browserid.dev.windows.7.chrome
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.chrome/1082/console
97%  (1/30) identity.browserid.dev.windows.xp.ie.8
  * INTERNAL SERVER ERROR (sauce labs selenium): https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.xp.ie.8/1067/
93%  (2/30) identity.browserid.dev.linux.firefox.13
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.firefox.13/1089/console
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.firefox.13/1086/console
83%  (5/30) identity.browserid.dev.windows.7.ie.9
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1092/
  * NEXT CLICK INEFFECTUAL: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1095/
  * INITIAL LOAD OF INCLUDE.JS FAILS: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1072/
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1085/
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1088/
73%  (8/30) identity.browserid.dev.linux.opera.12
  * (NETWORK) LOAD OF INCLUDE.JS FAILS: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1121/
  * TITLE EXTRACTION IN POLLING FAILS: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1119/
  * GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1111/
  * (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1106/
  * (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1108/
  * (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1103/
  * (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1102/
  * (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1098/


Summary:

17/210 test groups fail - 80% success rate

Broken down by failure categories:

(7) 41% are ci.mozilla.org unable to clone from github
(6) 35% are network connectivity on linux/opera
(2) 12% are uninvestigated issues in IE9
(1) 6% are uninvestigated errors in linux/opera
(1) 6% are internal saucelabs errors

NOTE: each of these 210 "tests" are actually four distinct sub-tests (4 different saucelabs VMs spun up),

I'm investigating the github errors, though it may just have been a github internal burp (all the affected jobs ran between 6:40 and 9:10 am this morning).

Opera isn't a focus leading up to beta 1, I'd suggest we defer troubleshooting those tests for the moment. Cosimo at Opera pinged me offering to help, and I owe him an explanation of the avoid-proxy stuff.

That leaves 4 tests out of 210, which puts us in 98% green range. I'm good to let the matter drop for now.