Skip to content

Instantly share code, notes, and snippets.

@lloyd
Created August 31, 2012 20:24
Show Gist options
  • Save lloyd/3558478 to your computer and use it in GitHub Desktop.
Save lloyd/3558478 to your computer and use it in GitHub Desktop.
automated test stability
<stability%> (<fail>/<success>) platform
Report of all errors with analysis by platform
100% (0/30) identity.browserid.dev.mac.firefox.14
100% (0/30) identity.browserid.dev.windows.7.firefox.13
97% (1/30) identity.browserid.dev.windows.7.chrome
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.chrome/1082/console
97% (1/30) identity.browserid.dev.windows.xp.ie.8
* INTERNAL SERVER ERROR (sauce labs selenium): https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.xp.ie.8/1067/
93% (2/30) identity.browserid.dev.linux.firefox.13
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.firefox.13/1089/console
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.firefox.13/1086/console
83% (5/30) identity.browserid.dev.windows.7.ie.9
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1092/
* NEXT CLICK INEFFECTUAL: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1095/
* INITIAL LOAD OF INCLUDE.JS FAILS: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1072/
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1085/
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9/1088/
73% (8/30) identity.browserid.dev.linux.opera.12
* (NETWORK) LOAD OF INCLUDE.JS FAILS: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1121/
* TITLE EXTRACTION IN POLLING FAILS: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1119/
* GIT CLONE ERROR: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1111/
* (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1106/
* (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1108/
* (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1103/
* (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1102/
* (NETWORK) load of 12done fails: https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.linux.opera.12/1098/
Summary:
17/210 test groups fail - 80% success rate
Broken down by failure categories:
(7) 41% are ci.mozilla.org unable to clone from github
(6) 35% are network connectivity on linux/opera
(2) 12% are uninvestigated issues in IE9
(1) 6% are uninvestigated errors in linux/opera
(1) 6% are internal saucelabs errors
NOTE: each of these 210 "tests" are actually four distinct sub-tests (4 different saucelabs VMs spun up),
@jaredhirsch
Copy link

I'm investigating the github errors, though it may just have been a github internal burp (all the affected jobs ran between 6:40 and 9:10 am this morning).

Opera isn't a focus leading up to beta 1, I'd suggest we defer troubleshooting those tests for the moment. Cosimo at Opera pinged me offering to help, and I owe him an explanation of the avoid-proxy stuff.

That leaves 4 tests out of 210, which puts us in 98% green range. I'm good to let the matter drop for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment