jaredhirsch

summary: default version continues to fail. any test >10min in duration is probably one of these.


default selenium version:

#1993
  - add-primary-to-primary: java failure
    - https://saucelabs.com/tests/f9ec6fa8f6984be680f2455af3fe2226#
  - cancel-account: java failure
    - https://saucelabs.com/tests/a113e0a0a4b24e51b4fe7d8c6a28ba6b#

jaredhirsch

Summary:
  - number of tests failing because not detecting 'li#loggedin span' (logged in node) on 123done, even though this is clearly visible on the page. wondering if maybe it's looking at the wrong window? maybe IE9 puts the newest window at the front, not back, of the window_handles array? I think for IE9 we might need to explicitly store the window handle at the start of the test.



Individual runs:

#16 - uncaught wd error caused failure:
  - https://ci.mozilla.org/view/Persona/job/identity.browserid.dev.windows.7.ie.9.latest-selenium/16/console
    returning-user | vista_ie_9 | /var/lib/jenkins/jobs/identity.browserid.dev.windows.7.ie.9.latest-selenium/workspace/automation-tests/node_modules/wd/lib/webdriver.js:70

jaredhirsch

{"0":"subject","1":"async error with json reporter"}
{"0":"context","1":"synchronous error"}
{"0":"vow","1":{"title":"reports error when in json mode","context":"synchronous error","status":"broken","exception":"\u001b[33m      \n      \u001b[32mactual\u001b[39m \u001b[31mexpected\u001b[39m\n      \n      \u001b[31mError: \u001b[39mfud\n      \u001b[39m\u001b[90m // v.js:15\u001b[39m"}}

/Users/jhirsch/codez/github/mozilla-browserid/automation-tests/tests/v.js:24
        throw new Error("fud");
              ^
Error: fud
    at suite.addBatch.asynchronous error.topic (/Users/jhirsch/codez/github/mozilla-browserid/automation-tests/tests/v.js:24:15)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

jaredhirsch

High-level items from the first meeting:

1. zeus level monitoring of outbound response codes
2. our fragile database setup (single writer with cross-colo comms)
3. personastatus.org - religious use builds confidence
4. our "playbook"
5. Scaling work - currently at 2.5M ADU
6. Email vendor capacity analysis

2. Database:

jaredhirsch

#!/usr/bin/env node

const runner = require('./lib/runner.js'),
  testSetup = require('./lib/test-setup.js'),
  assert = require('./lib/asserts.js');
  // assert = require('assert');

var browser;

const utils = require('./lib/utils.js');

jaredhirsch

2071
>>> reset-password-test.js.5: choose new password: succeeds 	0 ms	1
reset-password-test | vista_ie_9 | https://saucelabs.com/tests/8ff4d0dcaed540b18711e25daac96a50
reset-password-test | vista_ie_9 | https://saucelabs.com/tests/fe7b8d0fadbf46689a39dd8d9bdad48f
reset-password-test | vista_ie_9 | https://saucelabs.com/tests/8fd37126db754103966d16c0e7fb528c

>>> returning-user.js.10: sign in using primary, sign out, reload, click sign in, verify primary is selected: succeeds 
returning-user | vista_ie_9 | https://saucelabs.com/tests/516cbd071f6d45368540aa81975560da

jaredhirsch

Generate new email account
Create user
GET /wsapi/session_context
/wsapi/session_context returned 200
GET /wsapi/address_info
/wsapi/address_info returned 200
POST /wsapi/stage_user
/wsapi/stage_user returned 200
Timed out after 60000ms awaiting return of email lopez14501@personatestuser.org
ERROR: app: Timed out after 60000ms awaiting return of email lopez14501@personatestuser.org

jaredhirsch

Summary of recent runs:

Overall success rate: 20/47 runs ~ 45%
Failure breakdown:

10  wd bug (PR submitted: https://github.com/admc/wd/pull/64)
8   click missed element (clicked too soon?) https://github.com/mozilla/browserid/issues/2874
6   no clear reason why test failed https://github.com/mozilla/browserid/issues/2875
6   "We are very sorry there has been an error" in dialog/site
4   ERROR Selenium Server stopped responding (unclear why it stopped responding)

jaredhirsch

Summary of recent runs:

Overall success rate: 18/37 runs ~ 48%
Failure breakdown:

10  click missed element (in many cases, did give it focus)
7   "we are very sorry there has been an error"
5   health check failing mysteriously--we might be looking for the wrong el.
2   wd bug
2   personatestuser setup failure (server status 400, couldn't stage a user)

jaredhirsch

win/ff16: 10/10 success

win/chrome: 9/10 success
  1 missed click

  #2277:
    add-primary-to-primary: missed click on eyedeeme 'create account' button, then hung for 5 min.
    https://saucelabs.com/tests/c9339d1afbc74c91ae035b7655423749#

win/ie9: 8/10 success