nicegraham/gist:6231670

## gistfile1.txt
14:41:36 <nirvdrum_> asavchuk: You broke the grid.
14:41:38 <Dave_B_> using hub 2.35 throughout.... with node on 2.34 it works fine
14:41:40 → cgoldberg joined (~cgoldberg@c-174-63-110-226.hsd1.ma.comcast.net)
14:41:42 <Dave_B_> with node on 2.35 it doesn't
14:41:58 <nirvdrum_> This game is growing pretty tiresome.
14:42:21 <asavchuk> nirvdrum_: how ?
14:42:21 <Dave_B_> the changelog shows 'Updated operadriver to 1.4.' .... anyone know if that means I need to change how i tell it where to look?
14:42:27 <Dave_B_> I'm still using OPERA_PATH
14:42:36 <nirvdrum_> By changing the isAlive check for status call that older nodes don't support.
14:42:47 <nirvdrum_> It's completely taken my cluster of ios-drivers offline.
14:43:49 <asavchuk> hm :( has ios-driver another api ?
14:44:01 <nirvdrum_> We, as a project, committed to all old nodes working with a newer hub for the life the 2.x series.
14:44:58 <nirvdrum_> The "upgrade both" approach isn't feasible for most grids.
14:45:09 <nirvdrum_> Incidentally, the change to using a Location header also broke it.
14:45:25 <nirvdrum_> So, this is the second time within in a week my grid has gone offline by upgrading.
14:45:46 <simonstewart> That's unexpected
14:45:51 <asavchuk> nirvdrum_: i added code for supporting node which respond to /wd/hub/status it with 404 code.
14:46:05 <asavchuk> how ios-driver node respond to it ?
14:46:37 <nirvdrum_> Dunno.  I'm looking into it more now.
14:47:00 <nirvdrum_> ios-driver is unreegistered almost immediately as being down.
14:47:08 → JackDandy joined (~JackDandy@unaffiliated/jackdandy)
14:47:15 <nirvdrum_> I suspect this is the source of some of the support requests I've seen over in #ios-driver.
14:48:47 <nirvdrum_> simonstewart: I think the other problem was in the reverse direction.  New nodes couldn't talk to an older hub.
14:50:18 <simonstewart> Ah. That's less unexpected, though I'm surprised we ripped out the code to follow redirects from the clients
14:50:52 <simonstewart> I was okay with going along with taking out the redirect server-side
14:51:04 <jarib> i don't think it's been taken out client side
14:51:05 <nirvdrum_> We were less rigid about that one, but the idea was newer nodes should still talk to older hubs so you didn't have to take an entire cluster down just to get the latest firefox driver.
14:51:22 <simonstewart> Makes sense
14:51:41 <nirvdrum_> Obviously less of an issue when marionette comes to be.
14:52:04 <simonstewart> Ahh… m-day
14:52:05 <selbot2> M-Day: day in the future when Marionette is made available for non-debug builds
14:52:09 <simonstewart> Looking forward to it
14:53:31 <jarib> asavchuk: according to https://code.google.com/p/selenium/wiki/Grid2, browserTimeout has nothing to do with the hub->node socket connect timeout
14:53:40 <jarib> AFAICT
14:54:30 → gagangowda10 joined (~gagan@72.165.86.42)
14:54:48 <jarib> it deals with node->browser, not hub->node
14:55:53 <nirvdrum_> asavchuk: I haven't verified in Java, but it looks like the problem is the JSON returned doesn't have a "state" value.
14:56:08 <nirvdrum_> asavchuk: Although it seems you also handle that case.  So I'll have to look more into it.
14:56:23 <asavchuk> jarib: it was not mentioned in this doc, i think. https://gist.github.com/alex-savchuk/6231274
14:57:01 <asavchuk> jarib: in this gist i was changed only adding browserTimeout to use as socket timeout. so before browserTimeout was used as connection timeout
14:57:11 → dimacus joined (~dimacus@74.201.7.112)
14:57:13 <andreastt> AutomatedTester: The best bug discussion today: https://bugzilla.mozilla.org/show_bug.cgi?id=233154
14:57:54 <asavchuk> nirvdrum_: yes, i specially handle this case - RC node, as i know. maybe ios-driver returns json in another format?
14:59:07 → barancev joined (~kvirc@194.28.29.152)
14:59:58 <barancev> :log
14:59:58 <selbot2> https://raw.github.com/SeleniumHQ/irc-logs/master/2013/08/14.txt
15:00:13 <barancev> oops!
15:01:08  → FullTiltProspect joined  ⇐ dimacus quit
15:02:11 <seljenkinsbot> Project Firefox 10 XP Native Java Tests build #2188: FIXED in 9 min 53 sec: http://ci.seleniumhq.org:8080/job/Firefox%2010%20XP%20Native%20Java%20Tests/2188/
15:02:12  → dimacus joined  ⇐ aweiteka quit
15:04:00 <nirvdrum_> asavchuk: Well, you can't really rely on any particular field being there: https://code.google.com/p/selenium/wiki/JsonWireProtocol#/status
15:04:13 <nirvdrum_> I suppose that's why the original implementation checked the HTTP status code.
15:05:01  → aweiteka and shanly joined  ⇐ qba73 quit
15:10:20 <Dave_B_> I've written my Opera problem on SO if anyone has any input: http://stackoverflow.com/questions/18233771/does-a-new-path-need-specified-for-opera-using-remotewebdriver-2-35-0-vs-2-34-0
15:12:04 → nirvdrum joined (~nirvdrum@pool-173-48-98-124.bstnma.fios.verizon.net)
15:12:08 <asavchuk> nirvdrum_: agreed. i missed this point.
15:13:35 <nirvdrum_> No problem.  I just moved some meetings around to try to debug and get this fixed today.  If you're around, great.  If not, I'll get through it.
15:14:02 → fatguylaughing joined (~fatguylau@mds-65-64-83-45.meridiandata.com)
15:16:47 <asavchuk> nirvdrum_: i'll do it right now
15:16:51 → tourdedave joined (~tourdedav@c-68-48-54-131.hsd1.va.comcast.net)
15:17:18 <nirvdrum> asavchuk: Well, what's confusing is you catch and handle the JSONException.  I need to tune my logging and see what's actually being raised.
15:17:39 <nirvdrum> Looking for that single JSON element isn't guaranteed to work, but you should be degrading gracefully as far as I can tell.
15:18:11  ↔ mansi popped in
15:18:27 <asavchuk> nirvdrum: i think ios-driver returns something as {'status': 0} it leads to NPE
15:18:44 → mansi joined (~mansi@159.140.254.99)
15:19:04 <nirvdrum> It doesn't have "status" at all.  It does have "state": 0.
15:19:16 <nirvdrum> Sorry, reverse.
15:19:27 → intchanter-work joined (~dfackrell@69-195-66-137.unifiedlayer.com)
15:19:34 <nirvdrum> It doesn't have "state" but does have "status": 0.
15:26:50  → dgrace joined  ⇐ mansi and papegaaij quit
15:32:01 <dgrace> :log
15:32:01 <selbot2> https://raw.github.com/SeleniumHQ/irc-logs/master/2013/08/14.txt
15:32:12  → zx29b, mansi and Gwildor joined  ⇐ dimacus quit  ↔ rebelagentm popped in
15:37:27 — Dave_B_ bangs his head on his desk. Why won't you launch mr. opera?
15:37:36 → drt24 joined (drt24@pip.srcf.societies.cam.ac.uk)
15:38:13 <drt24> should I expect `./go clean && ./go` to succeed on HEAD of selenium?
15:38:14 <selbot2> shs96c about 1 hour ago - Bumping buck to the latest version. | https://code.google.com/p/selenium/source/detail?r=1cfa129 | http://dashboard.ci.seleniumhq.org/#/revision/1cfa129ff7f89f99323b3fd40aa5b1c7c3d36eff
15:38:37 <ddary> how long does it take until the log is written by the selbot? asking because if i click on above link the last log available is from 2013/08/08
15:39:21 → qaguy joined (~Adium@static-108-29-95-114.nycmny.fios.verizon.net)
	14:41:36 <nirvdrum_> asavchuk: You broke the grid.
	14:41:38 <Dave_B_> using hub 2.35 throughout.... with node on 2.34 it works fine
	14:41:40 → cgoldberg joined (~cgoldberg@c-174-63-110-226.hsd1.ma.comcast.net)
	14:41:42 <Dave_B_> with node on 2.35 it doesn't
	14:41:58 <nirvdrum_> This game is growing pretty tiresome.
	14:42:21 <asavchuk> nirvdrum_: how ?
	14:42:21 <Dave_B_> the changelog shows 'Updated operadriver to 1.4.' .... anyone know if that means I need to change how i tell it where to look?
	14:42:27 <Dave_B_> I'm still using OPERA_PATH
	14:42:36 <nirvdrum_> By changing the isAlive check for status call that older nodes don't support.
	14:42:47 <nirvdrum_> It's completely taken my cluster of ios-drivers offline.
	14:43:49 <asavchuk> hm :( has ios-driver another api ?
	14:44:01 <nirvdrum_> We, as a project, committed to all old nodes working with a newer hub for the life the 2.x series.
	14:44:58 <nirvdrum_> The "upgrade both" approach isn't feasible for most grids.
	14:45:09 <nirvdrum_> Incidentally, the change to using a Location header also broke it.
	14:45:25 <nirvdrum_> So, this is the second time within in a week my grid has gone offline by upgrading.
	14:45:46 <simonstewart> That's unexpected
	14:45:51 <asavchuk> nirvdrum_: i added code for supporting node which respond to /wd/hub/status it with 404 code.
	14:46:05 <asavchuk> how ios-driver node respond to it ?
	14:46:37 <nirvdrum_> Dunno. I'm looking into it more now.
	14:47:00 <nirvdrum_> ios-driver is unreegistered almost immediately as being down.
	14:47:08 → JackDandy joined (~JackDandy@unaffiliated/jackdandy)
	14:47:15 <nirvdrum_> I suspect this is the source of some of the support requests I've seen over in #ios-driver.
	14:48:47 <nirvdrum_> simonstewart: I think the other problem was in the reverse direction. New nodes couldn't talk to an older hub.
	14:50:18 <simonstewart> Ah. That's less unexpected, though I'm surprised we ripped out the code to follow redirects from the clients
	14:50:52 <simonstewart> I was okay with going along with taking out the redirect server-side
	14:51:04 <jarib> i don't think it's been taken out client side
	14:51:05 <nirvdrum_> We were less rigid about that one, but the idea was newer nodes should still talk to older hubs so you didn't have to take an entire cluster down just to get the latest firefox driver.
	14:51:22 <simonstewart> Makes sense
	14:51:41 <nirvdrum_> Obviously less of an issue when marionette comes to be.
	14:52:04 <simonstewart> Ahh… m-day
	14:52:05 <selbot2> M-Day: day in the future when Marionette is made available for non-debug builds
	14:52:09 <simonstewart> Looking forward to it
	14:53:31 <jarib> asavchuk: according to https://code.google.com/p/selenium/wiki/Grid2, browserTimeout has nothing to do with the hub->node socket connect timeout
	14:53:40 <jarib> AFAICT
	14:54:30 → gagangowda10 joined (~gagan@72.165.86.42)
	14:54:48 <jarib> it deals with node->browser, not hub->node
	14:55:53 <nirvdrum_> asavchuk: I haven't verified in Java, but it looks like the problem is the JSON returned doesn't have a "state" value.
	14:56:08 <nirvdrum_> asavchuk: Although it seems you also handle that case. So I'll have to look more into it.
	14:56:23 <asavchuk> jarib: it was not mentioned in this doc, i think. https://gist.github.com/alex-savchuk/6231274
	14:57:01 <asavchuk> jarib: in this gist i was changed only adding browserTimeout to use as socket timeout. so before browserTimeout was used as connection timeout
	14:57:11 → dimacus joined (~dimacus@74.201.7.112)
	14:57:13 <andreastt> AutomatedTester: The best bug discussion today: https://bugzilla.mozilla.org/show_bug.cgi?id=233154
	14:57:54 <asavchuk> nirvdrum_: yes, i specially handle this case - RC node, as i know. maybe ios-driver returns json in another format?
	14:59:07 → barancev joined (~kvirc@194.28.29.152)
	14:59:58 <barancev> :log
	14:59:58 <selbot2> https://raw.github.com/SeleniumHQ/irc-logs/master/2013/08/14.txt
	15:00:13 <barancev> oops!
	15:01:08 → FullTiltProspect joined ⇐ dimacus quit
	15:02:11 <seljenkinsbot> Project Firefox 10 XP Native Java Tests build #2188: FIXED in 9 min 53 sec: http://ci.seleniumhq.org:8080/job/Firefox%2010%20XP%20Native%20Java%20Tests/2188/
	15:02:12 → dimacus joined ⇐ aweiteka quit
	15:04:00 <nirvdrum_> asavchuk: Well, you can't really rely on any particular field being there: https://code.google.com/p/selenium/wiki/JsonWireProtocol#/status
	15:04:13 <nirvdrum_> I suppose that's why the original implementation checked the HTTP status code.
	15:05:01 → aweiteka and shanly joined ⇐ qba73 quit
	15:10:20 <Dave_B_> I've written my Opera problem on SO if anyone has any input: http://stackoverflow.com/questions/18233771/does-a-new-path-need-specified-for-opera-using-remotewebdriver-2-35-0-vs-2-34-0
	15:12:04 → nirvdrum joined (~nirvdrum@pool-173-48-98-124.bstnma.fios.verizon.net)
	15:12:08 <asavchuk> nirvdrum_: agreed. i missed this point.
	15:13:35 <nirvdrum_> No problem. I just moved some meetings around to try to debug and get this fixed today. If you're around, great. If not, I'll get through it.
	15:14:02 → fatguylaughing joined (~fatguylau@mds-65-64-83-45.meridiandata.com)
	15:16:47 <asavchuk> nirvdrum_: i'll do it right now
	15:16:51 → tourdedave joined (~tourdedav@c-68-48-54-131.hsd1.va.comcast.net)
	15:17:18 <nirvdrum> asavchuk: Well, what's confusing is you catch and handle the JSONException. I need to tune my logging and see what's actually being raised.
	15:17:39 <nirvdrum> Looking for that single JSON element isn't guaranteed to work, but you should be degrading gracefully as far as I can tell.
	15:18:11 ↔ mansi popped in
	15:18:27 <asavchuk> nirvdrum: i think ios-driver returns something as {'status': 0} it leads to NPE
	15:18:44 → mansi joined (~mansi@159.140.254.99)
	15:19:04 <nirvdrum> It doesn't have "status" at all. It does have "state": 0.
	15:19:16 <nirvdrum> Sorry, reverse.
	15:19:27 → intchanter-work joined (~dfackrell@69-195-66-137.unifiedlayer.com)
	15:19:34 <nirvdrum> It doesn't have "state" but does have "status": 0.
	15:26:50 → dgrace joined ⇐ mansi and papegaaij quit
	15:32:01 <dgrace> :log
	15:32:01 <selbot2> https://raw.github.com/SeleniumHQ/irc-logs/master/2013/08/14.txt
	15:32:12 → zx29b, mansi and Gwildor joined ⇐ dimacus quit ↔ rebelagentm popped in
	15:37:27 — Dave_B_ bangs his head on his desk. Why won't you launch mr. opera?
	15:37:36 → drt24 joined (drt24@pip.srcf.societies.cam.ac.uk)
	15:38:13 <drt24> should I expect `./go clean && ./go` to succeed on HEAD of selenium?
	15:38:14 <selbot2> shs96c about 1 hour ago - Bumping buck to the latest version. \| https://code.google.com/p/selenium/source/detail?r=1cfa129 \| http://dashboard.ci.seleniumhq.org/#/revision/1cfa129ff7f89f99323b3fd40aa5b1c7c3d36eff
	15:38:37 <ddary> how long does it take until the log is written by the selbot? asking because if i click on above link the last log available is from 2013/08/08
	15:39:21 → qaguy joined (~Adium@static-108-29-95-114.nycmny.fios.verizon.net)