gistfile1.txt

[19:50:01] <dpickett> hey not sure if it's been reported and I didn't know what the protocol was, but I get errors with the dependency api both locally and when deploying to heroku
[19:50:14] <dpickett> the status site doesn't seem to indicate an awareness of the issue
[19:51:06] --> markstarkman (~markstark@c-68-81-220-27.hsd1.pa.comcast.net) has joined #rubygems
[19:52:56] --> jesser (~jesser@2404:8600:33::3) has joined #rubygems
[19:54:03] <-- jesser (~jesser@2404:8600:33::3) has quit (Read error: Connection reset by peer)
[19:55:04] --> jesser (~jesser@2404:8600:33::3) has joined #rubygems
[19:56:36] <-- markstarkman (~markstark@c-68-81-220-27.hsd1.pa.comcast.net) has quit (Ping timeout: 272 seconds)
[19:58:23] <dpickett> I filed an issue to get clarity on what we should do in these circumstances
[19:58:28] <dpickett> https://github.com/rubygems/rubygems.org/issues/530
[19:58:29] <dwradcliffe> dpickett: Just saw the issue
[19:58:37] <dwradcliffe> dpickett: it seems to be working for me
[19:58:38] <qrush> status seems to be wrong if its down
[19:58:50] <qrush> evan: did you ping indirect on @rubygems_status ?
[19:58:58] <qrush> just seems like a massively bad way to handle communication of this
[19:58:59] <evan> yes.
[19:59:12] <dpickett> http://screens.launchware.com/4d3d71307797f94f6dbf78c0fdcdd90f.jpg
[19:59:34] <dpickett> I mentioned @rubygems in a tweet :-/ didn't know what to do really - that's why I popped in here
[19:59:56] <dpickett> looks like a few people have pinged the status twitter acct
[20:00:01] <dpickett> https://twitter.com/search?q=%40rubygems_status&src=typd
[20:00:37] <dwradcliffe> dpickett: that gem was yanked
[20:00:41] <-- terceiro_ (~terceiro@177.16.250.157) has quit (Quit: Ex-Chat)
[20:00:42] <dwradcliffe> which is why you're getting a 404 I bet
[20:00:52] <dpickett> derp :-( my bad
[20:00:52] <dwradcliffe> https://rubygems.org/gems/newrelic_rpm/versions
[20:01:06] <dwradcliffe> bundler doesn't handle that well
[20:01:45] <dpickett> running bundle update on it now - still getting an HTTPError on the dep api
[20:02:01] <dpickett> but shouldn't be prohibitive for deploys
[20:02:22] --> onemanjujitsu (~sebastian@c-98-204-157-98.hsd1.dc.comcast.net) has joined #rubygems
[20:02:25] <dwradcliffe> hmm
[20:02:55] <qrush> i just restarted bundler-api
[20:02:56] <qrush> heroku restart --app bundler-api
[20:02:56] <qrush> Restarting processes... done
[20:03:01] <qrush> any luck now?
[20:03:16] <dwradcliffe> oh yep, I'm seeing the error now
[20:03:26] <dwradcliffe> qrush: so, no
[20:03:47] <dpickett> http://beerbarrelpolitics.files.wordpress.com/2011/12/borat.jpg
[20:03:58] <qrush> https://metrics.librato.com/share/dashboards/18w46x4p looks pretty fucked
[20:04:14] <qrush> dpickett: it worked?
[20:05:32] <dwradcliffe> I'm getting HTTP/1.1 413 Request Entity Too Large
[20:06:00] <dpickett> yeah I tried a few different rails apps
[20:06:08] <dpickett> and they all did not report an HTTP error
[20:07:12] <qrush> Fetching from: https://rubygems.org/api/v1/dependencies?gems=rake,bubble-wrap,plist,motion-settings-bundle,motion-testflight
[20:07:12] <qrush> HTTP Redirection
[20:07:12] <qrush> Fetching from: http://bundler.rubygems.org/api/v1/dependencies?gems=rake,bubble-wrap,plist,motion-settings-bundle,motion-testflight
[20:07:12] <qrush> HTTP Success
[20:07:18] <qrush> that's seeming OK
[20:07:47] <qrush> I think the latest deploy they had is having trouble
[20:07:52] <qrush> dwradcliffe: what does bundle install --verbose say?
[20:08:01] <qrush> also if you can tell, is there > 100 gems requested? :)
[20:08:09] <dwradcliffe> gist coming
[20:08:39] <qrush> https://github.com/rubygems/bundler-api/compare/020c4f6f211df521e469042e8102740dd2bb92b9...master
[20:08:47] <qrush> i can bump that up
[20:09:08] --> sbeam (~quassel@pool-71-181-46-19.cncdnh.fast.myfairpoint.net) has joined #rubygems
[20:09:24] --> the_mentat (~the_menta@c-98-235-50-116.hsd1.pa.comcast.net) has joined #rubygems
[20:10:19] <dwradcliffe> ha, 101
[20:10:31] <indirect> qrush: yo
[20:10:45] <indirect> why would restarting help? :/
[20:12:11] <-- hahuang65 (~hahuang65@208.65.181.227.static.etheric.net) has quit (Quit: Textual IRC Client: http://www.textualapp.com/)
[20:12:46] <dpickett> hey indirect: per your comment on the GH issue I filed - not trying to be a pest, but seems like it was down for a few hours without an update to the status site
[20:13:03] <qrush> indirect: no idea
[20:13:11] <qrush> thats all i know how to do on heroku ;)
[20:13:12] <dpickett> just interested in helping out - discovered the issue and didn't really know what to do
[20:13:24] <qrush> indirect: sounds like that 100 limit is too low
[20:13:29] <qrush> and bundler is not handling it well
[20:13:56] <dwradcliffe> qrush: bundler handled it ok, dropped back to the full index
[20:14:08] <dwradcliffe> (for me)
[20:14:40] <indirect> qrush: what does "not handling it well" mean?
[20:14:50] <indirect> it says "request too large, fetching full index..."
[20:14:59] <indirect> and then proceeds to install successfully
[20:15:04] <qrush> https://twitter.com/search?q=%40rubygems_status%20&src=typd
[20:15:25] <qrush> the perception seems to be that it's down
[20:15:29] <qrush> at least for some people
[20:15:49] <qrush> https://twitter.com/markbates/status/301493113288019968
[20:15:55] <indirect> I am seeing this bug: https://twitter.com/btmspox/status/299275041047183360 (404s for s3 gemspecs)
[20:16:05] <qrush> you're watching the data but not the people :)
[20:16:10] <indirect> the bundler-api dynos get 404s for some gemspecs :(
[20:16:15] <qrush> https://twitter.com/kenn/status/301424417379057666
[20:16:22] <indirect> the yanked dependencies bug is a completely separate bug
[20:16:23] <indirect> sadly
[20:16:32] <indirect> mostly blocked on the s3 problem
[20:16:37] <qrush> newrelic's issues aside
[20:16:40] <qrush> can we bump that limit?
[20:17:06] <indirect> uh
[20:17:13] <indirect> ugh.
[20:17:58] <qrush> https://twitter.com/search/realtime?q=Bundler%3A%3AHTTPError&src=typd
[20:19:35] <lmarburger> qrush: so you're saying that people with > 100 gems see an error telling them that the API is down so they rightly assume that's a problem?
[20:19:42] <qrush> yes
[20:19:58] <qrush> they assume rubygems.org and bundler are broken
[20:20:07] <qrush> ^ unless if i'm reading those tweets wrong
[20:20:20] <dwradcliffe> qrush: some of those might just be yanked gems
[20:20:31] <lmarburger> i just did a ton of research on the logs and i don't think that > 100 gem bundles is the issue
[20:21:21] <indirect> maybe I need to write a blog post about this or something
[20:21:41] <indirect> qrush: so there are four entirely separate issues going on simultaneously :(
[20:22:00] <indirect> 1. some s3 production gemspecs aren't available in some areas right now
[20:22:04] <lmarburger> it'd be nice if there was a status code that bundler understood as "please try the full index" and not interpret as "things are horribly broken"
[20:22:26] <indirect> 2. some requests to bundler-api are getting 413s because they have too many gems for us to handle well
[20:22:33] <qrush> indirect: FWIW i'm working with thoughtbot to get the rubygems blog redesigned so we can have more of a public place to talk about this
[20:22:45] <indirect> 3. some requests to bundler-api are getting 500s because of intermittent load issues
[20:23:03] <indirect> 4. some people think that bundler-api means something it doesn't
[20:23:06] <lmarburger> i remember a while back we had issues with the bundler api and shut it down to force people to get the full index. everyone thought it was broken even though it installed things fine.
[20:23:11] <indirect> qrush: that would be great
[20:23:17] <indirect> lmarburger: yeah, exactly. wat :(
[20:23:39] <indirect> OH and 5. bundler-api is currently returning some yanked gems
[20:23:46] <qrush> guhhhh :(
[20:23:56] <lmarburger> dpickett: is everything working for you now?
[20:23:56] <indirect> yeah :(
[20:24:13] <dpickett> yessir
[20:24:17] <lmarburger> ok great.
[20:24:18] <indirect> qrush: if people actually can't bundle install, something other than the api is broken. because install doesn't require the api
[20:24:35] <lmarburger> dwradcliffe: is everything working for you too?
[20:24:39] <qrush> i think a full writeup is warranted
[20:24:40] <indirect> qrush: if people are having missing gemspec issues, that's a production rubygems problem that I don't even know how to diagnose :/
[20:24:42] <qrush> if you want to write about it
[20:24:47] <indirect> sure
[20:25:00] <qrush> we can repost to blog.rubygems.org whenever the redesign is done
[20:25:21] <dwradcliffe> lmarburger: yep. I only ever saw the >100 issue
[20:25:25] <qrush> we should point people to it
[20:25:31] <lmarburger> bundler api needs an endpoint on rubygems.org it can ping every minute or few minutes to get a list of gems that have changed (pushed, yanked, unyanked, etc.)
[20:25:37] <lmarburger> dwradcliffe: ok awesome
[20:25:54] <indirect> lmarburger: since time T, preferable
[20:25:57] <indirect> er preferably
[20:26:02] <lmarburger> right
[20:26:03] <qrush> lmarburger: i remember we talked about this forever ago
[20:26:10] <qrush> woo time :(
[20:26:12] <lmarburger> the conversation with raggi the other day got me thinking about this
[20:26:34] <indirect> qrush: the yanked gems issue is because the hook evan wrote is failing some unknown % of the time :(
[20:27:03] <indirect> I'm trying to get the manual synchronizer running, but that's failing out due to the s3 gemspec 404 issue
[20:27:10] <lmarburger> indirect: well hooks in general are failing
[20:27:12] <indirect> even though that gemspec is a totally fine 200 for my laptop :(
[20:27:22] <qrush> well S3 is eventually consistent
[20:27:23] <qrush> soooo
[20:27:23] <indirect> lmarburger: the bundler hooks are totally separate from the "hooks" though
[20:27:27] <qrush> :(
[20:27:28] --> markstarkman (~markstark@c-68-81-220-27.hsd1.pa.comcast.net) has joined #rubygems
[20:27:59] <indirect> according to this job, there appear to be at least 800 yanked gems that bundler-api incorrectly thinks are not yanked :(
[20:28:06] <qrush> awesome
[20:28:08] <lmarburger> i mean we have gem pushes that aren't showing up in the api logs. there are a bunch of things that could cause that failure so not sure where the problem lies
[20:28:25] <indirect> lmarburger: yeah, HTTP requests are not a reliable way to communicate state changes :(
[20:29:01] <qrush> you guys have push to both repos
[20:29:08] <qrush> if you need to fix shit
[20:29:13] <indirect> qrush: working on it!
[20:29:15] <qrush> i'm not sure how the integration works out
[20:29:15] <qrush> but!
[20:29:20] <indirect> so much shit has come up in the last couple of weeks :A(
[20:29:22] <indirect> sigh
[20:29:40] <indirect> totally not expecting you to supply anything
[20:30:00] <qrush> it's worth considering if bringing bundler-api's code into the rails app would help
[20:30:07] <qrush> instead of writing a lot of complicated syncing code
[20:30:25] <qrush> unless if thats worth it. i'm not sure
[20:30:42] <lmarburger> honestly i think this would be less complicated than the current sync doe
[20:30:56] <qrush> lmarburger: this being?
[20:30:57] <lmarburger> s/doe/code
[20:31:19] <qrush> because now on AWS we have the capability to spin up a shitload of instances if we need them
[20:31:29] <qrush> and i dont feel like we should be afraid of that given the ops attention we have
[20:31:30] <lmarburger> this being the sync endpoint i mentioned above. the api just pings the endpoint every N minutes instead of relying on webhooks.
[20:32:20] <-- benchMark (~mark@cpe-024-163-090-125.nc.res.rr.com) has quit
[20:32:58] <-- markstarkman (~markstark@c-68-81-220-27.hsd1.pa.comcast.net) has quit (Ping timeout: 256 seconds)
[20:33:59] <qrush> indirect: i think you should manually remove yanked ones
[20:34:08] <qrush> if you need me to give you a list i can do that
[20:34:12] <qrush> actualyl, fuck, i can't
[20:34:20] <qrush> i dont have ssh or any cert shit setup.
[20:34:30] <qrush> fuck i need to get setup :(
[20:35:13] <indirect> no, it's okay
[20:35:23] <indirect> looks like eventually consistent was eventually consistent
[20:35:23] <indirect> phew
[20:35:41] <qrush> ./play clowntown
[20:36:50] --> mr_ndrsn_ (~mrndrsn@74-130-209-106.dhcp.insightbb.com) has joined #rubygems
[20:37:11] <-- mr_ndrsn_ (~mrndrsn@74-130-209-106.dhcp.insightbb.com) has quit (Client Quit)
[20:37:53] <dwradcliffe> newrelic_rpm 3.5.6.46
[20:38:12] <dwradcliffe> net-ssh-multi 1.1.1
[20:38:21] <dwradcliffe> net-ssh-multi 1.1.2
[20:38:40] <indirect> VICTORY
[20:38:42] <indirect> # of gem versions added: 6
[20:38:43] <indirect> # of gem versions yanked: 889
[20:38:57] <dwradcliffe> indirect: nice!
[20:40:10] <-- onemanjujitsu (~sebastian@c-98-204-157-98.hsd1.dc.comcast.net) has quit (Quit: onemanjujitsu)
[20:43:10] <-- tenderlove (~tenderlov@pdpc/supporter/active/tenderlove) has quit (Remote host closed the connection)
[20:46:50] <qrush> yayaya
[20:46:58] <qrush> I'll tweet about that
[20:47:00] <qrush> i'm out for now