gistfile1.txt

Status #rubygemsX
[Status: @rubygems_status and http://status.rubygems.org | RubyGems 1.8.24 | http://rubygems.org/ | Idle for answers if it's not Pacific Time daytime]
[14:29] == Nilla_ [429948b1@gateway/web/freenode/ip.66.153.72.177] has joined #rubygems
[14:29] <mephux> wlll: it's a huge concern but once we do some forensics we'll know 100%
[14:29] == withloudhands [~robertwhi@rrcs-184-75-101-229.nyc.biz.rr.com] has joined #rubygems
[14:29] <Defiler> evan: I was thinking about adding a type whitelist to the YAML deserialization in rubygems; does that conflict with any of today's ongoing work?
[14:30] <benchMark> evan: Did you guys delete the exploit gem?
[14:30] <benchMark> I was expecting to see that in my grep.
[14:30] == nhocki_ [12bd2d9d@gateway/web/freenode/ip.18.189.45.157] has joined #rubygems
[14:30] <Defiler> Yeah qrush deleted it early on
[14:30] <benchMark> Oh.
[14:30] <Defiler> though it's here if you want to examine it https://gist.github.com/d891e876c53e55bf0920
[14:30] <envygeeks> did you guys at least back it up for forensics later?
[14:30] <benchMark> Nah, I just wanted to make sure the file list I got was complete.
[14:31] <evan> Defiler: tenderlove and I are working on that.
[14:31] <Defiler> OK
[14:31] <zzak> qrush: <3<3
[14:31] <evan> benchMark: I've got them.
[14:31] <evan> one sec.
[14:31] <benchMark> Based on the S3 file list qrush gave, the metadata suggests that the exploit wasn't used on any other gems.
[14:31] <Defiler> evan: let me know if you need more eyes or hands on anything in particular
[14:31] == kdaigle [~kyle@c-71-235-133-84.hsd1.ct.comcast.net] has joined #rubygems
[14:31] <danp_> benchMark: where's that list?
[14:32] == nhocki [12bd2d9d@gateway/web/freenode/ip.18.189.45.157] has quit [Ping timeout: 245 seconds]
[14:32] <benchMark> danp_: wget http://rubygems.org/all-gems.txt.gz
[14:32] == archie9 [43b40ab4@gateway/web/freenode/ip.67.180.10.180] has quit [Ping timeout: 245 seconds]
[14:32] == wfarr [~wfarr@foresight/developer/wfarr] has quit [Ping timeout: 245 seconds]
[14:33] == Nilla [429948b1@gateway/web/freenode/ip.66.153.72.177] has quit [Ping timeout: 245 seconds]
[14:33] <danp_> thx
[14:33] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has quit [Quit: Computer has gone to sleep.]
[14:33] <tenderlove> Defiler: https://gist.github.com/13c80114a2a23707cf19
[14:33] <@raggi> benchMark: was that a grep of everything?
[14:33] == emachnic [~emachnic@2602:306:241a:50c9:594f:c5ce:dc1b:8286] has joined #rubygems
[14:33] <benchMark> raggi: Yes.
[14:34] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has joined #rubygems
[14:34] <benchMark> I downloaded every single gem and extracted all the metadata to grep over.
[14:34] <danp_> cool
[14:34] <evan> i'm putting the expliot gems up now
[14:34] <evan> one sec.
[14:34] == tsykoduk [~tsykoduk@c-98-247-204-68.hsd1.wa.comcast.net] has joined #rubygems
[14:35] <@raggi> benchMark: ok, i'll leave mine running for a double check, but that looks good
[14:35] == beancuke [beancuke@2600:3c00::f03c:91ff:feae:a6d9] has joined #rubygems
[14:35] <@raggi> benchMark: we should get the same suspect lists :)
[14:35] <evan> https://www.dropbox.com/s/fi7r5oovqyrtacm/exploit-gems.zip
[14:35] == henrikhodne [henrikhodn@virgule.cluenet.org] has joined #rubygems
[14:35] == ddfreyne [~denisdefr@stoneship.org] has joined #rubygems
[14:36] == exploid [~Adium@CPEf81edff886c8-CM185933f80db9.cpe.net.cable.rogers.com] has left #rubygems []
[14:36] == exploid [~Adium@CPEf81edff886c8-CM185933f80db9.cpe.net.cable.rogers.com] has joined #rubygems
[14:36] == sferik [~textual@hattery.static.monkeybrains.net] has joined #rubygems
[14:36] <evan> so
[14:36] <evan> i'm downloading the S3 logs
[14:36] <@raggi> my top1m finished
[14:36] <@raggi> nothing in suspect list
[14:36] <evan> it's going to be a while
[14:36] <@raggi> :)
[14:36] <danp_> has anyone cross-ref'd S3 timestamps with expected data?
[14:36] == Radar [~Radar@li96-112.members.linode.com] has joined #rubygems
[14:36] <evan> danp_: working on it now
[14:37] <Defiler> tenderlove: Looks like what I had in mind; I was going to put the whitelist in the rubygems distro as an actual artifact and include it in the manifest and maybe add a task to the build that verifies it somehow
[14:37] <sikachu> wait, evan, so you actually manual scan those logs? :(
[14:37] == foul_owl [~kerry@2607:f700:0:fe:a95e:316d:a1db:b366] has quit [Quit: Leaving.]
[14:37] <benchMark> There were 105 URLs that I got errors for when I tried to download them.
[14:37] == exploid [~Adium@CPEf81edff886c8-CM185933f80db9.cpe.net.cable.rogers.com] has left #rubygems []
[14:37] <@raggi> benchMark: yeah, there's some index <-> file breakage
[14:38] <evan> sikachu: i'm just downloading them
[14:38] == exploid [~Adium@CPEf81edff886c8-CM185933f80db9.cpe.net.cable.rogers.com] has joined #rubygems
[14:38] <evan> so that I can write a script to match up PUT timestamps with database update times
[14:38] <@raggi> every time i run mirror, i mean to create tickets for them, i dont' have .org access to fix em :)
[14:38] == peregrine81 [~peregrine@108-231-126-61.lightspeed.milwwi.sbcglobal.net] has joined #rubygems
[14:38] <evan> raggi: well, let's fix that. :)
[14:38] <@raggi> evan: it wasn't useful until now - the last two years i've been too busy
[14:38] <evan> benchMark: those were probably perm yank'd gems
[14:38] <sikachu> I see, I thought you scan those by your eyes. that worried me a bit
[14:38] <sikachu> :)
[14:38] <benchMark> Missing files: https://gist.github.com/349d5e7952736b3ba861
[14:39] == bfleischer [~bfleische@corpo.mrskin.com] has quit [Remote host closed the connection]
[14:39] <@raggi> evan: now i have 20% time, and more free time generally :)
[14:39] <evan> sikachu: oh god no.
[14:39] <sikachu> good good
[14:39] <sikachu> lol
[14:39] == bfleischer [~bfleische@corpo.mrskin.com] has joined #rubygems
[14:40] <@raggi> sweet, load level 25. seems good.
[14:40] <wlll> raggi: How about this. Mark all current gems as 'tainted' on the rubygems server. New/signed gems pushed are not tainted. Get Yehuda to make bundler warn when installing tainted gems.
[14:40] <benchMark> I'm fairly confident the exploit wasn't used in any other gems, modulo the requirement that the gem URL list I got was complete and it wasn't used in one of the 105 missing files.
[14:40] <@raggi> wlll: we can't do that
[14:40] <kdaigle> Hey all. Happy to help however I can, just ping me to let me know.
[14:40] == kristopher [~kris@unaffiliated/kristopher] has joined #rubygems
[14:40] <@raggi> wlll: not without invalidating the current signing mechanisms at least (not that anyone except zenspider uses them)
[14:41] <@raggi> wlll: well, a few other hoe users i think
[14:41] <@raggi> wlll: but we can't modify signed gem packages
[14:41] <wlll> No, it'd be a flag in the Rubygems DB
[14:41] <@raggi> wlll: rubygems doesn't really work like that
[14:41] <evan> drbrain: are you validating your mirror against S3?
[14:41] <@raggi> wlll: bundler maybe could, but that whole area is already needs more work
[14:42] <@raggi> wlll: also, it doesnt' solve anything at all
[14:42] <benchMark> I can provide md5sums for all the gem files in my S3 download if helpful.
[14:42] <evan> benchMark: yes please
[14:42] <wlll> raggi: it doesn't? I could opt to not install tainted gems
[14:42] <@raggi> wlll: as not everything will get done, so it'll be so much warning spam, people will ignore it
[14:42] == j-v-e [~Julien_Ve@apo.aweber.com] has joined #rubygems
[14:42] <envygeeks> benchMark: md5's shouldn't be used for integrity
[14:42] <@raggi> wlll: people won't. they'll just ignore the message and install them
[14:42] <wlll> raggi: *I* won't :)
[14:42] <@raggi> wlll: we have tooo much heavily used stale stuff
[14:42] == justincampbell [~justincam@12.249.95.222] has joined #rubygems
[14:42] <wlll> people that care won't
[14:43] <benchMark> envygeeks: I'm well aware of the possibility for md5 collissions.
[14:43] <@raggi> wlll: practical reality, that's not what will happen. we've done this over and over again in this community, and it never works
[14:43] <j-v-e> howdy everyone. quick question: can a 'gem install' or 'gem update' succeed even if the connection to rubygems.org is blocked by a firewall ? with a local database or something similar ?
[14:43] <envygeeks> benchMark: it's not just possible, it's probable and easy to do :P
[14:43] <benchMark> I'm simultaneously pragmatic.
[14:43] <qrush> back
[14:43] <mephux> qrush: wb
[14:43] <j-v-e> we firewalled rubygems.org until confidence comes back
[14:43] <wlll> raggi: We've never had the gem server exploited
[14:44] <benchMark> Happy to do sha256sum so we can also find out that it's clean too.
[14:44] <Defiler> cloudfoundry operates a gem cache that is very paranoid about file hashes and such
[14:44] <Defiler> I don't have access to it anymore but that would be a nice cache to verify against
[14:44] <@raggi> wlll: no, but we've done this with security warnings in rack and rails, and we've had deprecation warnings of a similar style - they get ignored
[14:44] == wdperson [~wdperson@cpe-173-89-188-172.neo.res.rr.com] has joined #rubygems
[14:44] <@raggi> wlll: study some issue trackers, you'll see :)
[14:44] == jmadsen [~jmadsen@c-71-195-254-133.hsd1.ut.comcast.net] has joined #rubygems
[14:45] <bradland> evan: can you share any details about the data you're comparing the S3 timestamps against?
[14:45] == Andrew [4b656f82@gateway/web/freenode/ip.75.101.111.130] has joined #rubygems
[14:45] == Andrew [4b656f82@gateway/web/freenode/ip.75.101.111.130] has quit [Client Quit]
[14:45] <sikachu> http://kotaku.com/5980247/video-games-are-a-bigger-problem-than-guns-says-actual-us-senator
[14:45] == Andrew [4b656f82@gateway/web/freenode/ip.75.101.111.130] has joined #rubygems
[14:45] == Andrew has changed nick to Guest36578
[14:45] <wlll> raggi: we're talking about the possibility that some widely used gems have malicious code in them. Make people type "yes, I am aware that these gems are untrusted", then it's up to them.
[14:46] <evan> bradland: the plan was to take a pass over the logs and compare them with the DB values
[14:46] <qrush> had a terminal open and this came up while i was gone -
[14:46] <qrush> 255155-db1 kernel: EDAC MC0: UE row 0, channel-a= 1 channel-b= 2 labels "-": (Branch=0 DRAM-Bank=0 RDWR=Read RAS=895 CAS=0 FATAL Err=0x2 (Northbound CRC error on non-redundant retry))
[14:46] <qrush> wtf is that?
[14:46] <evan> thats not 100% definitely obviously, since the DB values could be tampered with
[14:46] <evan> qrush: thats me dumping the mem on the machine
[14:46] <z> qrush: faulty memory
[14:46] <@raggi> wlll: the other things we are doing will validate that
[14:46] <wlll> not to 100% as far as I can tell.
[14:46] <evan> and it seems to have triggered a memory fault
[14:46] <z> qrush: its an ECC correction
[14:46] <bradland> evan: it has the benefit of forcing the attacker to reconcile against the S3 logs though, which is more difficult to mask
[14:46] == brooks_1 [~brooks@SYS-MIZ-PC04.BOBST.NYU.EDU] has joined #rubygems
[14:46] <@raggi> qrush: it's caused by what evans doing
[14:46] <evan> bradland: exactly
[14:46] <@raggi> qrush: don't worry about it
[14:47] <@raggi> qrush: you'll probably see a ton more of them in dmesg
[14:47] == lemonodor [~jwiseman@adsl-76-214-11-187.dsl.lsan03.sbcglobal.net] has quit [Quit: lemonodor]
[14:47] <sikachu> whoops, wrong channel, sorry
[14:47] == jmadsen [~jmadsen@c-71-195-254-133.hsd1.ut.comcast.net] has quit [Client Quit]
[14:47] <Defiler> Here's the cloudfoundry code if anyone is curious https://github.com/cloudfoundry/vcap-staging/tree/master/lib/vcap/staging/plugin
[14:47] <Defiler> e.g. gem_cache.rb
[14:48] <wlll> Defiler: Do you know anyone at cloudfoundry available to help?
[14:48] <qrush> https://twitter.com/rubygems_status/status/296705350768406528 < this is great news
[14:48] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has quit [Quit: Computer has gone to sleep.]
[14:48] == jonathanwallace1 [~jonathanw@adsl-98-92-243-239.asm.bellsouth.net] has joined #rubygems
[14:48] <qrush> is the top 100 gems enough?
[14:48] <evan> benchMark: if you can post your checksums
[14:48] <qrush> should we do top 1000-10000?
[14:48] <evan> then we can get others to compare against them from mirrors
[14:49] <benchMark> evan: I'm running sha512sum n all of them.
[14:49] <evan> benchMark: great
[14:49] <benchMark> I'll have the output shortly.
[14:49] == jnimety [~jnimety@50-195-40-233-static.hfc.comcastbusiness.net] has left #rubygems ["Leaving..."]
[14:49] <evan> qrush: 1000 would be best.
[14:49] == atomgiant [~tdavies@166.205.48.8] has joined #rubygems
[14:49] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has joined #rubygems
[14:49] <@drbrain> benchMark: can you get me a sample of your output?
[14:49] <tmaher> benchMark - hi I'm from Heroku.  May I ask what you're using as the known-good copy of the hashes?
[14:49] == erichmenge [~erichmeng@unaffiliated/erichmenge] has joined #rubygems
[14:50] <qrush> evan: ok, i can do that
[14:50] <evan> tmaher: we're compiling it various mirrors
[14:50] <benchMark> tmaher: I got a list of URLs for every gem on S3 from qrush.
[14:50] <evan> that were created in the past
[14:50] <@raggi> my latest script also finished, and looking good
[14:50] <benchMark> tmaher: Also, Hi. This is Mark Imbriaco. ;-)
[14:50] <tmaher> Oh, hi Mark !
[14:50] == wfarr [~wfarr@99.sub-70-199-83.myvzw.com] has joined #rubygems
[14:50] <evan> once mark's checksum are up
[14:50] <envygeeks> I'm a bit confused at why you guys are only doing it for the most popular gems when logic says that an attacker is gonna strategize which means he's gonna pick gems that play to his goal IMO, which means he could very well pick a gem that isn't popular just because it targets who he wants, so why not take the time to do it to all gems?
[14:50] <evan> someone needs to write a script to hash a gem and check it against his list
[14:50] <benchMark> drbrain: https://gist.github.com/5d15fde7c50bf2446af1
[14:51] <evan> envygeeks: we're going to do them all
[14:51] <evan> envygeeks: but we have to start somewhere
[14:51] <qrush> envygeeks: we are going to verify them all...trying to do this one step at a time
[14:51] <@drbrain> benchMark: thanks, I can generate the same from my two month old mirror
[14:51] <benchMark> envygeeks: I grepped for the payload across all 299k gems and am generating sha512sum for all of them as well.
[14:51] <benchMark> drbrain: Awesome. I'll post my full output somewhere as soon as it finishes up.
[14:51] <danp_> tmaher: I believe drbrain has a separate copy of gem repo which he is also checksumming
[14:52] <danp_> to compare against what benchMark has put together
[14:52] <@drbrain> benchMark: it'll probably take me over an hour to get a list, I'm standing up a cluster to check it
[14:52] == lemonodor [~jwiseman@cpe-75-83-152-209.socal.res.rr.com] has joined #rubygems
[14:52] == xternal [~xternal@ip68-5-230-247.oc.oc.cox.net] has joined #rubygems
[14:52] <benchMark> drbrain: Sure thing. FWIW, a box with SSDs churns through all 299k gems shockingly fast.
[14:52] <evan> we can use benchMark's hash set as a baseline
[14:52] <benchMark> I'm using a hi1.4xlarge instance on EC2 and it's ripping through them.
[14:52] <mastahyeti> evan: mark: I'm thinking of using the version tags from github as a known good source. thoughts?
[14:52] == cktricky [45ffb314@gateway/web/freenode/ip.69.255.179.20] has joined #rubygems
[14:52] <evan> and then we can begin to validate any gems found against it
[14:53] <@drbrain> benchMark: my mirror is on the AT&T cloud
[14:53] <evan> mastahyeti: for various gems? sure.
[14:53] <qrush> I still have my original Gemcutter mac mini i can boot up with the original rubyforge gems :P
[14:53] <@drbrain> so I'll need to fetch them from the S3 equivalent before I can md5sum
[14:53] <benchMark> Gotcha.
[14:53] <@drbrain> err, sha512
[14:53] <evan> with benchMark's set public
[14:53] <qrush> actually wouldnt the gems be on rubyforge still? tcopeland evan ?
[14:53] <evan> we'll be able to ask gem authors to validate too
[14:53] <benchMark> I did that on the hi1.4xlarge as well. Recommend using puf with a lot of parallelism. :)
[14:53] <qrush> if we want to verify old ones?
[14:53] <@raggi> qrush: yes, they are
[14:54] <evan> qrush: you mean the actual .gem files?
[14:54] <@raggi> qrush: we actually still push rack there too, dunno how many others do
[14:54] * raggi considers rsyncing
[14:54] <tcopeland> some of them are in /var/www/gforge-files on rubyforge
[14:55] == revans [~revans@99-132-72-44.lightspeed.sndgca.sbcglobal.net] has joined #rubygems
[14:55] <evan> ok
[14:55] == listrophy [~listrophy@bendyworkers.com] has joined #rubygems
[14:55] <evan> we'll validate against those too
[14:55] <danp_> might be interesting to think about at least capturing checksum data at gem push time
[14:55] <evan> basically, we want to validate any and all .gem files
[14:55] <tcopeland> looks like I deleted /var/www/gems when I shut that down.  But they are still all on the bytemark mirror
[14:55] <evan> against benchMark's set
[14:55] <@raggi> rsyncing
[14:55] <tcopeland> # hostname && ls -l /var/www/gems/ | wc -l
[14:55] <tcopeland> rubyforge.vm.bytemark.co.uk
[14:55] <tcopeland> 97620
[14:55] <evan> to try to cover as much ground as we can
[14:55] <evan> tcopeland: is that staging?
[14:55] <z> tcopeland: that's about 200,000 short.
[14:56] == franckverrot_ [~cesario@ks210523.kimsufi.com] has joined #rubygems
[14:56] == fbernier [~openwrt@24.48.120.70] has joined #rubygems
[14:56] <evan> z: can you coordinate with benchMark to verify your mirror too?
[14:56] <qrush> z lol
[14:56] <tcopeland> yup, those are only the ones that had been sync'd to the bytemark mirror of rubyforge
[14:56] <danp_> benchMark: puf?
[14:56] <benchMark> danp_: puf is a parallel wget
[14:56] <benchMark> Basically.
[14:56] <benchMark> That's what I used to get all the gems from Cloudfront.
[14:57] <evan> I have to go get lunch
[14:57] <danp_> ah, I'm just using wget -i and it's almost done
[14:57] <evan> my low bloodsugar is preventing me from thinking clearly.
[14:57] == borski [~borski@c-67-180-18-116.hsd1.ca.comcast.net] has joined #rubygems
[14:57] <benchMark> Yeah, I think it took about 10 minutes with puf to get all 300k on a hi1.4xlarge.
[14:57] <evan> those of you that are verifying against benchMark's set
[14:57] <evan> please keep us updated
[14:57] <benchMark> I haven't published my SHAs yet.
[14:57] <borski> Hey guys - what can I do to help?
[14:57] <benchMark> Just over halfway done.
[14:57] <evan> yep
[14:57] <raz> as a reminder (you probably have done it already): don't forget to check what other credentials (ssh keys, web SSL private key, rails secret etc.) the intruder may have obtained from the host
[14:58] <evan> raz: yep, we've done that.
[14:58] <raz> alright :)
[14:58] <evan> ok, i'll be back in about an hour
[14:58] <revans> Hi all, anything I can do to help?
[14:58] == nhocki_ [12bd2d9d@gateway/web/freenode/ip.18.189.45.157] has quit [Quit: Page closed]
[14:58] <evan> drbrain: can I leave you in charge to coordinate validating checksums?
[14:58] <@drbrain> evan: sure
[14:58] <evan> thanks.
[14:58] <evan> thats our next step everyone
[14:59] <evan> validating as many gems as we can once benchMark's set is up
[14:59] <evan> so whatever scripts you can put together to do that
[14:59] <benchMark> ETA just a couple minutes.
[14:59] <evan> would be great
[14:59] == Guest36578 [4b656f82@gateway/web/freenode/ip.75.101.111.130] has quit [Quit: Page closed]
[14:59] <qrush> " Yeah, I think it took about 10 minutes with puf to get all 300k on a hi1.4xlarge." < can you please document this somehow? We could totally allow more mirrors this way. i thought mirrors would take days.
[14:59] <evan> danp_: does heroku have a master bundle cache?
[14:59] == beancuke [beancuke@2600:3c00::f03c:91ff:feae:a6d9] has left #rubygems []
[14:59] == mitchellh [mitchellh@ec2-23-21-172-67.compute-1.amazonaws.com] has joined #rubygems
[15:00] <borski> benchMark: what's the structure of what youre gonna spit out? maybe i can get started on a script before its out :)
[15:00] <benchMark> qrush: puf -lc 600 -i url_list.txt
[15:00] <tmaher> evan - no - each app has its own individual cache.
[15:00] <benchMark> 600 simultaneous connections, saving to an SSD
[15:00] <evan> borski: sample: https://gist.github.com/5d15fde7c50bf2446af1
[15:00] <@raggi> qrush: just using xargs -P pulled them down in 30 minutes
[15:00] <tmaher> We're currently about to switch to doing local-cache-only deploys
[15:00] == jm___ [44329df6@gateway/web/freenode/ip.68.50.157.246] has joined #rubygems
[15:00] <evan> tmaher: ok. is it possible to run anything against as many of those caches as possible?
[15:00] <danp_> evan: no
[15:01] <danp_> evan: hmm, maybe
[15:01] <evan> basically, if we could use heroku as a picture in top gems
[15:01] <tmaher> evan - like, over checksums?
[15:01] <danp_> (no was for the first question)
[15:01] <evan> you can validate against benchMark's checksums
[15:01] == jcran [~jcran@metasploit/jcran] has joined #rubygems
[15:01] <evan> to get a level of confidence
[15:01] <evan> that would work mostly because a lot of apps will not have deployed in a while
[15:01] == cschneid [cschneid@2600:3c00::f03c:91ff:fedf:d92e] has joined #rubygems
[15:01] <qrush> wlll's concerns sound valid, what can we do beyond hashing to really trust everything again?
[15:01] <evan> and thus if there is anything tampered with
[15:02] <evan> qrush: i'm thinking about it.
[15:02] <qrush> marking *every* gem as tainted sounds kind of crazy to do
[15:02] <tmaher> We'd limit it down to just apps deployed in the last 24 hours, I suspect.
[15:02] == rubysolo [~rubysolo@c-67-190-173-72.hsd1.co.comcast.net] has joined #rubygems
[15:02] == brixen [~brixen@li253-247.members.linode.com] has joined #rubygems
[15:02] <evan> tmaher: that wouldn't tell us much
[15:02] <evan> tmaher: because those would have downloaded the same files benchMark just hashed
[15:02] <tmaher> evan: Oh, you're thinking use us as a 2nd cache.
[15:02] <evan> it would have to be older apps
[15:02] <tmaher> err, 2nd mirror
[15:02] <evan> right
[15:02] <evan> qrush: let me think about how to do further checking
[15:02] <tmaher> I think we can do that.  Stay tuned
[15:02] <evan> i'll be back in a little bit
[15:03] <evan> thanks again everyone for all the help.
[15:03] == jordanh [~jordan@94.197.46.151.threembb.co.uk] has joined #rubygems
[15:03] <wlll> Thanks evan
[15:03] == shockz [~shockz@zenofex.com] has joined #rubygems
[15:03] <benchMark> My pragmatic side thinks that it's vanishingly unlikely that this exploit was used maliciously based on everything we know right now. We can continue to try to get assurance based on checksums, etc. That's not to say there haven't been other exploits along other vectors, though.
[15:03] <danp_> evan: we are not sure if the actual .gem files stay around
[15:04] <qrush> top 1000 gems - https://gist.github.com/4676363
[15:04] == jordanh [~jordan@94.197.46.151.threembb.co.uk] has quit [Read error: Connection reset by peer]
[15:04] <wlll> benchMark: I agree it's unlikely, I may be being paranoid
[15:04] <qrush> URLs coming soonly
[15:05] == jordanh [~jordan@94.197.46.151.threembb.co.uk] has joined #rubygems
[15:06] <bradland> i would put some effort in to comparing the S3 last modified timestamps with database data indicating when a gem was last published.
[15:06] == mrtrick37 [~mrtrick@63.151.200.65] has joined #rubygems
[15:07] == nzkoz [~michael@203.171.56.174] has joined #rubygems
[15:07] <benchMark> Agree.
[15:07] == mletterle [~michael@ec2-23-22-221-78.compute-1.amazonaws.com] has joined #rubygems
[15:07] <bradland> you could use a datetime window then; limiting analysis to files that were modified only recently.
[15:07] <qrush> wlll:  ^^^ how about that?
[15:07] <danp_> I think evan is hoping to do that with S3 logs
[15:07] <sikachu> bradland: i thought evan already did that.
[15:07] <qrush> is there an easy way to get massive amounts of last-modified times?
[15:08] <qrush> i can do that with the top 100,1000 gems
[15:08] <qrush> via fog
[15:08] <wlll> qrush: that was what I thought the plan was.
[15:08] <danp_> not really. just HEAD the URLs
[15:08] <qrush> wondering if there's a way beyond that.
[15:08] <benchMark> Checksums: http://cl.ly/MYie
[15:08] <bradland> he's looking at logs, but i'm not sure what exactly he's doing with them
[15:08] <benchMark> The downloaded gz file should have the following sha512 checksum:
[15:08] <benchMark> 2c884bdcf364524d055a17b2403f10adca2f9465cb21e4c3e562b808ebe5b4ccb6e852c53f51f9e6418ec3d5781c827e07527bf8ecba08ec65f596186c769d42  rubygems-shas.txt.gz
[15:08] <sikachu> bradland: he was matching PUT time with the timestamp of the gem on S3
[15:08] <envygeeks> time to fireup some magically em-synchrony + faraday magic and hit S3 at 300 req a second!
[15:08] <sikachu> trying*
[15:09] <qrush> here's the URLs for top 1000 gems http://rubygems.org/top-1000.txt
[15:09] <raz> no wonder s3 is slow again, with everyone comparing rubygems :P
[15:09] <bradland> that should be extremely reliable then
[15:09] <sikachu> qrush: does Rubygems have some kind of flag that will block the downloads gem by gem?
[15:10] <@raggi> envygeeks: or just use xargs -P :-P
[15:10] <qrush> sikachu: we can individually pull gems from the index. why?
[15:10] <sikachu> so like, we can set a flag on every gem to disable its download. then enable them one by one after we check the checksum against the log
[15:10] <sikachu> or last-modified-date
[15:10] == tmaher [~Adium@c-50-136-136-212.hsd1.ca.comcast.net] has quit [Ping timeout: 276 seconds]
[15:10] == blacktip [~walsh@unaffiliated/blacktip] has joined #rubygems
[15:11] <sikachu> that way some of the top gems will be able to be downloaded first, while we have a script running to test the rest of the gems in the background
[15:11] <johnmwilliams_> Is there any reason to not publish a dump of the DB table so people can start comparing last-modified?
[15:11] == matthavener [~matt@cppreference.with-linux.com] has joined #rubygems
[15:12] == tmaher [~Adium@c-50-136-136-212.hsd1.ca.comcast.net] has joined #rubygems
[15:12] == yerhot [~yerhot@c-71-63-228-30.hsd1.mn.comcast.net] has joined #rubygems
[15:12] == jm_ [~jm_@rrcs-97-79-33-116.se.biz.rr.com] has quit [Quit: jm_]
[15:12] == cktricky [45ffb314@gateway/web/freenode/ip.69.255.179.20] has quit [Quit: Page closed]
[15:12] <qrush> johnmwilliams_:  we dont store that in the db
[15:13] <qrush> i mean, there's Version#created_at ...that wont match exactly though ( i think )
[15:13] <sikachu> does that make sense?
[15:13] <bradland> qrush: are the S3 uploads handled asynchronously? otherwise, that should be relatively close (seconds).
[15:13] == lianj [~lianj@subtle/user/lianj] has joined #rubygems
[15:13] <qrush> No, they're synchronous.
[15:14] == zdennis [~zdennis@199.116.52.94] has quit [Ping timeout: 256 seconds]
[15:14] <bradland> close enough to sniff test
[15:14] <danp_> evan: we may have cached .gem files, digging now
[15:15] == zdennis [~zdennis@199.116.52.94] has joined #rubygems
[15:15] == jjarmoc [~jjarmoc@ec2-107-21-146-162.compute-1.amazonaws.com] has joined #rubygems
[15:15] <qrush> bradland: johnmwilliams_ i can publish that i guess. Would a db dump of versions table be good, or should i just be [full_name, updated_at] ?
[15:15] <qrush> maybe just  csv?
[15:15] == JohnHirbour [~Adium@96.56.42.186] has joined #rubygems
[15:15] <bradland> i'd provide the minimum data required to do the analysis
[15:16] == cktricky [45ffb314@gateway/web/freenode/ip.69.255.179.20] has joined #rubygems
[15:16] <bradland> i don't know the schema, so i can't say what to exclude
[15:16] <wlll> Yeah, I don't want my credit card details going public too.
[15:16] <bradland> Version#cc_md5_hash... what's this?
[15:16] <bradland> :)
[15:16] == cianuro [b51cf6be@gateway/web/freenode/ip.181.28.246.190] has joined #rubygems
[15:16] <johnmwilliams_> qrush: then again with db access the updated_at is not 100% vaild if someone had DB access.
[15:17] == kristopher [~kris@unaffiliated/kristopher] has quit [Ping timeout: 248 seconds]
[15:17] <qrush> ...
[15:17] <bradland> johnmwilliams_: it tightens the window
[15:17] <johnmwilliams_> bradland: +1
[15:17] <bradland> we need the S3 last modified dates, because they are immutable
[15:17] <bradland> there is no S3 APi for updating them
[15:18] <kdaigle> bradland: +1
[15:18] <qrush> lets do one thing at a time here.
[15:18] <qrush> @_@
[15:18] <sikachu> yeah, that's why i'm for what's evan's doing, which is checking PUT log with the S3 timestamp.
[15:18] <qrush> let me get a csv of full_name, updated_at first.
[15:18] <sikachu> o u confused.
[15:19] <nzkoz> qrush: there's too many people here, you'll never get anything done.  You need fewer people working more carefully and documenting everything they're doing
[15:19] <sikachu> :)
[15:19] <nzkoz> tomorrow you'll have to write up everything that happened and how you know that things are fine, and you can't do that with everyone chipping in :/
[15:19] <qrush> nzkoz: evan's in charge...i'm just trying to help
[15:20] == tenderlove [~tenderlov@pdpc/supporter/active/tenderlove] has quit [Remote host closed the connection]
[15:20] == kurtisnelson [~kurtisnel@206.165.129.38] has joined #rubygems
[15:20] <qrush> i pulled people in to help with checksums, etc since i figured more can't hurt.
[15:20] == themcgruff [~Adium@pool-173-78-15-66.tampfl.fios.verizon.net] has joined #rubygems
[15:20] == beancuke [beancuke@2600:3c00::f03c:91ff:feae:a6d9] has joined #rubygems
[15:21] <nzkoz> if you have the s3 logs, can verify PUTs are included, and know when the compromise happened, you can answer the question of whether they've compromised anything.  Then double check by checking the last-modified of *everything*
[15:21] == kristopher [~kris@177.35.115.86] has joined #rubygems
[15:21] <bradland> nzkoz: that's exactly what i'm thinking. that would give us a high level of confidence that the gem data hasn't been tampered with.
[15:21] == stepheneb [~stepheneb@otrunk/stepheneb] has joined #rubygems
[15:22] <benchMark> The most interesting thing in the S3 logs right now is DELETEs.
[15:22] <raz> nzkoz: they don't know when the compromise happened
[15:22] <qrush> nzkoz: ok
[15:22] <benchMark> Because those are gems that I wasn't able to inspect manifests for.
[15:22] <qrush> benchMark: DELETEs are all me - only i can permadelete gems.
[15:22] <qrush> well anyone with ssh access could i guess
[15:22] <benchMark> They could be an attacker too.
[15:22] <benchMark> Right.
[15:22] <benchMark> That's the point. :)
[15:22] <nzkoz> qrush: if I had shell on that box, I can do all of that
[15:23] == dkubb [~dan@d173-183-66-240.bchsia.telus.net] has joined #rubygems
[15:23] <tcopeland> you bet
[15:23] <qrush> Apparently there's multi-factor auth for S3 - http://aws.amazon.com/s3/faqs/#How_can_I_ensure_maximum_protection_of_my_preserved_versions
[15:23] <qrush> we need to do this
[15:24] == d-rock_ [~drock@206-248-184-116.dsl.teksavvy.com] has joined #rubygems
[15:24] <sikachu> definitely
[15:24] <bradland> that'd be hot
[15:24] <benchMark> Yeah, and never delete another object.
[15:24] <benchMark> Just change the permissions.
[15:24] == jreading [~Adium@204.56.125.50] has left #rubygems []
[15:24] <nzkoz> you've disabled the old s3 key set and issued new ones and only stored those on brand new hardware right?
[15:24] == wbruce [~wbruce@host86-139-214-86.range86-139.btcentralplus.com] has joined #rubygems
[15:24] <qrush> nzkoz: key was reset, and i replaced it
[15:25] <bradland> hardware is the same though, right?
[15:25] <qrush> ERROR:  must be superuser to COPY to or from a file < Lame, can't make a csv export on pg...
[15:25] <qrush> bradland: yes
[15:25] == DanR_ [3feb76f6@gateway/web/freenode/ip.63.235.118.246] has joined #rubygems
[15:25] <nzkoz> qrush: pointless
[15:25] <bradland> there's still the matter of whether there is an active intruder
[15:25] <bradland> evan did a memory dump that i believe he was handing off to a security guy
[15:25] <nzkoz> that hardware is dead, you have to disconnect it from the internet, start over
[15:26] <raz> nzkoz++
[15:26] <qrush> Should I revoke the new S3 key I made? I don't know if evan is using it to suck down logs.
[15:26] <qrush> this shit is really overwhelming/stressful right now
[15:26] <nzkoz> qrush: start a small channel with just you and evan, invite helpful people from there on.  You need to try to relax, and be calm and careful
[15:26] <raz> keep it slow, nobody is blaming anyone
[15:27] <nzkoz> the worst thing to do with these things is rush, because that's when you fuck up
[15:27] == wfarr [~wfarr@99.sub-70-199-83.myvzw.com] has quit [Ping timeout: 244 seconds]
[15:27] <titanous> everyone can wait, take the time to do it right
[15:27] <bradland> qrush: keep in mind the things you can and cannot change. i never stress over things that i cannot change.
[15:27] == j-v-e [~Julien_Ve@apo.aweber.com] has left #rubygems []
[15:27] <the_mentat> brad land Do you know who the security guy looking at the memdump is?
[15:28] <raz> i think forensics (mem dumps et al) is really the least interesting thing to push on right now
[15:28] <bradland> the_mentat: lemme open my transcripts
[15:28] <qrush> i'm just going to wait until evan gets back. i'm not sure where he is with stuff. i was not told resetting keys on the existing box was a problem, nor was I aware it *could* be a problem
[15:28] == erbmicha [d0442605@gateway/web/freenode/ip.208.68.38.5] has joined #rubygems
[15:29] <bradland> qrush: it all comes back to the same issue: once someone runs code on the box, you can't trust it. that's the root of the problem
[15:29] <bradland> if you change the keys every five minutes, but someone is looking over your shoulder, it's all for naught
[15:29] <nzkoz> you've shut down gem downloads right?
[15:29] <bradland> as is the analysis occurring right now
[15:29] <wlll> nzkoz: no, they're still up
[15:29] <qrush> we have not shut down *anything*
[15:29] <z> nzkoz: nope, only uploads
[15:29] <qrush> beyond gem pushes
[15:30] == wfarr [~wfarr@204-16-157-26-static.ipnetworksinc.net] has joined #rubygems
[15:30] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has quit [Quit: Computer has gone to sleep.]
[15:30] <raz> what i'd like to see is the md5-validation finished (as good as possible) and rubygems come up on a freshly installed server, with gem/bundler enforcing signed gems by default. that way gem authors are encouraged to re-push signed versions of their gems ASAP, and users can still choose to install old versions by overriding -P.
[15:30] <raz> (my 2 cents)
[15:30] <@raggi> sorry for the delay
[15:30] <@raggi> i added parallel sha checking to my gist https://gist.github.com/4a4e005b575d71c92a24
[15:31] <@raggi> you may need to adjust paths for your mirror layout
[15:31] == erbmicha [d0442605@gateway/web/freenode/ip.208.68.38.5] has left #rubygems []
[15:31] <joealba> qrush: You're doing fine, and you're not alone. You've got great help here.  Changing keys was the right thing to do in case the old keys were compromised.  You'll just want new keys on the new hardware too for longer-term trust.
[15:31] <bradland> the_mentat: i think it was mephux. i can't really tell from my transcript.
[15:31] <chort0> raz: but memory dump has to be done before the info is lost forever. maybe doesn't need to be *analyzed* immediately, but a dump needs to be taken and saved off-line to removed storage
[15:31] <the_mentat> bradland makes sense, thx
[15:31] <mephux> whats up?
[15:31] <raz> chort0: sure, it may be academically interesting. it doesn't help to get rubygems running again (and in a more secure fashion).
[15:32] <chort0> uhh, you're missing the point big time
[15:32] <jcran> mephux: where did you source those 6 gems you linked on dropbox?
[15:32] <bradland> mephux: do you know if evan ever got the memory dump in the hands of HIDS expert?
[15:32] <chort0> there's no way to know what the attacker did without taking the memory dump immediately
[15:32] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has joined #rubygems
[15:32] <chort0> you can come back later and validate gems, you can't come back later and dump memory, because it's gone
[15:32] <mephux> bradland: not that i know of.
[15:32] <mephux> jcran: here
[15:32] <bradland> ok, looks like only evan can answer the memory dump question. we'll have to wait for him.
[15:32] <nzkoz> qrush: s3's etags are md5s, so you can do one bulk crawl fetching all the last-modified and md5s for every single gem, someone should be doing that, and then wait on evan for the rest
[15:33] <nzkoz> qrush: you should also be starting over with brand new hardware, new AMIs (assuming yours were custom built)
[15:33] <@raggi> nzkoz: we're running sha checks
[15:33] <raz> chort0: it's not very interesting what the attacker did. the box is compromised and gems _may_ be compromised. it's unlikely the mem-dump will provide useful information about that - and even if it does there's no way to know whether that info is complete.
[15:33] <nzkoz> raggi: but why bother, s3 has done that for you
[15:33] <qrush> benchMark: can puf do that ^ ?
[15:33] <bradland> nzkoz: server is not a VM (it's bare metal)
[15:33] <nzkoz> md5 vs sha is academic at this point
[15:33] <@raggi> nzkoz: s3 is the one that's pwnt
[15:33] <@raggi> nzkoz: i agree, benchMark chose sha512
[15:33] <@raggi> but meh
[15:33] <chort0> raz: you can't prove the gems are safe if you don't know what the attacker did. if they had local access they could tamper with anything
[15:33] <@raggi> doesn't matter
[15:33] <@raggi> it's only 300k files
[15:34] <benchMark> I chose sha512 because I didn't feel like arguing about whether md5 was good enough.
[15:34] <nzkoz> raggi: it does matter, you only have to md5sum *trusted* files, then compare against s3 without refreshing
[15:34] <chort0> you also can't prove attacker doesn't have continuing access to s3 if you don't know if they dropped a rootkit, had a reverse shell open, etc
[15:34] <benchMark> So I went in the extreme other direction.
[15:34] <nzkoz> but whatever, so long as that's happening
[15:34] <@raggi> chort0: why don't you write up all the possible vectors you can think of in a gist
[15:34] <benchMark> I can happily md5sum them all.
[15:34] <@raggi> chort0: and then write up checks against them alongside
[15:34] <@raggi> chort0: and we can check them off
[15:34] <benchMark> I'm not sure we have a set of trusted files though.
[15:34] <@raggi> instead of dropping it all here
[15:34] <raz> chort0: exactly. that's why they compare gems to assumed-good versions, and will hopefully relaunch with all gems marked as "tainted" (by defaulting to requiring signed gems).
[15:34] <chort0> i'm not even talking about the checks right now, i'm talking about CAPTURING the evidence
[15:34] <chort0> if you trample all over the crime scene, you'll never get the evidence
[15:35] <chort0> can't discuss what to look for until you have the evidence captured
[15:35] <danp_> we are working on coming up with a checksum corpus based on pushed apps
[15:35] <@raggi> chort0: evan started a memory dump, but this is all theoretical at this point,s o please write it all
[15:35] <mephux> raz: chort0 is 100%
[15:35] <mephux> right*
[15:35] == nootch [48e1afa9@gateway/web/freenode/ip.72.225.175.169] has joined #rubygems
[15:35] <chort0> it's very simple: 1) memory 2) disk
[15:35] <raz> chort0: it is just not very interesting. it may be interesting for a post-mortem, but not for recovery.
[15:35] <@drbrain> benchMark: great, I can't launch images on our cloud today
[15:35] <mephux> raz: ??
[15:35] <chort0> IF YOU HAVE A ROOTKIT IT DOES MATTER
[15:35] <chort0> how thick are you???
[15:35] == erbmicha [d0442605@gateway/web/freenode/ip.208.68.38.5] has joined #rubygems
[15:35] <mephux> raz: prove to me that the attacker didn't own every gem?
[15:35] == markstarkman [~markstark@c-68-81-220-27.hsd1.pa.comcast.net] has joined #rubygems
[15:35] <nzkoz> ah the internet
[15:35] <benchMark> drbrain: Can I fetch the gems from your S3 equiv or is it private?
[15:35] == andrewhubbs [4b656f82@gateway/web/freenode/ip.75.101.111.130] has joined #rubygems
[15:35] <mephux> raz: prove to me that they don't still have access?
[15:35] == meise_ [~dm@3st.be] has joined #rubygems
[15:36] <mephux> raz: you CAN'T
[15:36] <raz> oh my..  we should perhaps take this to a different channel (noise here)
[15:36] <qrush> chort0: mephux please, stop
[15:36] <mephux> which means no one will ever trust you believe you.
[15:36] <benchMark> Good grief.
[15:36] <mephux> we are trying to help you save face.
[15:36] <chort0> raz: you don't know what you're talking about. mephux and I do. we do this all day.
[15:36] <raz> mephux: the point is that you can't prove *either*, mem-dump or not
[15:36] == indirect [~indirect@c-98-248-247-227.hsd1.ca.comcast.net] has left #rubygems []
[15:36] <raz> mephux: the box is compromised - period.
[15:36] <mephux> raz: YES YOU CAN.. you're loony
[15:36] <erbmicha> how do i remove the http://rubygems.org source from the environment?
[15:36] <@drbrain> mephux: please stop
[15:36] <nootch> here to help ....
[15:37] <mephux> drbrain: stop what?
[15:37] <mephux> helping you..
[15:37] <@raggi> feeding
[15:37] <nootch> if that means not being a cook in the kitchen i can leave too
[15:37] <mephux> guys, please to make the security world hate you but not giving a fuck.
[15:37] == exploid [~Adium@CPEf81edff886c8-CM185933f80db9.cpe.net.cable.rogers.com] has quit [Quit: Leaving.]
[15:37] <mephux> by*
[15:37] <raz> mephux: i'm in #rmeta if you want to talk this out, this is not the place
[15:38] <mephux> raz: roger
[15:38] == bradrub [4c0e428c@gateway/web/freenode/ip.76.14.66.140] has joined #rubygems
[15:38] == devn [~devn@rot13.pbqr.org] has joined #rubygems
[15:38] <devn> <3
[15:38] <kristopher> "ah the internet" <3
[15:38] * devn sends super love vibes
[15:39] == hdm [~hdm@about/security/staff/hdm] has joined #rubygems
[15:39] <@raggi> benchMark: could you generate a list of md5sums?
[15:39] * nootch offers early Wednesday hugathon
[15:39] <@raggi> benchMark: save me pulling a fresh mirror to validate s3
[15:39] == beancuke [beancuke@2600:3c00::f03c:91ff:feae:a6d9] has left #rubygems []
[15:39] <@raggi> brb, fooding.
[15:39] == brooks_1 [~brooks@SYS-MIZ-PC04.BOBST.NYU.EDU] has left #rubygems []
[15:39] <benchMark> raggi: Happy to.
[15:39] <@raggi> thanks
[15:40] <@raggi> msh / HL me, i'm just going to grab a salad
[15:40] == kristopher [~kris@177.35.115.86] has quit [Changing host]
[15:40] == kristopher [~kris@unaffiliated/kristopher] has joined #rubygems
[15:40] <benchMark> Will do.
[15:40] <benchMark> Should be done in about 5 minutes.
[15:40] == kristopher [~kris@unaffiliated/kristopher] has quit [Quit: Leaving.]
[15:41] == kristopher [~kris@unaffiliated/kristopher] has joined #rubygems
[15:42] == cowboyd [~cowboyd@12.237.107.68] has joined #rubygems
[15:43] == hgmnz [~hgmnz@173.247.206.130] has quit [Remote host closed the connection]
[15:43] == derekprior [~textual@72.246.0.14] has joined #rubygems
[15:44] == tmaher [~Adium@c-50-136-136-212.hsd1.ca.comcast.net] has quit [Quit: Leaving.]
[15:44] == nootch_ [~nootch@cpe-72-225-175-169.nyc.res.rr.com] has joined #rubygems
[15:44] == boffbows1 [~boffbowsh@phobos.fatboylan.co.uk] has joined #rubygems
[15:45] == mattetti [6379faf9@gateway/web/freenode/ip.99.121.250.249] has joined #rubygems
[15:45] <breakingthings> Dear qrush, I just wish to impart some internet love upon thee for your hard times this day.
[15:45] <breakingthings> <3.jpeg
[15:45] == nootch [48e1afa9@gateway/web/freenode/ip.72.225.175.169] has quit [Ping timeout: 245 seconds]
[15:45] == khaase [~khaase@sinatra/rkh] has joined #rubygems
[15:46] == nakajima [~nakajima@cpe-68-173-50-36.nyc.res.rr.com] has joined #rubygems
[15:46] == boffbows1 has changed nick to boffbowsh
[15:47] == technomancy [~technoman@ec2-50-16-104-233.compute-1.amazonaws.com] has joined #rubygems
[15:47] == konopka_phone [~konopkaph@c-68-81-76-220.hsd1.pa.comcast.net] has joined #rubygems
[15:48] <@raggi> heh, haven't seen half you folks on irc in the last few years
[15:48] <devn> :)
[15:48] == jordanh [~jordan@94.197.46.151.threembb.co.uk] has quit [Read error: Connection reset by peer]
[15:49] <qrush> that is not an invitation to publicly post more exploits :P
[15:49] == vrillusions [~vr@rikku.vrillusions.com] has joined #rubygems
[15:49] == jordanh [~jordan@188.28.172.52.threembb.co.uk] has joined #rubygems
[15:49] == jordanh [~jordan@188.28.172.52.threembb.co.uk] has quit [Client Quit]
[15:49] <chort0> https://gist.github.com/4676817
[15:49] <chort0> there
[15:49] == kernelsmith [~kernelsmi@metasploit/kernelsmith] has joined #rubygems
[15:50] <chort0> if you were handling the evidence for prosecution of a crime, there would be more steps. i assume you're not going for prosecution in this case
[15:51] == pea53 [~Adium@c-98-245-87-57.hsd1.co.comcast.net] has joined #rubygems
[15:51] <chort0> if you are, then you need to take checksums of the images and label all physical devices with initials of who handled them. you'd also keep a chain of custody log for everyone who handled it and what time/date they did so
[15:52] == james__ [~james@188-222-159-187.zone13.bethere.co.uk] has joined #rubygems
[15:52] == cgcardona [~cgcardona@unaffiliated/cgcardona] has joined #rubygems
[15:52] <chort0> and just stating the obvious, you want to be careful about who handles the memory image, because it almost certainly has authentication tokens in it
[15:53] == hgmnz [~hgmnz@204.14.152.118] has joined #rubygems
[15:54] <postmodern> qrush, thanks for spear heading the review
[15:54] <qrush> chort0: nice, but i'm not sure what here is applicable. it's pretty obvious we need to reset the box at this point
[15:55] <benchMark> Need to replace the box, really.
[15:55] <raz> ideally you'd re-install to a fresh set of boxes (unrelated to the ones you have now)
[15:55] <qrush> i have no idea about the turnaround or timetable for that. *says phoenix three times in a mirror*
[15:56] == natron [~natron@unaffiliated/natron/x-287325] has joined #rubygems
[15:56] <chort0> qrush: Well, even needs to finish collecting the memory image before you shut it down. i've been trying to make that clear, if you don't get that now, it's gone forever, and therefore you cannot trust anything ever done on that box
[15:56] <raz> chort0: the box will be wiped clean (formatted). what happened on there is really only of forensic interest, and they already pulled the mem-dump i think, so this point can be marked as done.
[15:56] <chort0> also note that simply capturing the info isn't a commitment to analyzing it ever. it just means it's possible
[15:57] <@drbrain> qrush: is rubygems.org on AWS?
[15:57] <chort0> if you don't capture the memory and later wish you had, well too bad. if you do capture the memory and dont' do anything with it, no harm done
[15:57] == rcvalle [~rcvalle@redhat/rcvalle] has joined #rubygems
[15:57] == gazoombo [uid6629@gateway/web/irccloud.com/x-ljvxfzlhgxqtcuyy] has joined #rubygems
[15:57] <raz> now might be a good time for one of the sponsors to step in and donate a few machines
[15:57] <qrush> drbrain: only gems are on s3. the backend/db box is at rackspace, "frontend" boxes on RS cloud
[15:58] <raz> i'm sure engineyeard or whoever won't hesitate if you ask them
[15:58] == mrtrick37 [~mrtrick@63.151.200.65] has quit [Quit: Leaving]
[15:58] <raz> or even rackspace itself
[15:58] <postmodern> qrush, do you provision using chef or puppet?
[15:58] <postmodern> qrush, or is the OS setup manually?
[15:58] == biff_tannen [~I.Hate@66.175.107.200] has joined #rubygems
[15:59] <qrush> I think it's mostly manual :( i'm not in charge of ops stuff. There's some stuff here but i'm not sure how it's run or which ones. https://github.com/rubygems/rubygems.org-configs
[15:59] == Elhu [~elhu@128-79-110-56.hfc.dyn.abo.bbox.fr] has quit [Quit: Computer has gone to sleep.]
[15:59] == erbmicha [d0442605@gateway/web/freenode/ip.208.68.38.5] has left #rubygems []
[16:00] == danmcclain [~danmcclai@64.119.141.126] has quit [Quit: Textual IRC Client: www.textualapp.com]
[16:00] == harryv [~harry@iliad.devspool.com] has joined #rubygems
[16:00] <johnmwilliams_> Not sure if it is still wanted but I have the S3 lastmodified timesamps for the top-1000 list qrush posted earlier.
[16:01] == nootch_ [~nootch@cpe-72-225-175-169.nyc.res.rr.com] has quit [Remote host closed the connection]
[16:01] <qrush> sweet!
[16:01] <jjarmoc> drbrain: can you check PM?  sorry to interrupt but you may be interested in what I have to say..
[16:01] <@raggi> chort0: grep the backlog for evans notes, i'm pretty certain he started a dump
[16:01] <jjarmoc> alternately if there's someone else I can contact who has some capacity with rubygems, i'd be happy to explain privately
[16:01] <chort0> raggi: yes he started. no word on whether it finished
[16:01] <jjarmoc> just picked the most recent op in my timeline :)
[16:02] <@drbrain> jjarmoc: hi, see PM
[16:02] <chort0> jjarmoc: I think you want to talk with evan, but i believe he's getting food ATM
[16:02] <chort0> btw jjarmoc works in incident response as well (different company)
[16:03] <jjarmoc> I've got drbrain in PM.. thanks all
[16:03] <benchMark> raggi: http://cl.ly/MY8P -- md5sum output.
[16:03] <kernelsmith> jjarmoc:  that's no good for us :/
[16:03] <benchMark> raggi: 7d7a5da5ebb14760d3fbb92f635bcbba08fe8c35af510b03e39c6106c6ba1b86dcce1328ebffcd3989940edb45f888100d6f002d697f412e91c32bd5af56cd6e  rubygems-md5.txt.gz
[16:04] <@raggi> benchMark: thanks
[16:04] == calmyournerves [~calmyourn@78-235.63-188.cust.bluewin.ch] has joined #rubygems
[16:04] == withloudhands [~robertwhi@rrcs-184-75-101-229.nyc.biz.rr.com] has quit [Quit: withloudhands]
[16:05] == bch820 [~bhenerey@50-73-122-41-ip-static.hfc.comcastbusiness.net] has joined #rubygems
[16:05] == gorsuch [~user@ip72-198-91-203.ok.ok.cox.net] has left #rubygems ["ERC Version 5.3 (IRC client for Emacs)"]
[16:05] == tmaher [~Adium@204.14.152.118] has joined #rubygems
[16:06] <meise_> please don't use md5 checksums!
[16:06] <@drbrain> meise_: please read backtrace
[16:06] <@drbrain> we've got sha512 and md5
[16:07] == ZachBeta [~ZachBeta@c-76-19-166-51.hsd1.ct.comcast.net] has joined #rubygems
[16:07] == thorncp [thorncp@2600:3c01::f03c:91ff:fe93:d182] has joined #rubygems
[16:07] <meise_> k.
[16:07] == withloudhands [~robertwhi@rrcs-184-75-101-229.nyc.biz.rr.com] has joined #rubygems
[16:08] <envygeeks> that checksum definitely wasn't a md5 sum, that sumbitch was way too big
[16:08] <@drbrain> envygeeks: inside is MD5s
[16:08] <envygeeks> ah ok
[16:08] == sferik [~textual@hattery.static.monkeybrains.net] has quit [Ping timeout: 252 seconds]
[16:09] <johnmwilliams_> qrush: top 1000 gems S3 last-modified: http://cl.ly/0L3M31022Q46
[16:09] == iamjarvo [~Adium@c-76-98-135-214.hsd1.pa.comcast.net] has joined #rubygems
[16:09] <johnmwilliams_> qrush: 82889aadfdbbe89e0d816243d619b3b86cbed6bd46d04b7bd1f8745ad3303beb58eb7061420d545d2efe2301cff278e111ec34488e646abb5e48f2c95601c188  top_1000_gem_last_modified
[16:09] == scooter__ [d8291803@gateway/web/freenode/ip.216.41.24.3] has joined #rubygems
[16:09] <@drbrain> benchMark: since my cloud is down I'm making checksums on my local machine, 6000 out of ~250,000
[16:09] <qrush> nzkoz:  ^^^
[16:10] == tarcieri [~bascule@cryptosphere.org] has joined #rubygems
[16:10] <tarcieri> ohai
[16:10] <benchMark> drbrain: cool. I'm going to have to bail soon, but my data is available at the links earlier.
[16:10] <@drbrain> yeah, I downloaded it
[16:10] == mbj [~mbj@p5DC07DE6.dip.t-dialin.net] has joined #rubygems
[16:11] <tmaher> FYI - We (Heroku) have just re-enabled deploys, but bundler is running in local only mode.
[16:11] <qrush> cool
[16:11] == lsegal [uid1565@gateway/web/irccloud.com/x-sbidkgfggzehebdq] has joined #rubygems
[16:12] == konopka_phone [~konopkaph@c-68-81-76-220.hsd1.pa.comcast.net] has quit [Quit: Colloquy for iPhone - http://colloquy.mobi]
[16:12] == exploid [~Adium@mctnnbsa59w-156034059206.dhcp-dynamic.FibreOP.nb.bellaliant.net] has joined #rubygems
[16:13] == egypt [~egypt@metasploit/egypt] has joined #rubygems
[16:13] == scooter__ [d8291803@gateway/web/freenode/ip.216.41.24.3] has quit [Client Quit]
[16:13] <cgcardona> tmaher: good to know
[16:14] == mbj_ [~mbj@p5DC0688B.dip.t-dialin.net] has joined #rubygems
[16:14] <dukedave> tmaher: Haha, amazing
[16:15] == tmaher1 [~Adium@c-50-136-136-212.hsd1.ca.comcast.net] has joined #rubygems
[16:15] <dukedave> We just finally cracked and used the work around ~5 mins ago :D
[16:15] <tmaher> what - I closed my laptop. d oh
[16:15] <cgcardona> tmaher: thanks for the update. good to know.
[16:15] <cgcardona> letting folks know on twitter/irc
[16:16] == yegct [4496069c@gateway/web/freenode/ip.68.150.6.156] has joined #rubygems
[16:17] == mbj [~mbj@p5DC07DE6.dip.t-dialin.net] has quit [Ping timeout: 245 seconds]
[16:18] == mbj_ has changed nick to mbj
[16:18] == vladster [~user@74.212.146.174] has joined #rubygems
[16:18] <tmaher> dukedave cgcardona - to get the shiny new "local only" mode, you'll need to unset your buildpack
[16:18] == mbj [~mbj@p5DC0688B.dip.t-dialin.net] has quit [Client Quit]
[16:18] <dukedave> tmaher I figure
[16:18] <dukedave> I was about 10 seconds too late unsetting
[16:19] <dukedave> (Codeship was deploying so couldn't cancel)
[16:19] <dukedave> Rolled back though
[16:19] <tmaher> Totally understandable.
[16:19] <cgcardona> tmaher: noted
[16:19] == harryv [~harry@iliad.devspool.com] has left #rubygems []
[16:20] <dukedave> Hmm, if I rollback the rollback, now I've config:remove'd the custom build pack, will it deploy with the local one?
[16:20] == cicero [c@poundcs.org] has joined #rubygems
[16:20] <qrush> dukedave: please take heroku stuff to a different channel
[16:20] <@drbrain> dukedave: can you take it to #heroku or similar?
[16:20] <dukedave> Sorry,
[16:20] <dukedave> Didn't even notice, I'm idling in both :)
[16:21] <@drbrain> np
[16:21] == xorrbit [~Noob@206.220.196.52] has joined #rubygems
[16:21] <yegct> Sorry to bug you guys. Any chance of finding out which gem was known to be compromised? I know investigation continues, but I'd like to know if our app is known compromised right now.
[16:21] <qrush> the gem was deleted
[16:22] <benchMark> qrush: Mad props to you and the rest of the RubyGems team who bust your ass on it all the time.
[16:22] <qrush> gems, really
[16:22] <yegct> If you can't share that information, I understand. I've looked all around for it.
[16:22] <@drbrain> yegct: we're working to verify existing gems now
[16:22] <qrush> yegct: to our knowledge, so far, no other gems have been modified. we're double checking all of the existing ones to make sure about this
[16:22] == eka [~eka@190.237.15.76] has joined #rubygems
[16:22] <bradland> yegct: the canonical PoC gem was called exploit
[16:22] <yegct> Okay, thanks. Heh.
[16:23] == pea53 [~Adium@c-98-245-87-57.hsd1.co.comcast.net] has left #rubygems []
[16:23] == funcuddles [~funcuddle@204.57.118.174] has quit [Ping timeout: 246 seconds]
[16:23] <calmyournerves>  Thanks for the great effort anyone involved!
[16:23] == tmaher1 [~Adium@c-50-136-136-212.hsd1.ca.comcast.net] has quit [Ping timeout: 248 seconds]
[16:23] <yegct> Thanks for all your hard work. I've had days like this myself, and they aren't fun.
[16:23] == cowboyd [~cowboyd@12.237.107.68] has quit [Remote host closed the connection]
[16:24] == exploid [~Adium@mctnnbsa59w-156034059206.dhcp-dynamic.FibreOP.nb.bellaliant.net] has quit [Quit: Leaving.]
[16:24] <cgcardona> thanks for the effort everyone. totally appreciated
[16:25] == tenderlove [~tenderlov@pdpc/supporter/active/tenderlove] has joined #rubygems
[16:26] <qrush> benchMark: calmyournerves yegct cgcardona thanks but we're not done yet
[16:26] == kdaigle [~kyle@c-71-235-133-84.hsd1.ct.comcast.net] has quit [Read error: Connection reset by peer]
[16:27] <nzkoz> qrush: just remember you'll have to write up everything after the fact, so I hope someone's keeping a list of "things we did" somewhere
[16:28] == tenderlo_ [~tenderlov@pdpc/supporter/active/tenderlove] has joined #rubygems
[16:28] == j [5ddc002d@gateway/web/freenode/ip.93.220.0.45] has joined #rubygems
[16:28] <bradland> nzkoz: https://docs.google.com/document/d/10tuM51VKRcSHJtUZotraMlrMHWK1uXs8qQ6Hmguyf1g/edit#
[16:28] <bradland> it's not detailed, but items under #3 are being worked
[16:28] == j has changed nick to Guest61
[16:28] <nzkoz> k
[16:29] <qrush> nzkoz: yeah i think we'll have to have a ghangout about this...i'm not sure what else i can log here. i'm gisting all of the results of queries for gems, but that's all i'm taking care of.
[16:29] == _eric [~eric@scribble.5stops.com] has joined #rubygems
[16:29] <qrush> system wise for sure, i'm not sure what the deal there is though
[16:29] <nzkoz> qrush: just make sure the others are doing that too and you'll be fine, I'm going to stop reading in here now but you can ping me if I can be of assistance
[16:29] == catphish [~charlie@2001:9d8:2005:12::3] has joined #rubygems
[16:29] == crankharder [~crankhard@ip68-98-153-131.dc.dc.cox.net] has joined #rubygems
[16:30] <hone> nzkoz: thanks for the help man!
[16:30] == tenderlove [~tenderlov@pdpc/supporter/active/tenderlove] has quit [Ping timeout: 240 seconds]
[16:30] <evan> hi y'all
[16:30] <bradland> an important piece of data will be a timestamp of when the analysis has been completed
[16:30] <calmyournerves> If you guys need any help, just shoot
[16:30] <_eric> have there been any posts of what is known so far?
[16:30] <johnmwilliams_> bradland: I posed the last-modified times ^ for the top 1000 gems that qrush posted.
[16:30] <bradland> once the hardware is replaced, you'll only need to analyze files with S3 modified timestamps later than that date
[16:30] == korishev [~korishev@rrcs-24-173-70-118.sw.biz.rr.com] has joined #rubygems
[16:30] <evan> was a compromised gem detected?
[16:31] <evan> or was yegct refering to the expliot gem?
[16:31] <bradland> evan: he was just asking
[16:31] == stefan_n [~stefan_n@pool-71-164-229-139.dllstx.fios.verizon.net] has joined #rubygems
[16:31] <bradland> johnmwilliams_: thx
[16:32] <johnmwilliams_> I believe that qrush is getting the timestamps from the DB now...
[16:32] <qrush> sorry, getting distracted...
[16:32] <catphish> qrush: keep up the good work
[16:32] <lsegal> we've checked timestamps on our gems, they look fine
[16:32] <bradland> johnmwilliams_:  did you save the script you used to generate these?
[16:32] == kdaigle [~kyle@c-71-235-133-84.hsd1.ct.comcast.net] has joined #rubygems
[16:32] <evan> johnmwilliams_: what did you use to generate http://cl.ly/0L3M31022Q46
[16:32] <evan> ?
[16:32] <bradland> jinx!
[16:33] <johnmwilliams_> evan: head request via curl
[16:33] <bradland> is the S3 last modified meta the same as the file last modified?
[16:33] <bradland> i'm not sure it is
[16:33] == cowboyd [~cowboyd@12.237.107.68] has joined #rubygems
@drbrain
@raggi
_eric
_maes_1
abuiles_
adam12
adf
amateurhuman
andrewhubbs
anon4224124
aquaranto
asenchi
Ash
ashedryden
atomgiant
autojack
badkins
bcardarella
bch820
bdrewery
benchMark
benhamill
bfleischer
biff_tannen
blacktip
blowmage
boffbowsh
borski
bradland
bradrub
brad[]
breakingthings
brixen
Cakey
calmyournerves
catphish
cgcardona
chort0
cianuro
cicero
ckrailo
cktricky
cmeiklejohn
corundum
cout
cowboyd
crandquist
crankharder
cschneid
d-rock_
daaaan
danp_
DanR_
darix
dbussink
ddfreyne
ddollar
Defiler
derekprior
devn
dkubb
dreww
dukedave
dwradcliffe
egypt
eighthbit
eka
elskwid
emachnic
envygeeks
erichmenge
evan
fbernier
foca
foohey
franckverrot_
fromonesrc
gazoombo
ged
Gixug
graphex
gravely
greggroth
grims
Guest61
Guest85414
guilleiguaran
hakunin
hdm
henrikhodne
hgmnz
hone
iamjarvo
ignatz
imajes
infinity432
james__
jarib
jaxx
jcaudle
jcran
jeffreybaird
jhelwig
jjarmoc
jm___
joealba
joewilliams
JohnHirbour
johnmwilliams_
jonathanwallace1
JSharp
jshsu
justincampbell
kaichanvong
kallistec
kdaigle
kentaro
kernelsmith
khaase
korishev
koudelka
kristopher
kurtisnelson
lemonodor
letch_
lianj
listrophy
lmarburger
lsegal
lucas
macek
markstarkman
martinisoft
mastahyeti
mattetti
matthavener
meise_
mephux
mitchellh
mletterle
morphis
mose
mpapis
nakajima
nateberkopec
natron
Nilla_
niska
noah1
notnerb
nz
nzkoz
ohcibi
patricksroberts_
peregrine81
pignata
postmodern
pwelch
qrush
Radar
Raisins
rajiv
randym
raz
rcvalle
revans
riddle
rondale_sc
rpc__
rubysolo
rwjblue
samkottler
shockz
shtirlic
sikachu
sstarr
stayarrr
stefan_n
stepheneb
stevenharman
taotetek
tarcieri
tcopeland
technomancy
tenderlo_
terceiro
terracotta
the_mentat
themcgruff
thorncp
titanous
tmaher
tsykoduk
Unixmonkey
vladster
vrillusions
wayneeseguin
wbruce
wdperson
weeb1e
wesgarrison_
wfarr
withloudhands
wlll
wycats_
xerxas
xorrbit
xternal
xymox
yeban
yegct
yerhot
z
ZachBeta
zdennis
zzak