Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Basecamp is under network attack (DDoS)

Basecamp was under network attack

The attack detailed below has stopped (for the time being) and almost all network access for almost all customers have been restored. We're keeping this post and the timeline intact for posterity. Unless the attack resumes, we'll post a complete postmortem within 48 hours (so before Wednesday, March 26 at 11:00am central time).

Criminals have laid siege to our networks using what's called a distributed denial-of-service attack (DDoS) starting at 8:46 central time, March 24 2014. The goal is to make Basecamp, and the rest of our services, unavailable by flooding the network with bogus requests, so nothing legitimate can come through. This attack was launched together with a blackmail attempt that sought to have us pay to avoid this assault.

Note that this attack targets the network link between our servers and the internet. All the data is safe and sound, but nobody is able to get to it as long as the attack is being successfully executed. This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.

We're doing everything we can with the help of our network providers to mitigate this attack and halt the interruption of service. We're also contacting law enforcement to track down the criminals responsible. But in the mean time, it might be a rough ride, and for that we're deeply sorry.

DDoS criminals have attacked and tried to extort many services lately. Just a few weeks ago, Meetup was attacked, and it took a whole weekend of fire fighting before they were out of the woods. There is unfortunately no single, quick fix to these attacks, so we regretfully ask for your patience in advance. As said, we're doing everything we can, and will work as quickly as possible, but it's impossible to give a clear timeline for ultimate resolution.

The only thing we're certain of of is that, like Meetup, we will never negotiate by criminals, and we will not succumb to blackmail. That would only set us up as an easy target for future attacks.

We'll keep everyone updated through and Twitter (@37signals). Again, terribly sorry about this lousy way to start the week.


UPDATE: Attacker identified as being responsible for similar attacks (9:55am central time, March 2014)

We've learned that the very same criminals currently attacking and trying to extort us hit others just last week. We're comparing notes with everyone affected who have been in touch. The blackmail came from an address matching this pattern: dari*** If you have been extorted by this person, please get in contact so we can compare notes on both technical defenses and the law enforcement effort to hunt them down.


UPDATE: Law enforcement efforts pooled, attack currently waning (10:21am central time, March 2014)

We've pooled our law enforcement efforts with the other victims now, and are working with the same agent on the case. While tracking down these criminals is notoriously hard, we'll do our very best to bring them to justice.

At the moment it seems that the attack has also let up a bit. Our network providers have been doing a good job dealing with up to a 20Gbps attack. But from what we've heard from the other victims, the criminals are capable of even more than that, so we're not out of the woods yet.


UPDATE: Main attack has stopped, but still some network issues (10:41am central time, March 2014)

The main attack seems to have stopped now, but we're still dealing with a variety of network issues. Basecamp and the other services should currently be accessible to most customers, but not all. We're working fast and diligently to resolve all lingering issues. We may well still be attacked again, but for now it's about cleaning up the damage. We again thank everyone for their patience. This has been a horrible morning.


UPDATE: Service restored for 95% of customers, still working on last 5% (10:56am central time, March 2014)

With the main attack stopped, we've been able to restore service for about 95% of all customers. We're still working on restoring everything for everyone everywhere, though. When these attacks happen, the rest of the internet will sometimes put you in quarentine to prevent the fire from spreading. So even after an attack has stopped, it can take a while before you're allowed to leave quarentine. That's the phase we're currently in.

Reminder: The attack has stopped for now, but there's no guarentee it will not resume. Other victims have told us about how the attacker would take a break, and then try again later with a different method. Hopefully that will not be the case, but we remain on the highest alert for now.

Best of luck! Thanks for the clear, concise update. We're standing with you guys!

I really appreciate that you guys are standing up to this, even if it makes it harder for me to get into Basecamp for a while.

Thanks for fighting the good fight. Someone has to stand up to this, and I'm glad it's you.

Good luck.

Thanks for the update. :)

Ah man. Best of luck. Thanks for keeping us posted.

Fight the good fight brudda.

Good luck to the entire team and hang in there. Sorry your team has to deal with criminals like this, but in the end I'm betting on the Basecamp team.

We're with you guys. Fight the good fight!

Fight on!

Sad that while people build up, others try to tear down and profit from it. Best of luck.

Thank you so much for the update. This is ridiculous on their part. I fully support you all's position on this.

iwanow commented Mar 24, 2014

Good luck! Thanks for the info

NKjoep commented Mar 24, 2014

Stay strong!

Do you guys use any mitigation services like CloudFlare etc?

I hope they share the code used for this attack here in github.

Are you guys in one of the Salt Lake City datacenters by chance? It looks like we're being attacked too...

Ugh, happy Monday! Thanks for the update. Good luck!

Ugh. Who would do this on a monday morning? Anyways, best of luck guys.

I'd get in my race car and drive really fast.

acidhax commented Mar 24, 2014

Shit, that sucks. Time to give your network engineers a raise.

gorsuch commented Mar 24, 2014


Thanks for the update.

Good luck! Thanks for updates, and thank you for standing up to bullies.

The good news is that you'll hopefully put in some countermeasures going forward.

Best of luck.

Not to be negative or start a flame war on here, but how certain are you that are information is really secure during this attack? As a server administrator myself, I've seen plenty of DDOS attacks that lead to getting access into a server, compromising the security of the primary data on it. What assurances do we have that the information could not be easily obtained if this attack leads to an account breach on the server? Is our information stored via any hashing methods? Is the data stored in an encrypted, turn-key decrypting system? Or is the information stored as plain text in a database? If the data is not secured in any way on the server, it creates a major security risk for my company as we have hundreds of logins stored on our Basecamp account.

Just another form of terrorism. Glad to see you guys are fighting to fix it and not caving to these guys.

Good luck!!

Thanks. I can't believe this. Good luck hunting them down.

Good luck

afdiaz commented Mar 24, 2014

Go Basecamp! This will make you guys stronger. We hope they hunt down the responsible people.

smutek commented Mar 24, 2014

Looks like you're back up - hope you've got it sorted.

Best of luck.

Thanks for clarification. Best of luck!

@saintisaiah congrats! You've seen a unicorn. DDOS attacks don't lead to data vulnerabilities.

SPGB commented Mar 24, 2014

@saintisaiah Why are you storing "hundreds of logins" on Basecamp?

Passwords should be unique so even if a data breach occurs your other credentials aren't compromised.

Agreed that there could be something more going on behind the scenes.

@saintisaiah I'd love to see examples of what you describe.

I would like to wish you the best of luck! I really appreciate the transparency in this matter.

@saintisaiah, this is a network attack. The attackers are merely using a flood of traffic so Basecamp becomes overwhelmed and can't serve normal requests.

@saintisaiah pro admin skillz bro

walker commented Mar 24, 2014

@saintisaiah: el oh el.

Thanks for the info!

Dang... I just started a brand new project today. It's crazy how dependent you become on third part software to complete projects... Looks like it's time to break out the ol pen and paper.

rosem commented Mar 24, 2014

when it's all done, you'll have the best worst monday story to tell. :)

This is the first time I've ever seen any 37s product suffer from this sort of attack. That tells me you guys are either really good at masking it to us, have really robust systems in place, have designed great software that helps to prevent these sort of mass outages, or a great - and dedicated - team of people behind the products.

Oh wait, all of those are probably true.. :)

Seriously, wouldn't think of spending my money on any of your competitors. Well done 37s and team.

Dirty pirate bastards. Stand your ground, 37!!

plus- commented Mar 24, 2014

@everyone mocking @saintisaiah,
DDoS can be used on purpose to mask a server breach attack.

@SPGB - There are hundreds of unique logins because we have hundreds of clients. The storing of the logins on basecamp is not my choice and was against my advisement to the company owner. That being said...

To everyone who is taking what I'm saying out of context so rudely, please understand that I did not say that DDoS is used for gaining access to a server, but that it could LEAD to it. Here are a few ways that a DDoS attack could be used as a step to gaining access to a server.

1.) Overwhelming primary defenses. when you are conducting a DoS attack, the primary defense mechanisms get caught up in it too. They can be overwhelmed and as a result, they may: a) not respond appropriate b) can hang altogether c) the watchers watching them are distracted, or your efforts are lost in the DoS logs. In addition, if the primary systems are rolled over to secondary systems (which often happens) those may not be up-to-date configurations, or you could catch the roll-over in a time-gap whereby the synchronization of everything is not current.

2.) Overwhelming primary systems may expose flaws. A DoS attack may be used to expose flaws that could be exploited. Could be procedural flaws, it could be system flaws. It could be that as a result of a DoS attack you force the organization to upgrade, and during the upgrade you take that window of opportunity to exploit.

3.) The DoS attack is a decoy. Classic attack... What magicians do all the time. Watch the left hand while I steal with my right. The DoS has so much focus of the organization, secondary routes into the system (physical, social, or technical) could be undermanned, provisioned or systems can be more easily by-passed without being noticed.

4.) Secondary Route/Bridging Exploitation. An extended DoS attack that can be sustained, can force business units within the organization to move to secondary systems paths (networks) to keep critical business going. While some of these secondary paths may be well-planned and secured, many are not. For instance a business unit may stand up a wifi, or mifi device and start using it as their business network without any security infrastructure. If an attacker is actively monitoring and profiling, they may be able to capture and attack these very vulnerable paths and now you have a direct, totally unsecured bridged network into the organizations intranet.


Best of luck getting it sorted, terrible start to the week!

I have a major project update kicking into action today and thought it was my connection that was struggling until I went on to Twitter to see if anyone else was experiencing it too. 10/10 for your updates and communication with your users - setting a great example under unfortunate circumstances.

great transparency!!!

Take em down!

@plus- Thanks for helping to point that out. Seems like these hecklers think pretty linearly rather than seeing the big picture.

When you find these guys, send in the drones!

olsonea commented Mar 24, 2014

Any chance that discussion-by-email functionality is unaffected?

lsmolic commented Mar 24, 2014

The Zone will be one of danger?

pgib commented Mar 24, 2014


Thanks for the update guys, zero problem with service-outage when it's explained like this. God speed!

metelyk commented Mar 24, 2014

thank you for the honest update, good luck guys!

Great communication!

Thanks for the updates guys.

chhhris commented Mar 24, 2014

@saintisaiah while I'm sorry you got heckled, your retort is super interesting. Thanks for sharing.

The openness that you guys/gals are employing is phenomenal. So far, the best way you could've handled this.

Great update. Thanks for all your hard work

@chhhhris Thanks, happy to help!

@richardtabor - Agreed, although I'm concerned about the server security, the transparency about the matter and the frequent updates to keep us informed is greatly appreciated and admired, as most companies would sweep the truth under the rug. Thanks @dhh !

Thanks for the update - I really appreciate the open communication. My biz depends on yours, so I hope you're back up soon. I stand with you. Good luck.

Jokes on the attacker, really. This is going to turn into another successful book!

When I read the headline, I thought it was because of your name change and someone was pissed. ¯_(ツ)_/

Syerram commented Mar 24, 2014

Great job guys and thanks for the update. Dont give in.

Is it possible to share some of the technical details on the counter attack?

Thanks for the great communication. It's much appreciated. Congrats on fighting back successfully... we're behind you!

Dorian commented Mar 24, 2014

Funny that the basecamp status page is down so they have to use a gist and twitter, but thanks for the informative update.

Good luck over there. I know this is rough time for your Ops folks

Fight on.

Basecamp has helped my globally dispersed startup team stay in touch. You guys are awesome and we're 100% behind you! Good luck.

vrash commented Mar 24, 2014

Yikes! Good luck!

@saintisaiah Nice plagiarism ---> Stop trolling this thread please.

@chhhris @saintisaiah stole that whole response from a Stack Exchange answer.

@inspectorfegter I didn't say I wrote it, I merely took a bookmark I had and pasted in what best explained it. You need to grow up sir. But to make you happy, I'll cite the source in the post above, which I forgot to do.

Fight the good fight.

six0h commented Mar 24, 2014

Good luck guys!

@saintisaiah No, grown ups cite their sources and don't act like other's IP is their own.

argen commented Mar 24, 2014

Good luck guys! Hope you can cease their fire and catch them after that.

Best of luck guys! My company just went through that recently as well. Nowhere near as bad as 20Gbps, but enough to put extra security and proxies in place to lighten the attack. Ours lasted for about 3 days, although the first 24 hours was the worse and affected our site along with our client sites so it was something that had to be resolved asap.

@inspectorfegter Believe what you want. Not once in my responses did I say that I personally wrote it. It was my fault for not citing, regardless of the fact that I forgot and I apologize to anyone who was greatly affected by the notion that I had originally wrote it. But trying to help explain my initial comment which was so badly taken out of context was not meant to make me some beacon of knowledge that everyone should look to, but to help raise awareness that this attack should concern others of the safety of their proprietary information.

Your comments are negative, malicious and no better than the actions of these criminals attacking 37Signals and their Basecamp product website. How about adding something constructive and positive to the conversation, rather than trying to prove some irrelevant point to make yourself look cool?

smutek commented Mar 24, 2014

^ just stop? ^

@saintisaiah I have not attacked you personally nor will I. In two posts, you have called me a child and a criminal. I was pointing out the fact that what you posted was absolutely not original to you and that you are just adding a layer of paranoia to this thread that shouldn't exist in the first place. Why drum up additional concerns when there is no evidence of anything other than a denial-of-service attack? In fact, your arguments can only serve to harm 37 signals by calling into question their integrity of data storage at a time like this. It also was a bit foolish to expose your organization's practice of storing plaintext user/passwords in Basecamp publicly. This serves only to make your account a rich target for script kiddies around the globe. - Signed The Cool, but Childish Criminal

pmahnke commented Mar 24, 2014

give us the person's email address, we can assign hundreds of todo's to them! see how they like it!


It is great to see a open post like this. Good luck !

Can you tell what mitigation services/ methods you are using it will be helpful for the community sorry if i missed some online link briefing about this.

woow this is what I call a GOOD way to communicate a serious problem, Good job 37Signals team!

@inspectorfegter You accused me of theft, which is a personal attack. Though your claim was valid, which I promptly corrected when it was pointed out, it could have been pointed out in a more mature way, hence why I told you to grow up. I did not say you were a criminal, but that your comments were no better than the actions against Basecamp today, in that they were purely negative and destructive.

My team and I have already changed all of our passwords and stored them in a different service specifically for passwords, so I'm not worried about it.

And regardless of having additional evidence or not, it's not unreasonable to ask 37Signals (which I assumed they are monitoring since they have consistently updated the post) how secure my company's information really is. What should I do? Send a contact request through their website and sit idly by while I wait into oblivion for a response? If it's a concern for me, it's undoubtedly a concern for many others, and I'd rather try to get an answer now rather than sweep it under the rug and wait for a more serious security compromise in the future.

I'll just leave it at that and walk away, as this conversation is not meant for us to bicker back and forth, but for customers to be heard and updated throughout this issue.

memiux commented Mar 24, 2014

Just a quick reminder.

We're changing our name. 37signals is now Basecamp. "37signals" goes into the history books. From now on, we are Basecamp. Basecamp the company, Basecamp the product. We're one and the same.


@inspectorfegter +1. I feel so sad that they found themselves in a tough situation like this. Please support them and help them if you would. Please mind that this is not the time to criticise. Good luck 37s and DHH.


ghost commented Mar 24, 2014

In an age of the 400gbps+ ntp attacks they took you down with 20gbits and you were not prepared ? are you kidding me right now ? Are you living in 2004 ?

teapot commented Mar 25, 2014

@xnljfr 20Gbits of malicious traffic rushing in per second can potentially have catastrophic effects on almost any network, however, it's all dependant on how that traffic is used.
Reading through this gist it seems the information regarding the attack is very vague, so there's no real telling what type of attack was used. Additionally, they may not receive attacks very often, if not at all, so spending thousands upon thousands of dollars protecting against such things may not of been on their mind or something they felt the need to do.

Also attacks ranging from 20Gbps and upwards were not reported until 2007 (according to the below source), so if they were living in 2004, they'd have something quite nice to go in the history books :).

Here is an (outdated) graph of some of the largest reported DDoS attacks:

The full page/pdf can be found here.

In regards to NTP attacks; they have been proclaimed to be dead from the lack of vulnerable servers available to execute the attack, so it's doubtful we'll be hearing much more in regards of future attacks. This doesn't mean we won't be seeing attacks as large if not even larger in the future however; there are still many attack vectors to be used that have quite the potential if used correctly.

On an almost final note, I'd wish you the best of luck with the attack issues and hope they come to a halt!

All the best in the fight and thanks for the transparency and commitment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment