bcherny

Notes on Martin Kleppmann's excellent Designing Data-Intensive Applications.

Chapter 1: Reliable, Scalable, and Maintainable Applications

• Data Systems
  • Dimensions to consider when thinking about data systems: access patterns, performance characteristics, implementations.
  • Modern data systems often blur the lines between databases, caches, streams, etc.
• Reliability
  • Systems should perform the expected function at a given level of performance, and be tolerant to faults and user mistakes
• Fault: One component of a system deviating from its spec. Prefer tolerating faults over preventing them (except for things like security issues). Faults stem from hardware failures, software failures, and human error (in a study, config errors caused most outages).

MWins

Project Ideas

Ok. I'm going to list off some ideas for projects. You will have to determine if any particular idea is good enough to include in a portfolio. These aren't creative ideas. They likely already exist. Some are way too advanced while others are simplistic.

I will recommend to post any project you make to github and make a github project page for it. Explain in as much detail as possible how you made it, how it can be improved etc. Document it.

If you pick an advanced idea, setup a development roadmap and follow it. This will show some project management skills.

Another piece of advice for those who are design challenged. Use different front end frameworks and use different themes for those frameworks to provide appealing designs without looking like yet another bootstrap site.

ptarjan

Scaling your API with rate limiters

The following are examples of the four types rate limiters discussed in the accompanying blog post. In the examples below I've used pseudocode-like Ruby, so if you're unfamiliar with Ruby you should be able to easily translate this approach to other languages. Complete examples in Ruby are also provided later in this gist.

In most cases you'll want all these examples to be classes, but I've used simple functions here to keep the code samples brief.

Request rate limiter

This uses a basic token bucket algorithm and relies on the fact that Redis scripts execute atomically. No other operations can run between fetching the count and writing the new count.

Chaser324

Whether you're trying to give back to the open source community or collaborating on your own projects, knowing how to properly fork and generate pull requests is essential. Unfortunately, it's quite easy to make mistakes or not know what you should do when you're initially learning the process. I know that I certainly had considerable initial trouble with it, and I found a lot of the information on GitHub and around the internet to be rather piecemeal and incomplete - part of the process described here, another there, common hangups in a different place, and so on.

In an attempt to coallate this information for myself and others, this short tutorial is what I've found to be fairly standard procedure for creating a fork, doing your work, issuing a pull request, and merging that pull request back into the original project.

Creating a Fork

Just head over to the GitHub page and click the "Fork" button. It's just that simple. Once you've done that, you can use your favorite git client to clone your repo or j

chitchcock

Stevey's Google Platforms Rant

I was at Amazon for about six and a half years, and now I've been at Google for that long. One thing that struck me immediately about the two companies -- an impression that has been reinforced almost daily -- is that Amazon does everything wrong, and Google does everything right. Sure, it's a sweeping generalization, but a surprisingly accurate one. It's pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn't let me show it to anyone, even though recruiting loved it.

I mean, just to give you a very brief taste: Amazon's recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they've made to level it out. And their operations are a mess; they don't real