Skip to content

Instantly share code, notes, and snippets.

@adibenc

adibenc/Big List Of 20 Common Bottlenecks.md

Last active October 28, 2021 04:01
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save adibenc/41aa87c7c84466c0ca41280c01da90b8 to your computer and use it in GitHub Desktop.
Save adibenc/41aa87c7c84466c0ca41280c01da90b8 to your computer and use it in GitHub Desktop.
Download ZIP
Big List Of 20 Common Bottlenecks
Raw
Big List Of 20 Common Bottlenecks.md

http://highscalability.com/blog/2012/5/16/big-list-of-20-common-bottlenecks.html

Big List Of 20 Common Bottlenecks

WEDNESDAY, MAY 16, 2012 AT 9:15AM

In Zen And The Art Of Scaling - A Koan And Epigram Approach, Russell Sullivan offered an interesting conjecture: there are 20 classic bottlenecks. This sounds suspiciously like the idea that there only 20 basic story plots. And depending on how you chunkify things, it may be true, but in practice we all know bottlenecks come in infinite flavors, all tasting of sour and ash.

One day Aurelien Broszniowski from Terracotta emailed me his list of bottlenecks, we cc’ed Russell in on the conversation, he gave me his list, I have a list, and here’s the resulting stone soup.

Russell said this is his “I wish I knew when I was younger" list and I think that’s an enriching way to look at it. The more experience you have, the more different types of projects you tackle, the more lessons you’ll be able add to a list like this. So when you read this list, and when you make your own, you are stepping through years of accumulated experience and more than a little frustration, but in each there is a story worth grokking.

Database:

  1. Working size exceeds available RAM

  2. Long & short running queries

  3. Write-write conflicts

  4. Large joins taking up memory

  5. Virtualisation:

  6. Sharing a HDD, disk seek death

  7. Network I/O fluctuations in the cloud

Programming:

  1. Threads: deadlocks, heavyweight as compared to events, debugging, non-linear scalability, etc...

  2. Event driven programming: callback complexity, how-to-store-state-in-function-calls, etc...

  3. Lack of profiling, lack of tracing, lack of logging

  4. One piece can't scale, SPOF, non horizontally scalable, etc...

  5. Stateful apps

  6. Bad design : The developers create an app which runs fine on their computer. The app goes into production, and runs fine, with a couple of users. Months/Years later, the application can't run with thousands of users and needs to be totally re-architectured and rewritten.

  7. Algorithm complexity

  8. Dependent services like DNS lookups and whatever else you may block on.

  9. Stack space

Disk:

  1. Local disk access

  2. Random disk I/O -> disk seeks

  3. Disk fragmentation

  4. SSDs performance drop once data written is greater than SSD size

OS:

  1. Fsync flushing, linux buffer cache filling up

  2. TCP buffers too small

  3. File descriptor limits

  4. Power budget

Caching:

  1. Not using memcached (database pummeling)

  2. In HTTP: headers, etags, not gzipping, etc..

  3. Not utilising the browser's cache enough

  4. Byte code caches (e.g. PHP)

  5. L1/L2 caches. This is a huge bottleneck. Keep important hot/data in L1/L2. This spans so much: snappy for network I/O, column DBs run algorithms directly on compressed data, etc. Then there are techniques to not destroy your TLB. The most important idea is to have a firm grasp on computer architecture in terms of CPUs multi-core, L1/L2, shared L3, NUMA RAM, data transfer bandwidth/latency from DRAM to chip, DRAM caches DiskPages, DirtyPages, TCP packets travel thru CPU<->DRAM<->NIC.

CPU:

  1. CPU overload

  2. Context switches -> too many threads on a core, bad luck w/ the linux scheduler, too many system calls, etc...

  3. IO waits -> all CPUs wait at the same speed

  4. CPU Caches: Caching data is a fine grained process (In Java think volatile for instance), in order to find the right balance between having multiple instances with different values for data and heavy synchronization to keep the cached data consistent.

  5. Backplane throughput

Network:

  1. NIC maxed out, IRQ saturation, soft interrupts taking up 100% CPU

  2. DNS lookups

  3. Dropped packets

  4. Unexpected routes with in the network

  5. Network disk access

  6. Shared SANs

  7. Server failure -> no answer anymore from the server

Process:

  1. Testing time

  2. Development time

  3. Team size

  4. Budget

  5. Code debt

Memory:

  1. Out of memory -> kills process, go into swap & grind to a halt

  2. Out of memory causing Disk Thrashing (related to swap)

  3. Memory library overhead

  4. Memory fragmentation

  5. In Java requires GC pauses

  6. In C, malloc's start taking forever

If you have any more to add or you have suggested fixes, please jump in.

Thanks to Aurelien and Russell for all their applied brain power.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment