Skip to content

Instantly share code, notes, and snippets.

@srghma

srghma/Server crashes Ive experienced and their causes.md

Last active July 23, 2018 13:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save srghma/395e43a3453679564901a236e068663e to your computer and use it in GitHub Desktop.
Save srghma/395e43a3453679564901a236e068663e to your computer and use it in GitHub Desktop.
Download ZIP
Server crashes Ive experienced and their causes
Raw
Server crashes Ive experienced and their causes.md

Rails, bad configuration

We thought it was memory leak, but it was bad configuration

configuration that didnt work before:

  1. puma (RAILS_MAX_THREADS: 8, WEB_CONCURRENCY: 8)
  2. db pool - 10
  3. memory cache

worked:

  1. puma (RAILS_MAX_THREADS: 2, WEB_CONCURRENCY: 8)
  2. db pool - 100
  3. redis + elasticache

Conclusion: ???

Rails, very big publication content because of inlined base64 images

Someone created publication with inlined image <img src="data:base64...."> and server crashed because publication content were parsed for [shortcodes]

We thought it was memory leak, but it was not.

We uploaded such images and problem disappeared.

Lessons learned:

  • bundle exec derailed exec perf:mem_over_time doesnt show right stats without GC.start on each request
  • memory graph should be as logaripghm, if its increasing linearly - bad, if after 200k req its still increasing a little - fine

How it could be prevented:

  • check maybe only some request cause crash
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment