Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save gazay/3b518f72266b5a7e88ff to your computer and use it in GitHub Desktop.
Save gazay/3b518f72266b5a7e88ff to your computer and use it in GitHub Desktop.

Long story short, Celluloid versions 0.17+ have a memory leak.

The Reason behind this is that completed Celluloid threads are never cleaned up.

We have discovered that our Sidekiq process is leaking memory when we have a lot of tasks that were failed because of exceptions. Unfortunately, having a lot of failed tasks is specific for our application — we do have a lot of small queued jobs to work with social network APIs and other external services.

You can reproduce the problem with this: https://gist.github.com/gazay/3aa78e515ab05cb79f76

Dead jobs   rss      alive/total
1:          15MB     (Fibers: 4/4, Threads: 3/3)
1000:       52MB     (Fibers: 1003/2002, Threads: 3/1002)

Initial issue is described here: celluloid/celluloid#670

And the fix is here: https://github.com/celluloid/celluloid/commit/5725feeec7e78e83298daacb670a85f20971f3ff

The fix is is included in Celluloid 0.17.2 that was released on September 30th, 2015.

Кратко - потому что версии 0.17+ текут.

Суть: мертвые потоки не чистятся

Заметили утечку памяти при большом количестве упавших тасок (для приложения специфично падение тасок, так как много мелких тасок работающих с соц сетями и другими внешними сервисами).

Воспроизвести можно следующим образом: https://gist.github.com/gazay/3aa78e515ab05cb79f76

Умерших задач  rss  живые/всего
1:             15MB (Fibers: 4/4, Threads: 3/3)
1000:          52MB (Fibers: 1003/2002, Threads: 3/1002)

Проблема: celluloid/celluloid#670

Фикс: https://github.com/celluloid/celluloid/commit/5725feeec7e78e83298daacb670a85f20971f3ff

Включен в релиз 0.17.2 от 30-го сентября

@gazay
Copy link
Author

gazay commented Oct 1, 2015

Many thanks to @brainopia for helping me to find this out and make reproduce steps

@sponomarev
Copy link

thanx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment