Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

Long story short, Celluloid versions 0.17+ have a memory leak.

The Reason behind this is that completed Celluloid threads are never cleaned up.

We have discovered that our Sidekiq process is leaking memory when we have a lot of tasks that were failed because of exceptions. Unfortunately, having a lot of failed tasks is specific for our application — we do have a lot of small queued jobs to work with social network APIs and other external services.

You can reproduce the problem with this: https://gist.github.com/gazay/3aa78e515ab05cb79f76

Dead jobs   rss      alive/total
1:          15MB     (Fibers: 4/4, Threads: 3/3)
1000:       52MB     (Fibers: 1003/2002, Threads: 3/1002)

Initial issue is described here: https://github.com/celluloid/celluloid/issues/670

And the fix is here: https://github.com/celluloid/celluloid/commit/5725feeec7e78e83298daacb670a85f20971f3ff

The fix is is included in Celluloid 0.17.2 that was released on September 30th, 2015.

Кратко - потому что версии 0.17+ текут.

Суть: мертвые потоки не чистятся

Заметили утечку памяти при большом количестве упавших тасок (для приложения специфично падение тасок, так как много мелких тасок работающих с соц сетями и другими внешними сервисами).

Воспроизвести можно следующим образом: https://gist.github.com/gazay/3aa78e515ab05cb79f76

Умерших задач  rss  живые/всего
1:             15MB (Fibers: 4/4, Threads: 3/3)
1000:          52MB (Fibers: 1003/2002, Threads: 3/1002)

Проблема: https://github.com/celluloid/celluloid/issues/670

Фикс: https://github.com/celluloid/celluloid/commit/5725feeec7e78e83298daacb670a85f20971f3ff

Включен в релиз 0.17.2 от 30-го сентября

@gazay

This comment has been minimized.

Copy link
Owner Author

commented Oct 1, 2015

Many thanks to @brainopia for helping me to find this out and make reproduce steps

@sponomarev

This comment has been minimized.

Copy link

commented Oct 1, 2015

thanx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.