gazay

Long story short, Celluloid versions 0.17+ have a memory leak.

The Reason behind this is that completed Celluloid threads are never cleaned up.

We have discovered that our Sidekiq process is leaking memory when we have a lot of tasks that were failed because of exceptions. Unfortunately, having a lot of failed tasks is specific for our application — we do have a lot of small queued jobs to work with social network APIs and other external services.

You can reproduce the problem with this: https://gist.github.com/gazay/3aa78e515ab05cb79f76

gazay

Начиная с версии 2.2.0 (даже с ее превью), руби (mri) может быть причной падения вашего сервера по ООМ. Моя длительная, многомесячная эпопея с поиском загадочного бага, которую в подробностях я опишу в следующей статье, окончилась написанием патча в руби. Сейчас я хотел бы объяснить суть этого бага и суть патча.

Я бы ничего не нашел без помощи моих друзей - Равиля Байрамгалина, который прошел со мной весь путь, начиная с анализа первых графиков падений и погружения меня во все потаенные тонкости дебага руби приложений, и Владимира Меньшакова, который в C творит чо хочет вообще и помогал мне с дебагом, относящимся к C стороне выполнения руби приложений.

gazay

Keybase proof

I hereby claim:

• I am gazay on github.
• I am gazay (https://keybase.io/gazay) on keybase.
• I have a public key whose fingerprint is C42D 2AF6 2D78 8EC6 FE95 DB99 C862 2100 4667 8AC5

To claim this, I am signing this object:

gazay

"require 'spec_helper'\n\ndescribe 't34' do\n  let(:target) {\n\"class X\n  def test_method(arg1)\n  end\nend # X\"\n  }\n\n  let(:target2) {\n\"class X\n  def test_method(xxx, arg2)\n  end\nend # X\"\n  }\n\n  let(:source) {\n\"class X\n  def test_method(arg1, arg2)\n  end\nend\"\n  }\n\n  let(:rewriter) {\n    T34::Rewriter.new source\n  }\n\n  it 'finds methods' do\n    expect(rewriter.methods(:test_method)).to be_kind_of Array\n  end\n\n  it 'finds method nodes' do\n    expect(rewriter.methods(:test_method).map(&:class).compact).to eq [T34::Rewriter::API::MethodNode]\n  end\n\n  it 'manipulates methods' do\n    res = rewriter.methods(:test_method) do |method_node|\n      method_node.args = method_node.args[0...-1]\n    end\n    expect(res[0].args.size).to eq 1\n  end\n\n  it 'manipulates arguments by name' do\n    res = rewriter.methods(:test_method) do |method_node|\n      method_node.args = method_node.args.select { |it| it.name != 'arg2' }\n    end\n    expect(res[0].args.size).to eq 1\n    expect(rewr

gazay

require 'fileutils'

class Helper

  attr_accessor :dir, :temp_dir
  attr_reader :template_path, :original_dir

  def initialize(dir = nil)
    @template_path = File.expand_path('../../app_template', __FILE__)
    @original_dir = FileUtils.pwd

gazay

#!/usr/bin/env ruby

client_dir = File.expand_path(File.join(File.dirname(__FILE__), '..', 'proxy'))
$LOAD_PATH.unshift(client_dir) unless $LOAD_PATH.include?(client_dir)
require 'client'

# Get parameters and start the server
if ARGV.size == 2
  port, proxy = ARGV
  port = port.to_i

gazay

# Based on https://gist.github.com/74107 by # Copyright (C) 2009 Torsten Becker <torsten.becker@gmail.com>
#
# Rewrited by gazay

require 'socket'
require 'uri'

class Proxy

  attr_accessor :socket

gazay

require 'sidekiq/cli'
require 'sidekiq/api'

require 'celluloid'

puts Sidekiq::Stats.new.queues
Sidekiq::Queue.new('default').clear
Sidekiq::RetrySet.new.clear

module Count

gazay

# implementation of Range#=== in 1.8.7 - it was the same method as Range#include?:
<<-EOS
static VALUE
range_include(range, val)
    VALUE range, val;
{
    VALUE beg, end;

    beg = rb_ivar_get(range, id_beg);
    end = rb_ivar_get(range, id_end);

gazay

# Sort by memsize from first 50 lines of dump (can be just with `cat` command)
head -50 input.json | jq --slurp '[.[] | {address: .address, type: .type, file: .file, line: .line, method: .method, memsize: .memsize }] | sort_by(.memsize) | map([.address, .type, .file, "\(.line)", .method, "\(.memsize)"] | join(" ")) | reverse' | head -20

# Group objects by type-class-struct-name, count them, count sum of their memsize and sort them by count
# Useful when you didn't trace allocations
cat input_file | jq --slurp '[.[] | {type: "\(.type) \(.class) \(.struct) \(.name)", memsize: .memsize}] | group_by(.type) | map(reduce .[] as $item ({type: .[0].type, memsize: .[0].memsize, count: 0}; {type: .type, memsize: (.memsize + $item.memsize), count: (.count + 1)})) | sort_by(.count) | map(["\(.count)", "\(.memsize)", .type] | join(" : ")) | reverse' > output_file

# Group objects by allocation file-line, count them and their sum of memsize and sort by memsize
cat input_file | jq --slurp '[.[] | {address: .address, type: