MySQL server has gone away fix

connection_fix.rb

# If your workers are inactive for a long period of time, they'll lose
# their MySQL connection.
#
# This hack ensures we re-connect whenever a connection is
# lost. Because, really. why not?
#
# Stick this in RAILS_ROOT/config/initializers/connection_fix.rb (or somewhere similar)
#
# From:
#   http://coderrr.wordpress.com/2009/01/08/activerecord-threading-issues-and-resolutions/

module ActiveRecord::ConnectionAdapters
  class MysqlAdapter
    alias_method :execute_without_retry, :execute

    def execute(*args)
      execute_without_retry(*args)
    rescue ActiveRecord::StatementInvalid => e
      if e.message =~ /server has gone away/i
        warn "Server timed out, retrying"
        reconnect!
        retry
      else
        raise e
      end
    end
  end
end

reconnect: true not solve that?

reconnect: true not solve that?

My testing indicates that it does not. I'm using mysql2

My testing indicates that it does not. I'm using mysql2

Do you have any generic code which is DB independent? Like for oracle.

Do you have any generic code which is DB independent? Like for oracle.

It's not working on a Rails 3.0.7 project. It says it can not find execute method to alias. Any ideas?

It's not working on a Rails 3.0.7 project. It says it can not find execute method to alias. Any ideas?

I'm getting a stack level too deep error with this patch and Rails 3.0.7, Ruby 1.9.2-p0.

I'll check the forks for an improved version.

I'm getting a stack level too deep error with this patch and Rails 3.0.7, Ruby 1.9.2-p0.

I'll check the forks for an improved version.

If you're using the mysql2 gem (the current default), you need to change line 13 to the following:

class Mysql2Adapter

If you're using the mysql2 gem (the current default), you need to change line 13 to the following:

class Mysql2Adapter

Hey there, isn't the "reconnect" option on the database config file supposed to do the exact same thing?

Hey there, isn't the "reconnect" option on the database config file supposed to do the exact same thing?

Supposedly reconnect true in the database.yml file will drop other environment configs such as utf8 when it reconnects. Did you ever figure out why the execute method is undefined? Having the same issue.

Supposedly reconnect true in the database.yml file will drop other environment configs such as utf8 when it reconnects. Did you ever figure out why the execute method is undefined? Having the same issue.

Is there any reason to use this in applications using ActiveRecord versions ~> 3.0 when you're taking care to call #verify_active_connections! before accessing the database?

ActiveRecord::Base.connection_handler.verify_active_connections!

I'm just wondering if these kinds of exceptions still crop up for people and reconnect code like this is necessary when the above technique is used to preemptively nuke stale connections from the connection pools?

Is there any reason to use this in applications using ActiveRecord versions ~> 3.0 when you're taking care to call #verify_active_connections! before accessing the database?

ActiveRecord::Base.connection_handler.verify_active_connections!

I'm just wondering if these kinds of exceptions still crop up for people and reconnect code like this is necessary when the above technique is used to preemptively nuke stale connections from the connection pools?

whith mysql2 gem, line 18 becomes : rescue Mysql2::Error => e

whith mysql2 gem, line 18 becomes : rescue Mysql2::Error => e

If someone is gettting a "stack level too deep", it's because Rails is requiring the initializer multiple times. I made a GIST with the correction (and supporting Mysql2 too)

If someone is gettting a "stack level too deep", it's because Rails is requiring the initializer multiple times. I made a GIST with the correction (and supporting Mysql2 too)

@twinge +1for the Mysql2Adapter fix.

@twinge +1for the Mysql2Adapter fix.

I found that this fix caused me a whole lot of trouble with data integrity when I upgraded to Rails 3.1 from 2.3. I don't know what changed between the two versions, but basically, my wait_timeout was set to 10 seconds. In the case that a single transaction took longer than that 10 seconds, the connection timed out returning the "sever has gone away" exception. This monkey patch would then catch that, reconnect, and retry the last statement that caused the exception. However, the first connection, which started the transaction, would have all of its statements rolled back due to the disconnect. So this patch's retry would cause the application to just chug along on subsequent statements not even knowing that the beginning statements had been rolled back. This led to child rows pointing to parent rows that did not even exist in the db (since the parents had been rolled back). I tested this out with a short wait_timeout: 5 in database.yml.

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
end

# ...

User.transaction do
  u = User.create!
  sleep(10)
  p = Post.new
  p.user = u
  p.save!
end

I found that a post row would be created with user_id = 1 even though no user rows existed. Now, when I removed the monkey patch in this gist and instead used reconnect: true, mysql2 seems to correctly restart the transaction and everything works out well.

I think reconnect: true is much safer since the mysql client has pre-defined behavior on what to do when a timeout happens: http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html. Anyway, I hope this helps anyone else who may or may not have been bashing his or her head against the wall trying to figure out why their database integrity is wacky.

I found that this fix caused me a whole lot of trouble with data integrity when I upgraded to Rails 3.1 from 2.3. I don't know what changed between the two versions, but basically, my wait_timeout was set to 10 seconds. In the case that a single transaction took longer than that 10 seconds, the connection timed out returning the "sever has gone away" exception. This monkey patch would then catch that, reconnect, and retry the last statement that caused the exception. However, the first connection, which started the transaction, would have all of its statements rolled back due to the disconnect. So this patch's retry would cause the application to just chug along on subsequent statements not even knowing that the beginning statements had been rolled back. This led to child rows pointing to parent rows that did not even exist in the db (since the parents had been rolled back). I tested this out with a short wait_timeout: 5 in database.yml.

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
end

# ...

User.transaction do
  u = User.create!
  sleep(10)
  p = Post.new
  p.user = u
  p.save!
end

I found that a post row would be created with user_id = 1 even though no user rows existed. Now, when I removed the monkey patch in this gist and instead used reconnect: true, mysql2 seems to correctly restart the transaction and everything works out well.

I think reconnect: true is much safer since the mysql client has pre-defined behavior on what to do when a timeout happens: http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html. Anyway, I hope this helps anyone else who may or may not have been bashing his or her head against the wall trying to figure out why their database integrity is wacky.

@agibralter: I did some reading on reconnect: true and it seems like it doesn't guarantee that the connection encoding will be set to utf8 on reconnection. Have you run into any latin1 character encodings?

@agibralter: I did some reading on reconnect: true and it seems like it doesn't guarantee that the connection encoding will be set to utf8 on reconnection. Have you run into any latin1 character encodings?

@ryansch Interesting -- no I haven't noticed it yet. I guess I could set up an example to test it out though... I'm a bit pressed for time right now though so I may not get around to it for a bit.

@ryansch Interesting -- no I haven't noticed it yet. I guess I could set up an example to test it out though... I'm a bit pressed for time right now though so I may not get around to it for a bit.

We tried the connection fix on a site that is running on Rails2.3.14 with a lot of traffic and run into all kind of strange locking issues on the database. Once we removed the fix, it was all fine again. Can't really explain why this happend.

So.. I have been looking for alternative solutions. I have seen people who suggested to add ActiveRecord::Base.verify_active_connections! to your perform. I used an alternative approach by using the after_fork hook inside resque.rb in the config/initializers folder.

Resque.after_fork = Proc.new { 
  ActiveRecord::Base.verify_active_connections!  
}

This seems to work fine. I wonder why other people are not suggesting this simple approach. Am I missing something?

We tried the connection fix on a site that is running on Rails2.3.14 with a lot of traffic and run into all kind of strange locking issues on the database. Once we removed the fix, it was all fine again. Can't really explain why this happend.

So.. I have been looking for alternative solutions. I have seen people who suggested to add ActiveRecord::Base.verify_active_connections! to your perform. I used an alternative approach by using the after_fork hook inside resque.rb in the config/initializers folder.

Resque.after_fork = Proc.new { 
  ActiveRecord::Base.verify_active_connections!  
}

This seems to work fine. I wonder why other people are not suggesting this simple approach. Am I missing something?

I ended up doing the following.

module BaseJob
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def perform(*args)
      ActiveRecord::Base.verify_active_connections!
    end
  end
end

class FooJob
  include BaseJob

  @queue = :foo_queue

  def self.perform(arg)
    super
    foo_stuff
  end
end

@mcjansen - Your after_fork solution might be a cleaner way to get this done.

I ended up doing the following.

module BaseJob
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def perform(*args)
      ActiveRecord::Base.verify_active_connections!
    end
  end
end

class FooJob
  include BaseJob

  @queue = :foo_queue

  def self.perform(arg)
    super
    foo_stuff
  end
end

@mcjansen - Your after_fork solution might be a cleaner way to get this done.

Turns out there is already a plugin that is using the after_fork approach:
https://github.com/wireframe/resque-ensure-connected

Turns out there is already a plugin that is using the after_fork approach:
https://github.com/wireframe/resque-ensure-connected

I had a bit trouble with the gist when migrating to mysql2 adapter. After checking the rails code I found out that the rails team implemented an AbstractMysqlAdapter so I changed the gist as follows. This should work with mysql and mysql2 adapter, right?

Original

class MysqlAdapter

New

class AbstractMysqlAdapter

I had a bit trouble with the gist when migrating to mysql2 adapter. After checking the rails code I found out that the rails team implemented an AbstractMysqlAdapter so I changed the gist as follows. This should work with mysql and mysql2 adapter, right?

Original

class MysqlAdapter

New

class AbstractMysqlAdapter

Because the resque-ensure-connected gem did not solved my issue when the SQL server goes down a longer time I added this gem. I'm open for discussions: https://github.com/dei79/mysql_stay_connected

Because the resque-ensure-connected gem did not solved my issue when the SQL server goes down a longer time I added this gem. I'm open for discussions: https://github.com/dei79/mysql_stay_connected

This should do the trick:

Resque.after_fork do
  clear_active_connections!
end

This would work both before & after forking the child, since it wouldn't close active connections, but it would check everything back in to the connection pool, and checking back out from the pool verifies the connections.

This should do the trick:

Resque.after_fork do
  clear_active_connections!
end

This would work both before & after forking the child, since it wouldn't close active connections, but it would check everything back in to the connection pool, and checking back out from the pool verifies the connections.

Oops -- that should have been:

Resque.after_fork do
  ActiveRecord::Base.clear_active_connections!
end

Oops -- that should have been:

Resque.after_fork do
  ActiveRecord::Base.clear_active_connections!
end