Skip to content

Instantly share code, notes, and snippets.

@defunkt
Created November 19, 2009 20:00
  • Star 76 You must be signed in to star a gist
  • Fork 44 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save defunkt/238999 to your computer and use it in GitHub Desktop.
MySQL server has gone away fix
# If your workers are inactive for a long period of time, they'll lose
# their MySQL connection.
#
# This hack ensures we re-connect whenever a connection is
# lost. Because, really. why not?
#
# Stick this in RAILS_ROOT/config/initializers/connection_fix.rb (or somewhere similar)
#
# From:
# http://coderrr.wordpress.com/2009/01/08/activerecord-threading-issues-and-resolutions/
module ActiveRecord::ConnectionAdapters
class MysqlAdapter
alias_method :execute_without_retry, :execute
def execute(*args)
execute_without_retry(*args)
rescue ActiveRecord::StatementInvalid => e
if e.message =~ /server has gone away/i
warn "Server timed out, retrying"
reconnect!
retry
else
raise e
end
end
end
end
@raphaelcosta
Copy link

reconnect: true not solve that?

@shwoodard
Copy link

My testing indicates that it does not. I'm using mysql2

@tispratik
Copy link

Do you have any generic code which is DB independent? Like for oracle.

@nikosd
Copy link

nikosd commented Jun 3, 2011

It's not working on a Rails 3.0.7 project. It says it can not find execute method to alias. Any ideas?

@etagwerker
Copy link

I'm getting a stack level too deep error with this patch and Rails 3.0.7, Ruby 1.9.2-p0.

I'll check the forks for an improved version.

@twinge
Copy link

twinge commented Jul 18, 2011

If you're using the mysql2 gem (the current default), you need to change line 13 to the following:

class Mysql2Adapter

@nikosd
Copy link

nikosd commented Jul 22, 2011

Hey there, isn't the "reconnect" option on the database config file supposed to do the exact same thing?

@ScotterC
Copy link

Supposedly reconnect true in the database.yml file will drop other environment configs such as utf8 when it reconnects. Did you ever figure out why the execute method is undefined? Having the same issue.

@zettabyte
Copy link

Is there any reason to use this in applications using ActiveRecord versions ~> 3.0 when you're taking care to call #verify_active_connections! before accessing the database?

ActiveRecord::Base.connection_handler.verify_active_connections!

I'm just wondering if these kinds of exceptions still crop up for people and reconnect code like this is necessary when the above technique is used to preemptively nuke stale connections from the connection pools?

@aimerickdesdoit
Copy link

whith mysql2 gem, line 18 becomes : rescue Mysql2::Error => e

@mauricioszabo
Copy link

If someone is gettting a "stack level too deep", it's because Rails is requiring the initializer multiple times. I made a GIST with the correction (and supporting Mysql2 too)

@jspooner
Copy link

@twinge +1for the Mysql2Adapter fix.

@agibralter
Copy link

I found that this fix caused me a whole lot of trouble with data integrity when I upgraded to Rails 3.1 from 2.3. I don't know what changed between the two versions, but basically, my wait_timeout was set to 10 seconds. In the case that a single transaction took longer than that 10 seconds, the connection timed out returning the "sever has gone away" exception. This monkey patch would then catch that, reconnect, and retry the last statement that caused the exception. However, the first connection, which started the transaction, would have all of its statements rolled back due to the disconnect. So this patch's retry would cause the application to just chug along on subsequent statements not even knowing that the beginning statements had been rolled back. This led to child rows pointing to parent rows that did not even exist in the db (since the parents had been rolled back). I tested this out with a short wait_timeout: 5 in database.yml.

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
end

# ...

User.transaction do
  u = User.create!
  sleep(10)
  p = Post.new
  p.user = u
  p.save!
end

I found that a post row would be created with user_id = 1 even though no user rows existed. Now, when I removed the monkey patch in this gist and instead used reconnect: true, mysql2 seems to correctly restart the transaction and everything works out well.

I think reconnect: true is much safer since the mysql client has pre-defined behavior on what to do when a timeout happens: http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html. Anyway, I hope this helps anyone else who may or may not have been bashing his or her head against the wall trying to figure out why their database integrity is wacky.

@ryansch
Copy link

ryansch commented Feb 20, 2012

@agibralter: I did some reading on reconnect: true and it seems like it doesn't guarantee that the connection encoding will be set to utf8 on reconnection. Have you run into any latin1 character encodings?

@agibralter
Copy link

@ryansch Interesting -- no I haven't noticed it yet. I guess I could set up an example to test it out though... I'm a bit pressed for time right now though so I may not get around to it for a bit.

@mcjansen
Copy link

We tried the connection fix on a site that is running on Rails2.3.14 with a lot of traffic and run into all kind of strange locking issues on the database. Once we removed the fix, it was all fine again. Can't really explain why this happend.

So.. I have been looking for alternative solutions. I have seen people who suggested to add ActiveRecord::Base.verify_active_connections! to your perform. I used an alternative approach by using the after_fork hook inside resque.rb in the config/initializers folder.

Resque.after_fork = Proc.new { 
  ActiveRecord::Base.verify_active_connections!  
}

This seems to work fine. I wonder why other people are not suggesting this simple approach. Am I missing something?

@ryansch
Copy link

ryansch commented Apr 13, 2012

I ended up doing the following.

module BaseJob
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def perform(*args)
      ActiveRecord::Base.verify_active_connections!
    end
  end
end
class FooJob
  include BaseJob

  @queue = :foo_queue

  def self.perform(arg)
    super
    foo_stuff
  end
end

@mcjansen - Your after_fork solution might be a cleaner way to get this done.

@mcjansen
Copy link

Turns out there is already a plugin that is using the after_fork approach:
https://github.com/wireframe/resque-ensure-connected

@dei79
Copy link

dei79 commented Sep 1, 2012

I had a bit trouble with the gist when migrating to mysql2 adapter. After checking the rails code I found out that the rails team implemented an AbstractMysqlAdapter so I changed the gist as follows. This should work with mysql and mysql2 adapter, right?

Original

class MysqlAdapter

New

class AbstractMysqlAdapter

@dei79
Copy link

dei79 commented Sep 1, 2012

Because the resque-ensure-connected gem did not solved my issue when the SQL server goes down a longer time I added this gem. I'm open for discussions: https://github.com/dei79/mysql_stay_connected

@trevorturk
Copy link

This should do the trick:

Resque.after_fork do
  clear_active_connections!
end

This would work both before & after forking the child, since it wouldn't close active connections, but it would check everything back in to the connection pool, and checking back out from the pool verifies the connections.

@trevorturk
Copy link

Oops -- that should have been:

Resque.after_fork do
  ActiveRecord::Base.clear_active_connections!
end

@joshuapinter
Copy link

For anyone running into this while forking in Rails, try clearing the existing connections before forking and then establish a new connection for each fork, like this:

# Clear existing connections before forking to ensure they do not get inherited.
::ActiveRecord::Base.clear_all_connections! 

fork do
  # Establish a new connection for each fork.
  ::ActiveRecord::Base.establish_connection 
  
  # The rest of the code for each fork...
end

See this StackOverflow answer here: https://stackoverflow.com/a/8915353/293280

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment