Skip to content

Instantly share code, notes, and snippets.

@jrafanie
Created October 20, 2014 15:27
Show Gist options
  • Save jrafanie/dac5e3aa4e58dbc523d5 to your computer and use it in GitHub Desktop.
Save jrafanie/dac5e3aa4e58dbc523d5 to your computer and use it in GitHub Desktop.
Killing postgres connections on edge rails doesn't retry user's requests
1) Backend postgres is killed:
03:27:13 ~/Code/rails42edge_app (2.0.0-p576) + be rails c
Warning: You're using Rubygems 2.0.14 with Spring. Upgrade to at least Rubygems 2.1.0 and run `gem pristine --all` for better startup performance.
Loading development environment (Rails 4.2.0.beta2)
irb(main):001:0> conn = ActiveRecord::Base.connection;
irb(main):002:0* conn.query("SELECT pg_backend_pid()").first.first
(0.2ms) SELECT pg_backend_pid()
=> "99609"
irb(main):003:0> Process.kill("KILL", 99609)
=> 1
irb(main):004:0> conn.tables
PG::ConnectionBad: PQconsumeInput() : SELECT tablename
FROM pg_tables
WHERE schemaname = ANY (current_schemas(false))
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() : SELECT tablename
FROM pg_tables
WHERE schemaname = ANY (current_schemas(false))
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:464:in `block in log'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:458:in `log'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/schema_statements.rb:97:in `tables'
from (irb):4
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/console.rb:110:in `start'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/console.rb:9:in `start'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/commands_tasks.rb:68:in `console'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands.rb:17:in `<top (required)>'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `require'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `block in require'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:233:in `load_dependency'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `require'
from /Users/joerafaniello/Code/rails42edge_app/bin/rails:8:in `<top (required)>'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `load'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `block in load'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:233:in `load_dependency'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `load'
from /Users/joerafaniello/.rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /Users/joerafaniello/.rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
2) Backend process is terminated
irb(main):001:0> conn = ActiveRecord::Base.connection;
irb(main):002:0* spid = conn.query("SELECT pg_backend_pid()").first.first
(0.1ms) SELECT pg_backend_pid()
=> "99805"
irb(main):003:0> conn.tables
=> ["schema_migrations"]
In psql in another terminal:
psql -U rails42edge_app rails42edge_app_development
> SELECT pg_terminate_backend( pid )
FROM pg_stat_activity
WHERE pid <> pg_backend_pid( ) -- 1. don't terminate your own session
AND datname = -- 2. don't terminate connections to
(SELECT datname -- other databases in the cluster
FROM pg_stat_activity
WHERE pid = pg_backend_pid( )
);
Back in console:
irb(main):004:0> conn.tables
FATAL: terminating connection due to administrator command
PG::UnableToSend: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: SELECT tablename
FROM pg_tables
WHERE schemaname = ANY (current_schemas(false))
ActiveRecord::StatementInvalid: PG::UnableToSend: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
: SELECT tablename
FROM pg_tables
WHERE schemaname = ANY (current_schemas(false))
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:464:in `block in log'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:458:in `log'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/schema_statements.rb:97:in `tables'
from (irb):4
Or terminate your current connection:
irb(main):007>sql = <<-SQL
SELECT pg_terminate_backend( pid )
FROM pg_stat_activity;
SQL
irb(main):008:0* conn.query(sql)
(2.3ms) SELECT pg_terminate_backend( pid )
FROM pg_stat_activity;
PG::UnableToSend: FATAL: terminating connection due to administrator command
: SELECT pg_terminate_backend( pid )
FROM pg_stat_activity;
ActiveRecord::StatementInvalid: PG::UnableToSend: FATAL: terminating connection due to administrator command
: SELECT pg_terminate_backend( pid )
FROM pg_stat_activity;
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:464:in `block in log'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:458:in `log'
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query'
from (irb):8
3) Postgres terminates timed out connections somehow....
4) Postgres is starting up... instead of throwing up... wait and retry the query/transaction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment