Created
October 20, 2014 15:27
-
-
Save jrafanie/dac5e3aa4e58dbc523d5 to your computer and use it in GitHub Desktop.
Killing postgres connections on edge rails doesn't retry user's requests
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1) Backend postgres is killed: | |
03:27:13 ~/Code/rails42edge_app (2.0.0-p576) + be rails c | |
Warning: You're using Rubygems 2.0.14 with Spring. Upgrade to at least Rubygems 2.1.0 and run `gem pristine --all` for better startup performance. | |
Loading development environment (Rails 4.2.0.beta2) | |
irb(main):001:0> conn = ActiveRecord::Base.connection; | |
irb(main):002:0* conn.query("SELECT pg_backend_pid()").first.first | |
(0.2ms) SELECT pg_backend_pid() | |
=> "99609" | |
irb(main):003:0> Process.kill("KILL", 99609) | |
=> 1 | |
irb(main):004:0> conn.tables | |
PG::ConnectionBad: PQconsumeInput() : SELECT tablename | |
FROM pg_tables | |
WHERE schemaname = ANY (current_schemas(false)) | |
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() : SELECT tablename | |
FROM pg_tables | |
WHERE schemaname = ANY (current_schemas(false)) | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:464:in `block in log' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/notifications/instrumenter.rb:20:in `instrument' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:458:in `log' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/schema_statements.rb:97:in `tables' | |
from (irb):4 | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/console.rb:110:in `start' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/console.rb:9:in `start' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/commands_tasks.rb:68:in `console' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands/commands_tasks.rb:39:in `run_command!' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/railties-4.2.0.beta2/lib/rails/commands.rb:17:in `<top (required)>' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `require' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `block in require' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:233:in `load_dependency' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `require' | |
from /Users/joerafaniello/Code/rails42edge_app/bin/rails:8:in `<top (required)>' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `load' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `block in load' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:233:in `load_dependency' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `load' | |
from /Users/joerafaniello/.rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require' | |
from /Users/joerafaniello/.rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require' | |
2) Backend process is terminated | |
irb(main):001:0> conn = ActiveRecord::Base.connection; | |
irb(main):002:0* spid = conn.query("SELECT pg_backend_pid()").first.first | |
(0.1ms) SELECT pg_backend_pid() | |
=> "99805" | |
irb(main):003:0> conn.tables | |
=> ["schema_migrations"] | |
In psql in another terminal: | |
psql -U rails42edge_app rails42edge_app_development | |
> SELECT pg_terminate_backend( pid ) | |
FROM pg_stat_activity | |
WHERE pid <> pg_backend_pid( ) -- 1. don't terminate your own session | |
AND datname = -- 2. don't terminate connections to | |
(SELECT datname -- other databases in the cluster | |
FROM pg_stat_activity | |
WHERE pid = pg_backend_pid( ) | |
); | |
Back in console: | |
irb(main):004:0> conn.tables | |
FATAL: terminating connection due to administrator command | |
PG::UnableToSend: server closed the connection unexpectedly | |
This probably means the server terminated abnormally | |
before or while processing the request. | |
: SELECT tablename | |
FROM pg_tables | |
WHERE schemaname = ANY (current_schemas(false)) | |
ActiveRecord::StatementInvalid: PG::UnableToSend: server closed the connection unexpectedly | |
This probably means the server terminated abnormally | |
before or while processing the request. | |
: SELECT tablename | |
FROM pg_tables | |
WHERE schemaname = ANY (current_schemas(false)) | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:464:in `block in log' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/notifications/instrumenter.rb:20:in `instrument' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:458:in `log' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/schema_statements.rb:97:in `tables' | |
from (irb):4 | |
Or terminate your current connection: | |
irb(main):007>sql = <<-SQL | |
SELECT pg_terminate_backend( pid ) | |
FROM pg_stat_activity; | |
SQL | |
irb(main):008:0* conn.query(sql) | |
(2.3ms) SELECT pg_terminate_backend( pid ) | |
FROM pg_stat_activity; | |
PG::UnableToSend: FATAL: terminating connection due to administrator command | |
: SELECT pg_terminate_backend( pid ) | |
FROM pg_stat_activity; | |
ActiveRecord::StatementInvalid: PG::UnableToSend: FATAL: terminating connection due to administrator command | |
: SELECT pg_terminate_backend( pid ) | |
FROM pg_stat_activity; | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `async_exec' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:147:in `block in query' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:464:in `block in log' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activesupport-4.2.0.beta2/lib/active_support/notifications/instrumenter.rb:20:in `instrument' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/abstract_adapter.rb:458:in `log' | |
from /Users/joerafaniello/.gem/ruby/2.0.0/gems/activerecord-4.2.0.beta2/lib/active_record/connection_adapters/postgresql/database_statements.rb:146:in `query' | |
from (irb):8 | |
3) Postgres terminates timed out connections somehow.... | |
4) Postgres is starting up... instead of throwing up... wait and retry the query/transaction | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment