Created
January 22, 2013 20:04
-
-
Save jgarber/4597934 to your computer and use it in GitHub Desktop.
RBX crash
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boocx@ip-10-71-58-116:~/boocx-api$ APP_ENV=production bundle exec rake sidekiq:workers | |
bundle exec sidekiq -c 50 -l logs/sidekiq.log -q main -v -r ./lib/loader.rb -e production | |
2013-01-22 20:03:22 +0000: Reconnecting from timeout. | |
Network Error: eventmachine not initialized: evma_set_pending_connect_timeout | |
E, [2013-01-22T20:03:22.863555 #28988] ERROR -- : Sidekiq::Processor crashed! | |
AWS::DynamoDB::Errors::InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. | |
The Canonical String for this request should have been | |
'POST | |
/ | |
content-length:240 | |
content-type:application/x-amz-json-1.0 | |
host:dynamodb.us-east-1.amazonaws.com | |
user-agent:aws-sdk-ruby/1.7.1 rbx/1.9.3 i686-pc-linux-gnu | |
x-amz-date:20130122T200321Z | |
x-amz-target:DynamoDB_20111205.UpdateItem | |
content-length;content-type;host;user-agent;x-amz-date;x-amz-target | |
aceeebe98fcf80a7adad199d1d49aa808d66379a6d875a8c97df7f3465132d55' | |
The String-to-Sign should have been | |
'AWS4-HMAC-SHA256 | |
20130122T200321Z | |
20130122/us-east-1/dynamodb/aws4_request | |
93c0d2f38123a8c02a20ea936bf94446c34ea24c490d415c52fafe36d4020c70' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/core/client.rb:318:in `return_or_raise' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/core/client.rb:419:in `client_request' | |
(eval):3:in `update_item' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/dynamo_db/attribute_collection.rb:451:in `do_updates' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/dynamo_db/attribute_collection.rb:387:in `update' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account_line_list.rb:12:in `append' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:58:in `perform_post' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/bfg-redis-lock-0.0.1/lib/redis-lock.rb:37:in `lock' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/bfg-redis-lock-0.0.1/lib/redis-lock.rb:189:in `lock' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/redis-namespace-1.2.1/lib/redis/namespace.rb:220:in `lock (method_missing)' | |
/home/boocx/boocx-api/lib/boocx/redis_lock.rb:8:in `lock' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/connection_pool-0.9.2/lib/connection_pool.rb:49:in `with' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq.rb:67:in `redis' | |
/home/boocx/boocx-api/lib/boocx/redis_lock.rb:7:in `lock' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:132:in `with_lock' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:57:in `perform_post' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/persistence/optimistic_locking.rb:32:in `stale_object_retry' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:56:in `perform_post' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/line_post.rb:7:in `perform' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/persistence/dynamo_db_storage.rb:11:in `with_consistent_reads' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/persistence/persistable.rb:26:in `with_consistent_reads' | |
/home/boocx/boocx-api/boocx-core/lib/boocx/core/line_post.rb:4:in `perform' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:43:in `process' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:79:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-failures-0.1.0/lib/sidekiq/failures/middleware.rb:8:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/server/unique_jobs.rb:9:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/timeout.rb:14:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/retry_jobs.rb:49:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/logging.rb:11:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/logging.rb:22:in `with_context' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/logging.rb:7:in `call' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:84:in `invoke' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:42:in `process' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:77:in `stats' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:41:in `process' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in `dispatch' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/celluloid-0.12.4/lib/celluloid/future.rb:18:in `initialize' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in `create' | |
kernel/bootstrap/thread19.rb:41:in `__run__' | |
--------------------------------------------- | |
CRASH: A fatal error has occurred. | |
Backtrace: | |
ruby[0x81d5ea4] | |
[0xb779e400] | |
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(+0x7614c)[0xb74d614c] | |
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(+0x7722f)[0xb74d722f] | |
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(__libc_malloc+0x6c)[0xb74d9ddc] | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(+0x87fb)[0xb582b7fb] | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(yajl_gen_alloc2+0x4e)[0xb5826cce] | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(yajl_gen_alloc+0x33)[0xb5826eb3] | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(+0x64ef)[0xb58294ef] | |
ruby[0x830d0b2] | |
ruby(_ZN8rubinius12NativeMethod23executor_implementationINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x2fa)[0x830e28a] | |
ruby(_ZN8rubinius11InlineCache11check_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0xac)[0x81e465c] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x22f)[0x824517f] | |
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x22f)[0x824517f] | |
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x22f)[0x824517f] | |
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_11OneArgumentEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x27a)[0x824668a] | |
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_17SplatOnlyArgumentEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x273)[0x8246af3] | |
ruby(_ZN8rubinius11InlineCache11empty_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x129)[0x81e5829] | |
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x6c)[0x81e484c] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509] | |
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5] | |
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f] | |
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49] | |
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_14ThreeArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x292)[0x8245da2] | |
ruby(_ZN8rubinius11InlineCache11empty_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x129)[0x81e5829] | |
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x6c)[0x81e484c] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509] | |
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5] | |
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f] | |
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49] | |
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_17SplatOnlyArgumentEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x273)[0x8246af3] | |
ruby(_ZN8rubinius11InlineCache11empty_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x129)[0x81e5829] | |
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x6c)[0x81e484c] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509] | |
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5] | |
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f] | |
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49] | |
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a] | |
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_14ThreeArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x292)[0x8245da2] | |
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x121)[0x81e4901] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509] | |
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5] | |
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f] | |
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49] | |
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a] | |
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5] | |
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f] | |
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49] | |
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x51ed)[0x82053cd] | |
Wrote full error report to: /home/boocx/.rbx/rubinius_last_error_28988 | |
Run 'rbx report' to submit this crash report! | |
rake aborted! | |
Command failed with status (100): [bundle exec sidekiq -c 50 -l logs/sidekiq....] | |
/home/boocx/boocx-api/lib/tasks/sidekiq.rake:10:in `__script__' | |
kernel/bootstrap/array.rb:68:in `each' | |
kernel/bootstrap/array.rb:68:in `each' | |
kernel/common/kernel.rb:598:in `load' | |
kernel/common/block_environment.rb:75:in `call_on_instance' | |
kernel/common/eval.rb:75:in `eval' | |
/home/boocx/.rvm/gems/rbx-head@boocx-api/bin/ruby_noexec_wrapper:14:in `__script__' | |
kernel/delta/codeloader.rb:68:in `load_script' | |
kernel/delta/codeloader.rb:118:in `load_script' | |
kernel/loader.rb:615:in `script' | |
kernel/loader.rb:816:in `main' | |
Tasks: TOP => sidekiq:workers | |
(See full trace by running task with --trace) | |
boocx@ip-10-71-58-116:~/boocx-api$ |
The memory violation appears to happen inside yajl doing a malloc call. Until the EM issues is addressed, it's not easy to determine whether yajl got bad data or if there's some issue with Rubinius C-API that yajl is using.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This doesn't look like a sidekiq problem. Looks like eventmachine choked on something. Rubinius certainly should not have crashed. What version of rbx are you running? (rbx -v)
And is this 1.8 or 1.9 mode?