RBX crash

gistfile1.txt

boocx@ip-10-71-58-116:~/boocx-api$ APP_ENV=production bundle exec rake sidekiq:workers
bundle exec sidekiq -c 50 -l logs/sidekiq.log -q main -v -r ./lib/loader.rb -e production
2013-01-22 20:03:22 +0000: Reconnecting from timeout.
Network Error: eventmachine not initialized: evma_set_pending_connect_timeout
E, [2013-01-22T20:03:22.863555 #28988] ERROR -- : Sidekiq::Processor crashed!
AWS::DynamoDB::Errors::InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/

content-length:240
content-type:application/x-amz-json-1.0
host:dynamodb.us-east-1.amazonaws.com
user-agent:aws-sdk-ruby/1.7.1 rbx/1.9.3 i686-pc-linux-gnu
x-amz-date:20130122T200321Z
x-amz-target:DynamoDB_20111205.UpdateItem

content-length;content-type;host;user-agent;x-amz-date;x-amz-target
aceeebe98fcf80a7adad199d1d49aa808d66379a6d875a8c97df7f3465132d55'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20130122T200321Z
20130122/us-east-1/dynamodb/aws4_request
93c0d2f38123a8c02a20ea936bf94446c34ea24c490d415c52fafe36d4020c70'

/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/core/client.rb:318:in `return_or_raise'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/core/client.rb:419:in `client_request'
(eval):3:in `update_item'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/dynamo_db/attribute_collection.rb:451:in `do_updates'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/aws-sdk-1.7.1/lib/aws/dynamo_db/attribute_collection.rb:387:in `update'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account_line_list.rb:12:in `append'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:58:in `perform_post'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/bfg-redis-lock-0.0.1/lib/redis-lock.rb:37:in `lock'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/bfg-redis-lock-0.0.1/lib/redis-lock.rb:189:in `lock'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/redis-namespace-1.2.1/lib/redis/namespace.rb:220:in `lock (method_missing)'
/home/boocx/boocx-api/lib/boocx/redis_lock.rb:8:in `lock'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/connection_pool-0.9.2/lib/connection_pool.rb:49:in `with'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq.rb:67:in `redis'
/home/boocx/boocx-api/lib/boocx/redis_lock.rb:7:in `lock'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:132:in `with_lock'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:57:in `perform_post'
/home/boocx/boocx-api/boocx-core/lib/boocx/persistence/optimistic_locking.rb:32:in `stale_object_retry'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/account.rb:56:in `perform_post'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/line_post.rb:7:in `perform'
/home/boocx/boocx-api/boocx-core/lib/boocx/persistence/dynamo_db_storage.rb:11:in `with_consistent_reads'
/home/boocx/boocx-api/boocx-core/lib/boocx/persistence/persistable.rb:26:in `with_consistent_reads'
/home/boocx/boocx-api/boocx-core/lib/boocx/core/line_post.rb:4:in `perform'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:43:in `process'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:79:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-failures-0.1.0/lib/sidekiq/failures/middleware.rb:8:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/server/unique_jobs.rb:9:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/timeout.rb:14:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/retry_jobs.rb:49:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/logging.rb:11:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/logging.rb:22:in `with_context'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:81:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/middleware/chain.rb:84:in `invoke'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:42:in `process'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:77:in `stats'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/sidekiq-2.6.0/lib/sidekiq/processor.rb:41:in `process'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in `dispatch'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/celluloid-0.12.4/lib/celluloid/future.rb:18:in `initialize'
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in `create'
kernel/bootstrap/thread19.rb:41:in `__run__'

---------------------------------------------
CRASH: A fatal error has occurred.

Backtrace:
ruby[0x81d5ea4]
[0xb779e400]
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(+0x7614c)[0xb74d614c]
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(+0x7722f)[0xb74d722f]
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6(__libc_malloc+0x6c)[0xb74d9ddc]
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(+0x87fb)[0xb582b7fb]
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(yajl_gen_alloc2+0x4e)[0xb5826cce]
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(yajl_gen_alloc+0x33)[0xb5826eb3]
/home/boocx/.rvm/gems/rbx-head@boocx-api/gems/yajl-ruby-1.1.0/lib/yajl/yajl.so(+0x64ef)[0xb58294ef]
ruby[0x830d0b2]
ruby(_ZN8rubinius12NativeMethod23executor_implementationINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x2fa)[0x830e28a]
ruby(_ZN8rubinius11InlineCache11check_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0xac)[0x81e465c]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x22f)[0x824517f]
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x22f)[0x824517f]
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_16GenericArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x22f)[0x824517f]
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_11OneArgumentEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x27a)[0x824668a]
ruby(_ZN8rubinius11InlineCache21check_cache_referenceEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x8c)[0x81e44cc]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1090)[0x8201270]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_17SplatOnlyArgumentEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x273)[0x8246af3]
ruby(_ZN8rubinius11InlineCache11empty_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x129)[0x81e5829]
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x6c)[0x81e484c]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509]
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5]
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f]
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49]
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_14ThreeArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x292)[0x8245da2]
ruby(_ZN8rubinius11InlineCache11empty_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x129)[0x81e5829]
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x6c)[0x81e484c]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509]
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5]
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f]
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49]
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_17SplatOnlyArgumentEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x273)[0x8246af3]
ruby(_ZN8rubinius11InlineCache11empty_cacheEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x129)[0x81e5829]
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x6c)[0x81e484c]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509]
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5]
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f]
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49]
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a]
ruby(_ZN8rubinius11MachineCode19execute_specializedINS_14ThreeArgumentsEEEPNS_6ObjectEPNS_5StateEPNS_9CallFrameEPNS_10ExecutableEPNS_6ModuleERNS_9ArgumentsE+0x292)[0x8245da2]
ruby(_ZN8rubinius11InlineCache16check_cache_polyEPNS_5StateEPS0_PNS_9CallFrameERNS_9ArgumentsE+0x121)[0x81e4901]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x1329)[0x8201509]
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5]
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f]
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49]
ruby(_ZN8rubinius4Proc5yieldEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsE+0x122)[0x8330032]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x396a)[0x8203b4a]
ruby(_ZN8rubinius16BlockEnvironment19execute_interpreterEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x1f5)[0x82d66c5]
ruby(_ZN8rubinius16BlockEnvironment6invokeEPNS_5StateEPNS_9CallFrameEPS0_RNS_9ArgumentsERNS_15BlockInvocationE+0x6f)[0x82d6e0f]
ruby(_ZN8rubinius16BlockEnvironment4callEPNS_5StateEPNS_9CallFrameERNS_9ArgumentsEi+0x59)[0x82d6f49]
ruby(_ZN8rubinius11MachineCode11interpreterEPNS_5StateEPS0_PNS_20InterpreterCallFrameE+0x51ed)[0x82053cd]


Wrote full error report to: /home/boocx/.rbx/rubinius_last_error_28988
Run 'rbx report' to submit this crash report!
rake aborted!
Command failed with status (100): [bundle exec sidekiq -c 50 -l logs/sidekiq....]
/home/boocx/boocx-api/lib/tasks/sidekiq.rake:10:in `__script__'
kernel/bootstrap/array.rb:68:in `each'
kernel/bootstrap/array.rb:68:in `each'
kernel/common/kernel.rb:598:in `load'
kernel/common/block_environment.rb:75:in `call_on_instance'
kernel/common/eval.rb:75:in `eval'
/home/boocx/.rvm/gems/rbx-head@boocx-api/bin/ruby_noexec_wrapper:14:in `__script__'
kernel/delta/codeloader.rb:68:in `load_script'
kernel/delta/codeloader.rb:118:in `load_script'
kernel/loader.rb:615:in `script'
kernel/loader.rb:816:in `main'
Tasks: TOP => sidekiq:workers
(See full trace by running task with --trace)
boocx@ip-10-71-58-116:~/boocx-api$ 

This doesn't look like a sidekiq problem. Looks like eventmachine choked on something. Rubinius certainly should not have crashed. What version of rbx are you running? (rbx -v)

And is this 1.8 or 1.9 mode?

The memory violation appears to happen inside yajl doing a malloc call. Until the EM issues is addressed, it's not easy to determine whether yajl got bad data or if there's some issue with Rubinius C-API that yajl is using.