CF custom buildpack failing to stage "Staging error: no available stagers"

CF_Apps_Not_Staging_No_Available_Stagers.md

Issue

After installing a new CF in AWS I am able to push and stage my first test app but not subsequent apps.

Failing environment:

• AWS
• CF[140|144|146]
• Stemcell 1226

Debugging

This is the end of the output from running:

$ cf push --buildpack=https://github.com/dmikusa-pivotal/cf-test-buildpack.git


REQUEST: PUT http://api.ft4.cpgpaas.net/v2/apps/80c0c3da-e21e-4c36-aafc-e3e75b9fcd99/bits
REQUEST_HEADERS:
  Authorization : [PRIVATE DATA HIDDEN]
REQUEST_BODY: {:resources=>"[]", :application=>#<UploadIO:0x007f8c2f1bf5b8 @content_type="application/zip", @original_filename="80c0c3da-e21e-4c36-aafc-e3e75b9fcd99.zip", @local_path="/var/folders/6g/hqkzb3ln3r79nqqf_9610ldc0000gn/T/80c0c3da-e21e-4c36-aafc-e3e75b9fcd99.zip", @io=#<File:/var/folders/6g/hqkzb3ln3r79nqqf_9610ldc0000gn/T/80c0c3da-e21e-4c36-aafc-e3e75b9fcd99.zip>, @opts={}>}
...RESPONSE: [201]
RESPONSE_HEADERS:
  connection : keep-alive
  content-length : 0
  content-type : application/json;charset=utf-8
  date : Mon, 04 Nov 2013 23:28:36 GMT
  server : nginx
  x-content-type-options : nosniff
  x-vcap-request-id : 68136c87-86a7-4bf9-a646-f7946e36e196
RESPONSE_BODY:
<<<
... OK
Preparing to start testapp1>>>
.  REQUEST: PUT http://api.ft4.cpgpaas.net/v2/apps/80c0c3da-e21e-4c36-aafc-e3e75b9fcd99
REQUEST_HEADERS:
  Authorization : [PRIVATE DATA HIDDEN]
  Content-Length : 34
  Content-Type : application/json
REQUEST_BODY: {"console":true,"state":"STARTED"}
.. RESPONSE: [400]
RESPONSE_HEADERS:
  connection : keep-alive
  content-length : 68
  content-type : application/json;charset=utf-8
  date : Mon, 04 Nov 2013 23:28:37 GMT
  server : nginx
  x-content-type-options : nosniff
  x-vcap-request-id : 956ae2b1-c872-4957-b42c-9b4be8050b42
RESPONSE_BODY:
{
  "code": 170001,
  "description": "Staging error: no available stagers"
}
<<<
... FAILED
Application failed to stage

Solution

I was not allocating enough memory to the DEA process on the DEA server. I was only allocating 1500Mb of the available 8Gb and hence there was not enough RAM to stage apps.

The following was causing me issues:

  dea_next:
    memory_mb: 1500
    memory_overcommit_factor: 1
    disk_mb: 16384
    disk_overcommit_factor: 4
    num_instances: 256

I needed to change the property:

    memory_mb: 1500

To utilise the full memory available on the DEA then re-deploy with BOSH.

See the VCAP-DEV mailing list post here: https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/T2o_joQn490/frDud1yh6cAJ

cloud_controller_ng.log

...
{"timestamp":1383598575.4598558,"message":"cc.update","log_level":"debug","source":"cc.api","data":{"request_guid":"b7901f3e-98d4-453f-8fee-b51e974176ef","guid":"26ca9f93-c3ce-4dbf-82b5-5b3720cec519","attributes":{"console":true,"state":"STARTED"}},"thread_id":45367600,"fiber_id":48692440,"process_id":7817,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/runtime/apps_controller.rb","lineno":84,"method":"update"}

{"timestamp":1383598575.5110884,"message":"Request failed with response code: 400 error code: 170001 error: Staging error: no available stagers","log_level":"debug","source":"cc.api","data":{"request_guid":"b7901f3e-98d4-453f-8fee-b51e974176ef"},"thread_id":45367600,"fiber_id":48692440,"process_id":7817,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sinatra/vcap.rb","lineno":66,"method":"block in registered"}

{"timestamp":1383598576.8734663,"message":"Sending registration: {:host=>\"172.31.0.109\", :port=>9022, :uris=>[\"api.ft4.cpgpaas.net\", \"api.ft4.cpgpaas.net\"], :tags=>{:component=>\"CloudController\"}, :index=>0, :private_instance_id=>nil}","log_level":"debug","source":"cf.registrar","data":{},"thread_id":19701420,"fiber_id":35300180,"process_id":7817,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1
...

stacktrace.log

backtrace":["/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_stager_task.rb:44:in `stage'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:61:in `stage_app'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:66:in `stage_if_needed'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:74:in `react_to_state_change'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:28:in `updated'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:474:in `after_commit'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/model/base.rb:1762:in `block in _save'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `block in after_transaction_commit'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `each'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `after_transaction_commit'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:294:in `remove_transaction'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:151:in `_transaction'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:103:in `block in transaction'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/connecting.rb:235:in `block in synchronize'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/connecting.rb:235:in `synchronize'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:96:in `transaction'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/runtime/apps_controller.rb:71:in `update'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/base.rb:96:in `dispatch'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `block in compile!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:966:in `route_eval'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `catch'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:948:in `block in route!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `block in call!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'",
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'"]}

terminal.log

Preparing to start bp-test-1>>>
REQUEST: PUT http://api.ft4.cpgpaas.net/v2/apps/ea4e229e-23d0-4dcd-a7fb-a35c9236cee3
REQUEST_HEADERS:
  Authorization : [PRIVATE DATA HIDDEN]
  Content-Length : 34
  Content-Type : application/json
REQUEST_BODY: {"console":true,"state":"STARTED"}
...RESPONSE: [400]
RESPONSE_HEADERS:
  connection : keep-alive
  content-length : 9785
  content-type : application/json;charset=utf-8
  date : Tue, 05 Nov 2013 02:17:58 GMT
  server : nginx
  via : 1.1 sjck-dmz-wsa-2.cisco.com:80 (Cisco-IronPort-WSA/7.5.2-118)
  x-content-type-options : nosniff
  x-vcap-request-id : 761d5e40-8d67-4523-9884-ae6ead0b566e
RESPONSE_BODY:
{"code":170001,"description":"Staging error: no available stagers","error_code":"CF-StagingError","types":["StagingError","Error"],"backtrace":["/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_stager_task.rb:44:in `stage'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:61:in `stage_app'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:66:in `stage_if_needed'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:74:in `react_to_state_change'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:28:in `updated'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:474:in `after_commit'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/model/base.rb:1762:in `block in _save'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `block in after_transaction_commit'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `each'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:182:in `after_transaction_commit'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:294:in `remove_transaction'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:151:in `_transaction'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:103:in `block in transaction'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/connecting.rb:235:in `block in synchronize'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/connecting.rb:235:in `synchronize'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-3.48.0/lib/sequel/database/transactions.rb:96:in `transaction'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/runtime/apps_controller.rb:71:in `update'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/base.rb:96:in `dispatch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `block in compile!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:966:in `route_eval'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:948:in `block in route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `block in call!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/commonlogger.rb:33:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:212:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:81:in `block in pre_process'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `catch'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `pre_process'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `call'","/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `block in spawn_threadpool'"]}

Issue resolved, updated Gist with more accurate error details and the solution.