TorqueBox on Heroku.md

With Heroku's new JRuby support you may have already seen that you can run TorqueBox Lite on Heroku. But, that only gives you the web features of TorqueBox. What about scheduled jobs, backgroundable, messaging, services, and caching?

With a small amount of extra work, you can now run the full TorqueBox (minus STOMP support and clustering) on Heroku as well! I've successfully deployed several test applications, including the example Rails application from our Getting Started Guide which has a scheduled job, a service, and uses backgroundable and messaging.

The version of TorqueBox you'll be running on Heroku is almost identical (two commits newer) than TorqueBox 2.2.0. We had to expose a small extra flag to control when TorqueBox actually binds to the HTTP port since Heroku uses that as the trigger that your application is up and ready to serve requests.

Steps Required

1. Create a JRuby application on Heroku, or convert an existing application to JRuby. Make sure your application works on JRuby on Heroku before throwing TorqueBox into the mix.
2. Save the standalone.xml from this gist into your application's root directory.
3. Add the TorqueBox incremental gem repository and the torquebox-server gem (with the same incremental version used here) to your Gemfile, as shown in the Gemfile example in this gist.
4. Replace the web: entry in your Procfile with the hilariously long one shown in this gist.
5. Add the new standalone.xml, changed Gemfile, and changed Procfile to git, commit, and push the changes.
6. The first time Heroku attempts to bundle install it will likely run out of memory. Our torquebox-server gem is a bit on the bloated side, but if it runs out of memory just git push again and things should install this time. You shouldn't run out of memory during a bundle install again unless you change the torquebox-server gem version.

If all goes well after the git push your application should boot up and be running on TorqueBox. I'd suggest you use heroku logs -t during the push to keep an eye on things. If you run into any errors while TorqueBox is starting, get in touch with us via @torquebox Twitter, #torquebox on irc.freenode.net, or any other method listed on our community page and we'll help you figure things out. A list of the most common things that could go wrong and how to fix them are below.

Potential Gotchas

Slug Size

Your slug size will likely be close to the limit. Most of my Rails application slugs were in the 190MB range, which is dangerously close to Heroku's 200MB limit. We could probably shave 20MB, probably more, off the size brought in by TorqueBox if we release a slightly customized version of the torquebox-server gem just for running on Heroku. If there's enough demand, we'll look into doing that.

If you hit this issue and are feeling adventerous, it should be possible to create a .slugignore file and play around with excluding some things from the torquebox-server gem. Entries like vendor/bundle/jruby/1.9/gems/torquebox-server-2.x.incremental.1295-java/jboss/docs, vendor/bundle/jruby/1.9/gems/torquebox-server-2.x.incremental.1295-java/jboss/bin/client should be unnecessary and shave off another 6-7MB. There are larger savings to be had by excluding some of the bundled JBoss modules, but that's a lot more work to know exactly which modules are required and which are not.

Out of Memory Errors

If you get an out of memory error in the PermGen space, bump up the value for "-XX:MaxPermSize=128m" in the web: Procfile entry. If you get an out of memory error in the Heap, bump up the value for "-Xmx256m" in the web: Procfile entry. The defaults we're using are probably a bit too conservative, but I wanted to make sure we don't exceed Heroku's soft cap of 512MB for most applications. There's a bit more overhead to the JVM memory usage than just adding together the above two values, but experiment with things and find the best values for your application.

Likewise, if the Heroku logs tell you that you've exceed the 512MB cap then you should lower one or both of the above memory values.

To keep memory usage low, I'd suggest not using the TorqueBox session store (since it stores session in-memory) or make heavy use of the Infinispan caching integration, since that is again stored in-memory by default.

Time Outs Waiting For Application to Boot

Heroku gives your application 60 seconds to come up from the time it tells the process to start. That's a main reason for the awkwardly long Procfile entry - we need to deploy and run your application within that 60 seconds. I haven't seen this error at all in my example applications since tuning the Procfile values, but if you do see it hopefully it's only every now and then. Heroku automatically tries to boot your application again and sometimes you just get a dyno that's running slower than others, so it should fix itself.

If you always get this error when booting your application, try to reduce any work being done in Rails initializers or similar things that happen during application boot. We'll be able to take several seconds off the boot time if we create a specialized version of torquebox-server for Heroku, as mentioned above in the slug size section, by eliminating the separate torquebox deploy step.

Heroku Spins Down Idle Dynos

Heroku spins down dynos that are idle, where idle is determined by how long it has been since your application serviced an external request. When it takes that dyno down all your background processing (jobs, services, etc) will go down with it. According to Heroku's Dynos article, you can prevent this idling of dynos by paying for more than one web dyno.

Provide Feedback

Whether you try things out and everything work flawlessly or you have a problem, please let us know! There are some things we can do to improve the experience on Heroku, but we need a rough idea of the demand to help prioritize those requests versus others.

If you're just here because you want a place to host TorqueBox applications but don't want to use Heroku, check out our other Hosting Options on the wiki.

Gemfile

source 'https://rubygems.org'
source 'http://repository-projectodd.forge.cloudbees.com/incremental/torquebox/1295/gem-repo/'

ruby '1.9.3', engine: 'jruby', engine_version: '1.7.1'

gem 'rails', '3.2.9'

gem 'activerecord-jdbcsqlite3-adapter', :group => :development
gem 'activerecord-jdbcpostgresql-adapter', :group => :production

gem 'jruby-openssl'

# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails',   '~> 3.2.3'
  gem 'coffee-rails', '~> 3.2.1'

  # See https://github.com/sstephenson/execjs#readme for more supported runtimes
  gem 'therubyrhino'

  gem 'uglifier', '>= 1.0.3'
end

gem 'jquery-rails'

gem 'torquebox-server', '2.x.incremental.1295'

Procfile

web: jruby -X-C -J-server -J-XX:+UseParallelOldGC -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 bin/torquebox deploy . --env=production && jruby -e "require 'torquebox/server'; TorqueBox::Server.setup_environment; ENV['APPEND_JAVA_OPTS'] = %Q(-XX:MaxPermSize=128m -Xmx256m -Djboss.server.config.dir=#{ENV['PWD']} -Dtorquebox.http.port=#{ENV['PORT']} -Dorg.torquebox.web.force_http_connector_start=true -Dorg.apache.tomcat.util.LOW_MEMORY=true); exec(%Q(#{ENV['JBOSS_HOME']}/bin/standalone.sh -Djruby.home=#{ENV['JRUBY_HOME']} --server-config=standalone.xml -Dorg.torquebox.web.http.maxThreads=15 -b 0.0.0.0))"

standalone.xml

<?xml version='1.0' encoding='UTF-8'?> 
<server xmlns='urn:jboss:domain:1.3'>
    <extensions>
        <extension module='org.jboss.as.clustering.infinispan'/>
        <extension module='org.jboss.as.cmp'/>
        <extension module='org.jboss.as.configadmin'/>
        <extension module='org.jboss.as.connector'/>
        <extension module='org.jboss.as.deployment-scanner'/>
        <extension module='org.jboss.as.ee'/>
        <extension module='org.jboss.as.ejb3'/>
        <extension module='org.jboss.as.jaxr'/>
        <extension module='org.jboss.as.jaxrs'/>
        <extension module='org.jboss.as.jdr'/>
        <extension module='org.jboss.as.jmx'/>
        <extension module='org.jboss.as.jpa'/>
        <extension module='org.jboss.as.jsr77'/>
        <extension module='org.jboss.as.logging'/>
        <extension module='org.jboss.as.mail'/>
        <extension module='org.jboss.as.messaging'/>
        <extension module='org.jboss.as.naming'/>
        <extension module='org.jboss.as.osgi'/>
        <extension module='org.jboss.as.pojo'/>
        <extension module='org.jboss.as.remoting'/>
        <extension module='org.jboss.as.sar'/>
        <extension module='org.jboss.as.security'/>
        <extension module='org.jboss.as.threads'/>
        <extension module='org.jboss.as.transactions'/>
        <extension module='org.jboss.as.web'/>
        <extension module='org.jboss.as.weld'/>
        <extension module='org.torquebox.bootstrap'/>
        <extension module='org.torquebox.core'/>
        <extension module='org.torquebox.cdi'/>
        <extension module='org.torquebox.web'/>
        <extension module='org.torquebox.messaging'/>
        <extension module='org.torquebox.security'/>
        <extension module='org.torquebox.jobs'/>
        <extension module='org.torquebox.services'/>
        <extension module='org.projectodd.polyglot.hasingleton'/>
        <extension module='org.projectodd.polyglot.cache'/>
    </extensions>
    <system-properties>
        <property name='org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR' value='false'/>
        <property name='org.apache.tomcat.util.net.WAIT_FOR_THREAD' value='false'/>
    </system-properties>
    <profile>
        <subsystem xmlns='urn:jboss:domain:logging:1.1'>
            <console-handler name='CONSOLE'>
                <level name='INFO'/>
                <formatter>
                    <pattern-formatter pattern='%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n'/>
                </formatter>
            </console-handler>
            <periodic-rotating-file-handler name='FILE'>
                <formatter>
                    <pattern-formatter pattern='%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n'/>
                </formatter>
                <file relative-to='jboss.server.log.dir' path='server.log'/>
                <suffix value='.yyyy-MM-dd'/>
                <append value='true'/>
            </periodic-rotating-file-handler>
            <logger category='com.arjuna'>
                <level name='WARN'/>
            </logger>
            <logger category='org.apache.tomcat.util.modeler'>
                <level name='WARN'/>
            </logger>
            <logger category='sun.rmi'>
                <level name='WARN'/>
            </logger>
            <logger category='jacorb'>
                <level name='WARN'/>
            </logger>
            <logger category='jacorb.config'>
                <level name='ERROR'/>
            </logger>
            <root-logger>
                <level name='INFO'/>
                <handlers>
                    <handler name='CONSOLE'/>
                    <handler name='FILE'/>
                </handlers>
            </root-logger>
            <logger category='org.jboss.jca.adapters.jdbc.extensions.mysql'>
                <level name='ERROR'/>
            </logger>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:cmp:1.0'/>
        <subsystem xmlns='urn:jboss:domain:configadmin:1.0'/>
        <subsystem xmlns='urn:jboss:domain:datasources:1.1'>
            <datasources>
                <datasource jndi-name='java:jboss/datasources/ExampleDS' pool-name='ExampleDS' enabled='true' use-java-context='true'>
                    <connection-url>
                        jdbc:h2:mem:test;DB_CLOSE_DELAY=-1
                    </connection-url>
                    <driver>
                        h2
                    </driver>
                    <security>
                        <user-name>
                            sa
                        </user-name>
                        <password>
                            sa
                        </password>
                    </security>
                </datasource>
                <drivers>
                    <driver name='h2' module='com.h2database.h2'>
                        <xa-datasource-class>
                            org.h2.jdbcx.JdbcDataSource
                        </xa-datasource-class>
                    </driver>
                </drivers>
            </datasources>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:deployment-scanner:1.1'>
            <deployment-scanner path='deployments' relative-to='jboss.server.base.dir' scan-interval='5000' deployment-timeout='1200'/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:ee:1.1'>
            <spec-descriptor-property-replacement>
                false
            </spec-descriptor-property-replacement>
            <jboss-descriptor-property-replacement>
                true
            </jboss-descriptor-property-replacement>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:ejb3:1.3'>
            <session-bean>
                <stateless>
                    <bean-instance-pool-ref pool-name='slsb-strict-max-pool'/>
                </stateless>
                <stateful default-access-timeout='5000' cache-ref='simple'/>
                <singleton default-access-timeout='5000'/>
            </session-bean>
            <mdb>
                <resource-adapter-ref resource-adapter-name='hornetq-ra'/>
                <bean-instance-pool-ref pool-name='mdb-strict-max-pool'/>
            </mdb>
            <pools>
                <bean-instance-pools>
                    <strict-max-pool name='slsb-strict-max-pool' max-pool-size='20' instance-acquisition-timeout='5' instance-acquisition-timeout-unit='MINUTES'/>
                    <strict-max-pool name='mdb-strict-max-pool' max-pool-size='20' instance-acquisition-timeout='5' instance-acquisition-timeout-unit='MINUTES'/>
                </bean-instance-pools>
            </pools>
            <caches>
                <cache name='simple' aliases='NoPassivationCache'/>
                <cache name='passivating' passivation-store-ref='file' aliases='SimpleStatefulCache'/>
            </caches>
            <passivation-stores>
                <file-passivation-store name='file'/>
            </passivation-stores>
            <async thread-pool-name='default'/>
            <timer-service thread-pool-name='default'>
                <data-store path='timer-service-data' relative-to='jboss.server.data.dir'/>
            </timer-service>
            <thread-pools>
                <thread-pool name='default'>
                    <max-threads count='2'/>
                    <keepalive-time time='100' unit='milliseconds'/>
                </thread-pool>
            </thread-pools>
            <iiop enable-by-default='false' use-qualified-name='false'/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:infinispan:1.3'>
            <cache-container name='hibernate' default-cache='local-query' module='org.jboss.as.jpa.hibernate:4'>
                <local-cache name='entity'>
                    <transaction mode='NON_XA'/>
                    <eviction strategy='LRU' max-entries='10000'/>
                    <expiration max-idle='100000'/>
                </local-cache>
                <local-cache name='local-query'>
                    <transaction mode='NONE'/>
                    <eviction strategy='LRU' max-entries='10000'/>
                    <expiration max-idle='100000'/>
                </local-cache>
                <local-cache name='timestamps'>
                    <transaction mode='NONE'/>
                    <eviction strategy='NONE'/>
                </local-cache>
            </cache-container>
            <cache-container name='polyglot' default-cache='sessions' aliases='torquebox'>
                <local-cache name='sessions' start='EAGER'>
                    <eviction strategy='LRU' max-entries='10000'/>
                    <expiration max-idle='100000'/>
                    <transaction mode='FULL_XA'/>
                </local-cache>
            </cache-container>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:jaxr:1.1'>
            <connection-factory jndi-name='java:jboss/jaxr/ConnectionFactory'/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:jaxrs:1.0'/>
        <subsystem xmlns='urn:jboss:domain:jca:1.1'>
            <archive-validation enabled='true' fail-on-error='true' fail-on-warn='false'/>
            <bean-validation enabled='true'/>
            <default-workmanager>
                <short-running-threads>
                    <core-threads count='1'/>
                    <queue-length count='50'/>
                    <max-threads count='10'/>
                    <keepalive-time time='10' unit='seconds'/>
                </short-running-threads>
                <long-running-threads>
                    <core-threads count='1'/>
                    <queue-length count='50'/>
                    <max-threads count='10'/>
                    <keepalive-time time='10' unit='seconds'/>
                </long-running-threads>
            </default-workmanager>
            <cached-connection-manager/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:jdr:1.0'/>
        <subsystem xmlns='urn:jboss:domain:jmx:1.1'>
            <show-model value='true'/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:jpa:1.0'>
            <jpa default-datasource=''/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:jsr77:1.0'/>
        <subsystem xmlns='urn:jboss:domain:mail:1.0'>
            <mail-session jndi-name='java:jboss/mail/Default'>
                <smtp-server outbound-socket-binding-ref='mail-smtp'/>
            </mail-session>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:messaging:1.2'>
            <hornetq-server>
                <persistence-enabled>
                    true
                </persistence-enabled>
                <journal-file-size>
                    102400
                </journal-file-size>
                <journal-min-files>
                    2
                </journal-min-files>
                <connectors>
                    <in-vm-connector name='in-vm' server-id='0'/>
                </connectors>
                <acceptors>
                    <in-vm-acceptor name='in-vm' server-id='0'/>
                </acceptors>
                <security-settings>
                    <security-setting match='#'>
                        <permission type='send' roles='guest'/>
                        <permission type='consume' roles='guest'/>
                        <permission type='createNonDurableQueue' roles='guest'/>
                        <permission type='deleteNonDurableQueue' roles='guest'/>
                    </security-setting>
                </security-settings>
                <address-settings>
                    <!--default for catch all-->
                    <address-setting match='#'>
                        <dead-letter-address>
                            jms.queue.DLQ
                        </dead-letter-address>
                        <expiry-address>
                            jms.queue.ExpiryQueue
                        </expiry-address>
                        <redelivery-delay>
                            0
                        </redelivery-delay>
                        <max-size-bytes>
                            20971520
                        </max-size-bytes>
                        <address-full-policy>
                            PAGE
                        </address-full-policy>
                        <message-counter-history-day-limit>
                            10
                        </message-counter-history-day-limit>
                    </address-setting>
                </address-settings>
                <jms-connection-factories>
                    <connection-factory name='InVmConnectionFactory'>
                        <connectors>
                            <connector-ref connector-name='in-vm'/>
                        </connectors>
                        <entries>
                            <entry name='java:/ConnectionFactory'/>
                        </entries>
                        <consumer-window-size>
                            1
                        </consumer-window-size>
                    </connection-factory>
                    <connection-factory name='RemoteConnectionFactory'>
                        <connectors>
                            <connector-ref connector-name='in-vm'/>
                        </connectors>
                        <entries>
                            <entry name='java:jboss/exported/jms/RemoteConnectionFactory'/>
                        </entries>
                        <consumer-window-size>
                            1
                        </consumer-window-size>
                    </connection-factory>
                    <pooled-connection-factory name='hornetq-ra'>
                        <transaction mode='xa'/>
                        <connectors>
                            <connector-ref connector-name='in-vm'/>
                        </connectors>
                        <entries>
                            <entry name='java:/JmsXA'/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>
                <jmx-management-enabled>
                    true
                </jmx-management-enabled>
                <security-enabled>
                    false
                </security-enabled>
            </hornetq-server>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:naming:1.2'>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:osgi:1.2' activation='lazy'>
            <properties>
                <!-- Specifies the beginning start level of the framework -->
                <property name='org.osgi.framework.startlevel.beginning'>
                    1
                </property>
            </properties>
            <capabilities>
                <!-- modules registered with the OSGi layer on startup -->
                <capability name='javax.servlet.api:v25'/>
                <capability name='javax.transaction.api'/>
                <!-- bundles started in startlevel 1 -->
                <capability name='org.apache.felix.log' startlevel='1'/>
                <capability name='org.jboss.osgi.logging' startlevel='1'/>
                <capability name='org.apache.felix.configadmin' startlevel='1'/>
                <capability name='org.jboss.as.osgi.configadmin' startlevel='1'/>
            </capabilities>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:pojo:1.0'/>
        <subsystem xmlns='urn:jboss:domain:resource-adapters:1.0'/>
        <subsystem xmlns='urn:jboss:domain:sar:1.0'/>
        <subsystem xmlns='urn:jboss:domain:security:1.2'>
            <security-domains>
                <security-domain name='other' cache-type='default'>
                    <authentication>
                        <login-module code='Remoting' flag='optional'>
                            <module-option name='password-stacking' value='useFirstPass'/>
                        </login-module>
                        <login-module code='RealmDirect' flag='required'>
                            <module-option name='password-stacking' value='useFirstPass'/>
                        </login-module>
                    </authentication>
                </security-domain>
                <security-domain name='jboss-web-policy' cache-type='default'>
                    <authorization>
                        <policy-module code='Delegating' flag='required'/>
                    </authorization>
                </security-domain>
                <security-domain name='jboss-ejb-policy' cache-type='default'>
                    <authorization>
                        <policy-module code='Delegating' flag='required'/>
                    </authorization>
                </security-domain>
            </security-domains>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:threads:1.1'/>
        <subsystem xmlns='urn:jboss:domain:transactions:1.2'>
            <core-environment>
                <process-id>
                    <uuid/>
                </process-id>
            </core-environment>
            <recovery-environment socket-binding='txn-recovery-environment' status-socket-binding='txn-status-manager'/>
            <coordinator-environment default-timeout='300'/>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:web:1.2' default-virtual-server='default-host' native='false'>
            <connector name='http' protocol='HTTP/1.1' scheme='http' socket-binding='http' enabled='false'/>
            <virtual-server name='default-host'>
                <alias name='localhost'/>
                <alias name='example.com'/>
            </virtual-server>
        </subsystem>
        <subsystem xmlns='urn:jboss:domain:weld:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-bootstrap:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-core:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-cdi:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-web:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-messaging:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-security:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-jobs:1.0'/>
        <subsystem xmlns='urn:jboss:domain:torquebox-services:1.0'/>
        <subsystem xmlns='urn:jboss:domain:polyglot-hasingleton:1.0'/>
        <subsystem xmlns='urn:jboss:domain:polyglot-cache:1.0'/>
    </profile>
    <interfaces>
        <interface name='management'>
            <inet-address value='${jboss.bind.address.management:127.0.0.1}'/>
        </interface>
        <interface name='public'>
            <inet-address value='${jboss.bind.address:127.0.0.1}'/>
        </interface>
        <!-- TODO - only show this if the jacorb subsystem is added  -->
        <interface name='unsecure'>
            <!--
              ~  Used for IIOP sockets in the standard configuration.
              ~                  To secure JacORB you need to setup SSL 
              -->
            <inet-address value='${jboss.bind.address.unsecure:127.0.0.1}'/>
        </interface>
    </interfaces>
    <socket-binding-group name='standard-sockets' default-interface='public' port-offset='${jboss.socket.binding.port-offset:0}'>
        <socket-binding name='http' port='${torquebox.http.port:8080}'/>
        <socket-binding name='osgi-http' interface='management' port='8090'/>
        <socket-binding name='remoting' port='4447'/>
        <socket-binding name='txn-recovery-environment' port='4712'/>
        <socket-binding name='txn-status-manager' port='4713'/>
        <outbound-socket-binding name='mail-smtp'>
            <remote-destination host='localhost' port='25'/>
        </outbound-socket-binding>
    </socket-binding-group>
</server>