This note will walk you though setting up and securing a Ubuntu 10.04 LTS then deploying a Rails application with mulit-stage deployment.
TODO:
- Add section for NGINX setup
The following section focuses on setting up and securing the Ubuntu 10.04 LTS server
Set the hostname information for the server.
$ echo "server123" > /etc/hostname
$ echo -e "\n111.222.333.444 server123.domain.com server123\n" >> /etc/hosts
$ hostname -F /etc/hostname
Log in as root and change the password
Add a user for yourself so you can manage the slice without being root. Also, create the deploy user account.
$ adduser username
$ adduser deploy
Add your user to the sudo group so you can use sudo. Older articles might suggest adding to the wheel group but I think that is a redhat thing. If you use visudo to look at the sudoers file you'll see on Ubuntu there already exists a sudo group with the correct settings
$ adduser username sudo
We need to lock down SSH access to the slice since by default root logins are allowed via passwords over port 22. We'll change things so that 1) only non-root accounts can access the server. 2) Access is only allowed using ssh keys and not passwords. 3) ssh is accessible on some high port other than 22
Since we'll be using ssh keys to access the server you'll need to get a copy of your key onto the server. This requires that you have a key pair already created on your local machine and that you can still authenticate to the server using your username/password (you haven't yet changed the sshd_config)
Send the public key from your local machine to your user account on the server.
$ scp keyfile.pub username@111.222.333.444:/home/username
Setup your user's ssh directory on the server and add the authorized key file. To create the ssh directory you can either have the slice make it for you by connecting to an ssh server or you can create the folder and permissions manually.
Connect to an ssh server to have the slice create your user's ssh folder and permissions.
$ ssh git@github.com
Or create the ssh folder and permissions manually
$ mkdir ~/.ssh
$ chown -R username:username ~/.ssh
$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/authorized_keys
Move the key you uploaded (via scp) to be the authorized keys file
$ mv ~/keyfile.pub ~/.ssh/authorized_keys
You can now make a terminal connection to from your local computer to the deployment server. Later we'll see how to enable Capistrano to do the same.
Since we're going to change a system configuration file I like to first make an archived copy of the original (called master) and recreate the file with only the active lines.
Make an archived copy of sshd_config
$ mv /etc/ssh/sshd_config /etc/ssh/sshd_config.master
Now recreate the file from the master with only the active lines (remove commented out lines and any newlines)
$ egrep -v "^#|^$" sshd_config.master > sshd_config
Edit the file sshd_config and add the following lines:
Port 3000 # or any port you like
PermitRootLogin no
PasswordAuthentication no
UseDNS no
Next we'll restart the ssh service but before doing that, make sure you leave an open ssh connection as root active so you can fix any problems with the ssh config if needed.
In your open ssh session as the root user, restart the ssh service
$ /etc/init.d/ssh reload
Now test that you can ssh into the slice using your user account with key in a new terminal window. DO NOT close your root ssh session window.
$ ssh -p 3000 username@111.222.333.444
If that was successful using your key (not your password) then you've configured ssh correctly and can now log out.
Tip You might also want to add your key to the root user's authorized_keys file so you can directly log in as root. Although now that you're user account is a member of the sudo group you can always sudo su - to become the root user.
View current rules
$ /sbin/iptables -L -n
Backup rules
$ /sbin/iptables-save > /etc/iptables.save
Create rules by starting with slicehost rules. Modify as needed (ex: the ssh port)
Apply those rules by saving to a file on the slice and using the restore tool
$ /sbin/iptables-restore < iptables.txt
Verify the rules now look correct
$ /sbin/iptables -L -n
And if so back them up again to the iptables.save file
$ /sbin/iptables-save > /etc/iptables.save
If you ever need to flush the rules here's how:
$ /sbin/iptables -F
Now set the rules to apply when the machine is rebooted by adding a script to /etc/network/if-pre-up.d/
$ vi /etc/network/if-pre-up.d/iptables
Paste the following into the iptables script
#!/bin/sh
sudo sh -c '/sbin/iptables-restore < /etc/iptables.save'
And make the script executable
$ /bin/chmod +x /etc/network/if-pre-up.d/iptables
This will run the iptables script every time the network interface comes up.
Below are two different ways to set the timezone for the deployment server.
$ sudo dpkg-reconfigure tzdata
$ cd /etc; mv localtime localtime.orig; ln /usr/share/zoneinfo/EST localtime
First update the package list then do a safe-upgrade
$ sudo aptitude update
$ sudo aptitude safe-upgrade
Here is a list of all the packages we'll install to host rails applications using apache and mysql. Note that the apache2-prefork-dev package includes a lot of dependancies that are needed
$ sudo aptitude install apache2 apache2-prefork-dev autoconf bison build-essential \
clang curl dnsutils git-core imagemagick libc6-dev libcurl4-openssl-dev libffi-dev
libmagickwand-dev libmysqlclient16 libmysqlclient16-dev libreadline6-dev libsqlite3-0
libsqlite3-dev libssl-dev libxml2 libxml2-dev libxslt-dev libxslt1.1 libxslt1-dev
libyaml-dev mysql-client mysql-server openssl sqlite3 wget zlib1g zlib1g-dev
See the Gist Ubuntu Packages for details on the packages to be installed. Note that RailsReady will not install all the packages you need for a complete system.git
To secure MySQL, create a password for the mysql root user (if you haven't already done so during install) and limit connections to localhost only.
The simplest way to change the mysql root password is to reconfigure the package. First verify the version of mysql and then run dpkg-reconfigure. This will prompt for the new password.
$ aptitude search mysql-server
$ dpkg-reconfigure mysql-server-5.1
Alternatively you can change the mysql users root password by stopping mysql and restarting it with the --skip-grant-tables and --skip-netoworking. These options will allow any user to log into mysql root without a password and disable networking (so no one can remotely connect to the mysql server during this time).
$ stop mysql
$ mysqld_safe --skip-grant-tables --skip-networking &
Now start the mysql client and change the password
$ mysql
mysql > UPDATE mysql.user SET Password=PASSWORD('') WHERE User='root';
mysql > FLUSH PRIVILEGES;
mysql > exit
Shutdown the mysql server again. This time you have to use pkill. Then start mysql normally again.
$ pkill mysql*
$ start mysql
To configure mysql so it will only accept connections through the localhost interface you need to set the bind address in /etc/mysql/my.cnf. Fortunately this is already set in 5.1 on Ubuntu 10.04.
You can use the following to manage the mysql process
$ sudo status mysql
$ stop mysql
$ start mysql
$ restart mysql
You can also use the classical way
$ sudo /etc/init.d/mysql stop
$ sudo /etc/init.d/mysql start
A lot of configuration files reference /tmp/mysql.sock so you can create a symlink for /var/run/mysqld/mysqld.sock to make that easier or just specify the Ubuntu location.
The following section focuses on server configuration needed to run Rails and multi-stage deployment of a Rails application.
RailsReady is a great project that will be used to compile ruby, install packages needed, install imagemagick, update system gems and instal bundler, passenger and rails
While there is support for installing RVM on staging and production servers (RailsReady) it makes things more complex because you have to setup Passenger to use the correct gemset, configure Capistrano to use the correct gemset, and configure the deploy user account to use the correct gemset (this last one I could not figure out).
The two main reasons for using RVM in any environment are to have multiple rubies and to have a unique gemset for each application. Usually there isn't a need for multiple rubies on a single server. If that is a need you have then you'll want to look at configuring Passenger and Capistrano to support RVM in your deployment.
Bundler already provides the second reason for using RVM - gemsets. Bundler will install the gems for your application into the shared/ directory of your app.
So as a general rule, use RVM in development and avoid it in staging/production.
If you have already installed RVM on your server and would like to remove it so you can use a standard ruby install do the following as root:
-
Implode RVM
$ rvm implode
-
Remove wrappers in
/usr/local/bin. Change into the directory and view the contents of each ruby related file (erb, gem, irb, rake, rdoc, ri, ruby, testrb). If it mentions rvm then its a wrapper and you'll need to remove it. -
Remove
/etc/profile.d/rvm
- Optional: Now run RailsReady
- Optional: If you installed Passenger using RVM you'll need to reinstall and reset the values in the apache config files (
/etc/apache2/mods-available)
The rest of these instructions will use RailsReady to compile Ruby.
To run RailsReady follow the README instructions from the project's site
$ wget --no-check-certificate https://github.com/joshfng/railsready/raw/master/railsready.sh && bash railsready.sh
RailsReady will update the system rubygem package but here are instructions for reference
Since you're not using RVM you will need to use sudo when installing gems.
Update rubygems to the latest version. Be sure to run as root
$ sudo gem update --system
Install some standard gems (bundler and passenger). I don't include rake anymore because I have enough apps the use rake 0.8.7. So I want Bundler to install the specific rake version per application.
$ sudo gem install bundler passenger --no-ri --no-rdoc
Run the necessary installer (nginx or apache)
$ sudo passenger-install-apache2-module
The passenger install returns instructions with code on how to configure Apache. It's not the recommended way for Ubuntu. The proper way to configure Apache2 on Ubuntu is to put the code into module files in /etc/apache2/mods-available.
Put the code from the Passenger installer into the files passenger.load and passenger.conf. You need to be root for this, sudo doesn't work.
$ sudo su -
$ cd /etc/apache2/mods-available
$ echo "LoadModule [...]" > /etc/apache2/mods-available/passenger.load
$ echo -e "PassengerRoot [...]\nPassengerRuby [...]" > /etc/apache2/mods-available/passenger.conf
Next enable the passenger module, as well as a few others we'll need in the near future (rewrite and auth_digest) and restart apache
$ a2enmod passenger
$ a2enmod rewrite
$ a2enmod auth_digest
$ /etc/init.d/apache2 restart
Possible Error: If you get the following error - "virtual memory exhausted: Cannot allocate memory". It is because Passenger ran out of memory during install. To fix stop any processes that are taking up memory (like apache, mysql) until you're done installing Passenger
Create the deploy mysql user account and allow deploy to connect to mysql server from localhost.
$ mysql -u root -p
mysql> use mysql;
mysql> INSERT INTO user (Host,User,Password) VALUES('%','username',PASSWORD('passwd'));
mysql> flush privileges;
mysql> grant usage on *.* to deploy@localhost identified by 'passwd';flush privileges;
Create the database and give access to the deploy mysql user
mysql> create database app_staging;
mysql> grant all privileges on app_staging.* to deploy@localhost;flush privileges;
mysql> exit
Keep in mind that since can only connect to the mysql server via local host you'll have to use an SSH tunnel if you want a client to connect to it (ex: Navicat). Here is a mysql command cheat sheet
Setup a directory for your app
$ sudo mkdir -p /var/www/sites/example.com
$ sudo chown deploy.deploy /var/www/sites/example.com
Create a virtual host by copying and paste the following into /etc/apache2/sites-available/example.com
<VirtualHost *:80>
ServerName example.com
DocumentRoot "/var/www/sites/example.com/current/public"
ErrorLog "/var/log/apache2/example.com-error_log"
CustomLog "/var/log/apache2/example.com-access_log" common
# Tells passenger to run in "staging" mode
RailsEnv staging
<Directory "/var/www/sites/example.com/current/public">
Options All
AllowOverride All
Order allow,deny
Allow from all
</Directory>
RewriteEngine On
# Remove the www
RewriteCond %{HTTP_HOST} ^www.example.com$ [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
# Enable digest authentication to protect site or add let the application
# handle it by adding code to an environments/[env].rb file.
# <Location />
# AuthName 'Private'
#
# AuthType Digest
# AuthDigestProvider file
# AuthDigestDomain /
# AuthUserFile /var/www/sites/example.com/shared/config/.htdigest
#
# Require valid-user
# </Location>
</VirtualHost>
Now enable the vhost. You have to become root and change into the sites-available directory
$ sudo su -
$ cd /etc/apache2/sites-available
$ a2ensite example.com
$ /etc/init.d/apache2 reload
TODO: Add nginx setup info
Setup the shared resources for deployments
On the server switch to the deploy user and create the shared directory for Capistrano
$ su - deploy
$ mkdir -p /var/www/sites/example.com/shared/config
Create the database config file for your app. Copy database.yml.example from source to shared/config or copy the following into /var/www/sites/example.com/shared/config/database.yml
staging:
adapter: mysql2
encoding: utf8
reconnect: false
database: appname_staging
pool: 5
username: deploy
password:
socket: /var/run/mysqld/mysqld.sock
If you enabled digest authentication in the vhost you'll need to create the password file in the shared/config directory for Capistrano. Create the htdigest password file and set the password. Note that the realm ("Private") needs to be the same value as AuthName in the vhost
$ htdigest -c /var/www/sites/example.com/shared/config/.htdigest 'Private' 'admin'
If you've decided to allow your application to handle the basic auth see how to do that in the Capistrano Multistage section below.
To use multiple environments with Capistrano you can use the capistrano-ext gem. Make the following configurations to the project on your local machine to create a staging environment
Add the capistrano gems to your gem file
gem 'capistrano'
gem 'capistrano-ext'
Now capify the project and customize your deploy.rb script.
$ capify
See the Gist Example Capistrano configuration for multi stage deployment for an example configuration. Read the comments for details about each setting.
Since we're using Capistrano's multistage extension, add a deployment directory and file for each environment to the repo.
$ mkdir config/deploy
$ touch config/deploy/staging.rb
$ touch config/deploy/production.rb
Add any environment customizations you need to those files. For example, to change where the app deploys to in stage, add the following to config/deploy/staging.rb
set :rails_env, "staging" # sets the env to use when running migrations
set :deploy_to, "/var/www/sites/stage.#{application}"
Although not required by Capistrano multistage you'll want to create an environment file for staging if you haven't already. Create the environments/staging.rb. Duplicate production.rb and customize from there as needed.
To have your application handle basic auth for apache add the following code to one of you environments/[env].rb files. For example, to protect the staging deployment with basic auth add the following to environments/staging.rb and change the username and password to what ever you want.
# Enable basic auth
config.middleware.insert_after(::Rack::Lock, "::Rack::Auth::Basic", "Staging") do |u, p|
[u, p] == ['username', 'password']
end
DNS Make sure your server is publicly accessible using the name in your config. If it is not yet, then make an entry in your /etc/hosts file.
Capistrano makes two SSH connections when deploying. The first is the connection from your local machine to the deploy server. The second is from the deploy server to GitHub to clone the code. There are two types of configurations that will handle this. The first is to use :forward_agent in the :ssh_options of the recipe. The second is to use the deploy key feature of GitHub.
Forwarding your SSH agent enables Capistrano to use your SSH connection from localhost to the server to connect to GitHub. It is the preferred setup because it is easy to setup, doesn't require management of several keys on the server and is arguably more secure. It also works great for your own private projects and scales well when using a team of other developers.
In your deployment recipe you need to specify the SSH connection details and that you want to forward your agent using the :ssh_options. Notice that you're specifying a key in the options. If you're developing with a team you should create a generic key which allows multiple deployers by just having access to the key.
Each person that wants to deploy the application will need to add the public key of the specified deploykey to their GitHub profile.
set :ssh_options, {
:keys => [File.join(ENV["HOME"], ".ssh", "deploykey")],
:forward_agent => true,
:port => 3000
}
You can confirm that the app user on the server is able to connect to the GitHub repo by running the following command from your local computer. If it returns the git ref then you're good to go.
$ cap staging invoke COMMAND='git ls-remote git@github.com:account/repo.git master'
If that doesn't work check to make sure the deploy key is in your local computer's ssh-agent and add it if it is not.
$ ssh-add -l
$ ssh-add ~/.ssh/deploykey
There is also a case where I had to have github added as a known host for the deploy user on the server. The fix was to simply log into the server as the deploy user and run ssh git@github.com which prompts you to add to the known hosts.
If you prefer to keep your connection details out of code you can make an entry for the server in your ~/.ssh/config file that specifies the key and port. I recommend leaving the forward agent option in the recipe so it is easier to identify how the connection is being made.
Using the deploy key feature allows the user account on the server to have its own key to access the GitHub repository. It requires that you have one key per user. That means you have to have a different user account for each application on the server or are only running one application per server. If you have for example a deploy user and you add more than one app for that deploy user, then using a deploy key feature won't work because there is no way to tell Capistrano which key to use to connect to GitHub. Using ssh-agent and making it persist with keychain might make this work but really that is too much effort for no reason.
To use this feature first create a key for the app user on the server and add that key as a "Deploy key" on the GitHub repo.
$ sudo su - deploy
$ mkdir ~/.ssh/deploy-keys
$ ssh-keygen -t rsa -C "hostname:appname"
You should be able to test your connection to GitHub by running the following command as the app user on the server
$ sudo su - deploy
$ ssh git@github.com
In your deploy recipe disable the option to forward you agent (you can keep the rest of the connection options if you want) and add the directive to prompt you for the passphrase.
default_run_options[:pty] = true
set :ssh_options, {
:keys => [File.join(ENV["HOME"], ".ssh", "decwise", "deploy")],
# :forward_agent => true,
:port => 35275
}
With this in place the app user on the server can now connect to the GitHub repo and pull down the code to deploy.
Before you deploy the first time, run through the following to make sure Capistrano is ready to go. Capistrano multistage allows you to specify the environment you want to deploy to. Since "staging" is configured to be the default stage in the deploy.rb script we could skip specifying the environment and it would still follow the staging.rb configurations.
$ cap staging invoke COMMAND=date # verify capistrano can connect to the server and run commands
$ cap staging deploy:setup # writes all the main files and dir to the server
$ cap staging deploy:check # verifies permissions and dependancies
$ cap staging deploy:update # pushes the code to the server and creates symlinks
(The following section is adapted from the Capistrano Wiki)
Once that is successful, log into the server as the deploy user to load up the schema. You'll need to specify the staging env because we told Passenger to run the app in staging mode in the vhost. Also note that the rake command will DELETE ALL DATA in the database. So only do this on new databases.
$ cd [deploy_to]/current
$ bundle exec rake RAILS_ENV=staging db:schema:load
Note that use of bundle exec is required if your system gems don't match your projects gems, eg rake 0.9.2 and 0.8.7 are installed (Gem Versioning and Bundler; Doing It Right)
If that worked then we now know that the staging database configuration is correct. If it fails, it is usually because your database configuration is incorrect (e.g., wrong database name, wrong user name, wrong password, or similar).
Once that’s done, we can test to see that our application will actually load by starting up a console instance. You'll need to specify the staging environment for the console to start up. Also, be sure to switch to the correct gemset if that applies
$ bundle show rails #=> should show the gem under the shared directory of your app
$ rails c staging
If all goes well, it will start up and present you with a prompt. Mimic an HTTP request now by using the “app” helper in the console. For now, just grab the root URL and see what happens:
>> app.get("/")
That will return the HTTP status code. If it returns 200 or 302 (or any other 2xx or 3xx code, or even 4xx if you haven’t configured the ”/” url for your application), you’re probably set. If it returns something in the 500’s, you’ll want to check your log/production.log and see why it blew up.
Next, let’s check that your web server is configured right to serve up the static assets. Choose one of your javascript or stylesheet files (we’ll go with prototype.js, since it’s pretty standard in most Rails applications, but you can choose whichever you want.) Open up a web browser and browse and visit
http://example.com/javascripts/prototype.js
If you see the contents of prototype.js in your browser, then you should be set. If not, you’ll need to dig around in your web server’s configuration, restarting your web server after each change, until it does.
Lastly, let’s try firing up the application layer and seeing if the dynamic content is being served correctly. From your local machine again, try this task:
$ cap staging deploy:start
If you get an error, look close at the output and see if you can figure out what went wrong.
Once deploy:start finishes successfully, we reach the moment of truth. By this point, if everything is configured correctly, you should be able to pull up your application in your browser. Let’s try it now and see what happens!
If it failed then you'll have to dig around to find out why. If it worked then proceed to the next couple of tests.
Continue testing by making sure Capistrano can successfully restart our application layer:
$ cap staging deploy:restart
Run that, and then go view your application in a browser again. If it worked, you should still be able to see your application. If it didn’t, then you’ll need to troubleshoot.
For the last test, let’s run a full deploy. By this point, we’ve tested our configuration, and the various parts of a deploy: pushing the code out, restarting the application layer, and so forth. So theoretically, nothing should go wrong! Here we go:
$ cap staging deploy
It should work! If it doesn’t, hunt down the errors and fix them, running “cap deploy” until it finally works. Once it does, subsequent deploys should "just work" as well.
I've based this setup process off of the following resources:
- Hivelogic Guide
- Slicehost Lucid Setup Part 1
- Slicehost Lucid Setup Part 2
- SSH Keys
- IPtables Guide
- MySQL Password Reset
- Capistrano Starter
- Capistrano Mulitstage
- Gem Versioning and Bundler; Doing It Right
- RailsReady
- Is A System Wide Install Of RVM A Bad Idea
- How To Remove RVM
- Ubuntu Packages
- Example Capistrano configuration for multi stage deployment
- Linux Hostname Configuration
- Pantz.org MySQL Commands