Why autostarting default config packages are a bad thing

autostart.md

For a moment, I'm going to throw away my automation and configuration management hat. I'll let you know when I put it back on. Also, let's ignore that we're talking about Riak specifically for a moment. Also also, let's ignore any (for a brief moment) the proper role of a package manager.

What's your target market?

If you're writing server software, you have two target markets. The system administrator/operations team and the developer.

Why do you want autostart?

The main reason you want autostart is to get people up and running quickly. To do this, you need to ship safe and sane defaults. This means something like a default configuration that listens only on localhost.

This is a great goal. People can easily install a package and BOOM, they can start using it. No additional thought required.

However, let's look at what you've just done. You've started a program running in the background (or worse - foreground) that, in the case of Riak, is now writing persistant data to the file system. You've just made some pretty arrogant assumptions about not only whether or not the user actually wants it to run full time but also about where that data should go.

So maybe you think, I'll add a post-install dialog for the user so they can tell me where they want the data. If autostarting hadn't hinted that the market you're catering to is the developer, requiring human interaction pretty much cemented it.

But wait! Debian package files support preseeding. This works for system admins too! Except for those who happen to not be using a debian based distro. So now you've narrowed your target market to developers on debian-based systems.

Doing yourself and your users a disservice

We're going to move firmly back into Riak territory now. An autostart configuration with localhost only listening does Riak a disservice. The ONLY valid use case for running in localhost-only mode is local testing/development. Even then it's probably not a valid use case. How many times has the Riak list seen people "benchmarking" Riak with a single node only to be told "You really should be using more than one node for this"?

The fact is that Riak is not a localhost-only single node system. It's a complex system (not in the difficulty sense). Designing a Riak cluster SHOULD require thought. Which backend do I want to use? How many nodes to I need to start? What should my ring creation size be? That last one in particular is a big one.

The reality is you can't reasonably take a single node Riak install that has had no additional configuration done and add it to a production cluster as is. You HAVE to make changes.

Mind you, Riak is wonderfully friendly to operations staff but by making your official packages autostart with what is essentially a developer config makes it LESS friendly to operations folks.

Adding automation into the mix

A bit of a side note. At my last company we were testing using Datastax Enterprise. The official packages behaved in exactly the way you're proposing now.

It was one of the single biggest pain points for me. When I went to automate the entire thing with Chef, I ended up having to jump through hoops cleaning up after the default behavior of the package that it would have been LESS work just having Chef install from tarball and setting up all the additional environment variables and symlinks!

I've attached two files that made up our DSE Analytics node install. Note the "cleanup-default-install". This was neccessary because when the default install started up, configuration data was actually written not only to a Cassandra keyspace itself but to the filesystem. Essentially to add the node to a cluster, I had to blow away the data dir AND clean up the local cached settings stuff. Note those recipes don't even deal with having to rebalance the ring. Let's not even get into the clusterfuck that we ran into because Datastax treated /tmp as persistent storage (which ubuntu happily blows away by default at each restart)

Mind you, Basho is not Datastax.

Basho should be encouraging customers down the proper path - that of automating node installation. Autostarting and shipping developer-only defaults does not encourage that. I would hate to manage a Riak cluster without automation tools. Technically I hate to manage ANYTHING without automation. You guys already point your customers in the right direction with being so operationally friendly. You HAVE a working developer-friendly setup in the 3-node quick start. Shit, you guys have even made EVERY erlang packagers life easy with Rebar.

Proper role of the package manager

Debian packages should NOT be looked to as best practices for how to install your software. Pre and Post steps in RPM packages should have never been invented.

IMHO (a very strong O), the role of the package manager should be to lay bits on the disk. Nothing more. I would even argue that creating users is not even the role of the package manager. Let's not even get into creating the user as a system user or not (useradd -r) which some packages do and some don't. I might not want /etc/skel copied over for that user.

Package manager's opinionated workflow are great for desktop systems. Debian packages are fairly decent at ensuring they never trample all over user changes. However servers are not desktops. The configuration is (or at least should be) a know state before the software is even installed. Default configuration files are useless for pretty much everyone.

Lay the bits on disk, ship well documented example configs and support storing persistent data in customizable locations. If you really want to make it easier for developers and the riak-curious to get started, ship a shell script that enables "developer-mode" - copy localhost-only configs over to real configs and start the service for the user (just don't add it to the system startup scripts!).

As someone who's going to be standing up a large multi-datacenter Riak cluster that will be automated with Chef in the near future, please don't make my job any harder ;)

default.rb

include_recipe "xfs"
include_recipe "apt"
include_recipe "jdk::sun"

package "mdadm"
# Clean up ubuntu default mount on Natty AMI
if node[:platform] == "ubuntu" && node[:lsb][:release] == "11.04"
  mount "/mnt" do
    device "/dev/xvdb"
    action [:umount, :disable]
  end

  mdadm "/dev/md0" do
    devices [ "/dev/xvdb", "/dev/xvdc" ]
    level 0
    action [:create, :assemble]
  end

  execute "build cassandra filesystem" do
    command "mkfs.xfs /dev/md0 -L data"
    not_if { File.exists?("/mnt/va/data/cassandra") }
    not_if "mount  | grep '/mnt'"
  end

  mount "/mnt" do
    device "/dev/md0"
    action [:mount, :enable]
  end
end

user "cassandra" do
  system true
  action [:create, :modify, :manage]
  home "/var/lib/cassandra"
  shell "/bin/bash"
  supports :manage_home => true
end

directory "/mnt/va/"
directory "/mnt/va/data"

directory "/mnt/va/data/cassandra" do
  owner "cassandra"
  group "cassandra"
  mode "0750"
end

directory "/var/log/cassandra" do
  owner "cassandra"
  group "cassandra"
  mode "0755"
end

link "/var/lib/cassandra/data" do
  to "/mnt/va/data/cassandra"
end

apt_repository "datastax" do
  uri "http://debian.datastax.com/enterprise"
  components ["stable", "main"]
  key "http://debian.datastax.com/debian/repo_key"
  action [:add]
end

execute "apt-get update"

bash "cleanup_default_install" do
  user "root"
  cwd "/var/lib/cassandra/data"
  code <<-EOH
  rm -rf /var/lib/cassandra/data/system/*
  EOH
  action :nothing
  not_if "test -f /var/lib/cassandra/.va"
end

service "dse" do
  action :nothing
  notifies :run, resources(:bash => "cleanup_default_install"), :immediately
end

package "dse-full" do
  action [:install]
  version "#{node[:dse][:cassandra_version]}"
  notifies :stop, resources(:service => "dse"), :immediately
end

file "/var/lib/cassandra/.va" do
  action :create_if_missing
end

if node.run_list.roles.include?("dse_hadoop_node")
  include_recipe "dse::hadoop"
end

if node.run_list.roles.include?("dse_cassandra_node")
  include_recipe "dse::cassandra"
end

hadoop.rb

okens = search(:cassandra, "id:tokens").first

cluster_name = node[:dse][:cluster_name]
node_token = tokens['nodes'][node.name] || ""
endpoint_snitch = node[:dse][:endpoint_snitch]

template "/etc/dse/cassandra/cassandra.yaml" do
  mode "0644"
  owner "root"
  group "root"
  action :create
  source "cassandra.yaml.erb"
  variables({:endpoint_snitch => endpoint_snitch, :cluster_name => cluster_name, :node_token => node_token, :seed => node.dse.seed})
end

runit_service "cassandra-hadoop" do
  action [ :enable, :start ]
  subscribes :restart, "template[/etc/dse/cassandra/cassandra.yaml]"
end

As a sysadmin I prefer packages to not autostart things but to set themselves up so that fully enabling them is easy once they've been configured. Ideally it should take a single standard-for-the-system command to enable all of the daemon starting (update-rc.d on Debian, chkconfig or the like on CentOS, etc), and this enabling should definitely include starting on reboot.

@jordansissel I watched your FPM talk, and have looked at it heavily for inspiration. It was long past due and a great gift to the community btw! We have some internal workings of something similar that is more heavily based around erlang and rebar so we can sanely package multiple erlang apps in a similar fashion.

@tels7ar packaging is hard. That's it.

No seriously we have several different users of riak. We have paying customers who mostly go the route of using our Chef cookbook or our profserv people to do the install. Understandably, it is a big priority to get things 100% good there, and how we figured out that Something Changed (tm) in our packages this go around. We also have OSS users of riak who use it production, some use Chef, some take our packages and know exactly what to do with them, and some even take our source and build their own packages (the NetBSD community did this). Lastly we have people who download the OSS version to just try it out. They will most likely put it on an EC2 free micro instance that's 32bit ubuntu. Getting things working perfect all these places is hard, possibly impossible, but it still needs to be done. I think the latter group wants it to autostart and just work, but they are probably the only group who wants that. So understandably when there are clashes in wants and needs, we will lean towards what we think is sane and what will make sense for the majority of our users. Auto-start on install is not wanted by 98% of people, and it doesn't make sense for riak, so we won't do that. Creating rc.d startup scripts though is something I think many people DO want, but we need to put a stop gap in there to make sure it has been configured first. We have ops / profserv people and they are all involved, but I asked on twitter mostly to get a view from outside our company. IMHO that is a REALLY good thing to do if you have to make dogmatic decisions like what a package is going to do when installed and configured. I'm certainly glad I did and got the feedback that I did. So I appreciate your comments.

@lusis We did disable autostart in DSE 1.0.1 release. I'm not aware of anything using /tmp in DSE.