Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Why autostarting default config packages are a bad thing

For a moment, I'm going to throw away my automation and configuration management hat. I'll let you know when I put it back on. Also, let's ignore that we're talking about Riak specifically for a moment. Also also, let's ignore any (for a brief moment) the proper role of a package manager.

What's your target market?

If you're writing server software, you have two target markets. The system administrator/operations team and the developer.

Why do you want autostart?

The main reason you want autostart is to get people up and running quickly. To do this, you need to ship safe and sane defaults. This means something like a default configuration that listens only on localhost.

This is a great goal. People can easily install a package and BOOM, they can start using it. No additional thought required.

However, let's look at what you've just done. You've started a program running in the background (or worse - foreground) that, in the case of Riak, is now writing persistant data to the file system. You've just made some pretty arrogant assumptions about not only whether or not the user actually wants it to run full time but also about where that data should go.

So maybe you think, I'll add a post-install dialog for the user so they can tell me where they want the data. If autostarting hadn't hinted that the market you're catering to is the developer, requiring human interaction pretty much cemented it.

But wait! Debian package files support preseeding. This works for system admins too! Except for those who happen to not be using a debian based distro. So now you've narrowed your target market to developers on debian-based systems.

Doing yourself and your users a disservice

We're going to move firmly back into Riak territory now. An autostart configuration with localhost only listening does Riak a disservice. The ONLY valid use case for running in localhost-only mode is local testing/development. Even then it's probably not a valid use case. How many times has the Riak list seen people "benchmarking" Riak with a single node only to be told "You really should be using more than one node for this"?

The fact is that Riak is not a localhost-only single node system. It's a complex system (not in the difficulty sense). Designing a Riak cluster SHOULD require thought. Which backend do I want to use? How many nodes to I need to start? What should my ring creation size be? That last one in particular is a big one.

The reality is you can't reasonably take a single node Riak install that has had no additional configuration done and add it to a production cluster as is. You HAVE to make changes.

Mind you, Riak is wonderfully friendly to operations staff but by making your official packages autostart with what is essentially a developer config makes it LESS friendly to operations folks.

Adding automation into the mix

A bit of a side note. At my last company we were testing using Datastax Enterprise. The official packages behaved in exactly the way you're proposing now.

It was one of the single biggest pain points for me. When I went to automate the entire thing with Chef, I ended up having to jump through hoops cleaning up after the default behavior of the package that it would have been LESS work just having Chef install from tarball and setting up all the additional environment variables and symlinks!

I've attached two files that made up our DSE Analytics node install. Note the "cleanup-default-install". This was neccessary because when the default install started up, configuration data was actually written not only to a Cassandra keyspace itself but to the filesystem. Essentially to add the node to a cluster, I had to blow away the data dir AND clean up the local cached settings stuff. Note those recipes don't even deal with having to rebalance the ring. Let's not even get into the clusterfuck that we ran into because Datastax treated /tmp as persistent storage (which ubuntu happily blows away by default at each restart)

Mind you, Basho is not Datastax.

Basho should be encouraging customers down the proper path - that of automating node installation. Autostarting and shipping developer-only defaults does not encourage that. I would hate to manage a Riak cluster without automation tools. Technically I hate to manage ANYTHING without automation. You guys already point your customers in the right direction with being so operationally friendly. You HAVE a working developer-friendly setup in the 3-node quick start. Shit, you guys have even made EVERY erlang packagers life easy with Rebar.

Proper role of the package manager

Debian packages should NOT be looked to as best practices for how to install your software. Pre and Post steps in RPM packages should have never been invented.

IMHO (a very strong O), the role of the package manager should be to lay bits on the disk. Nothing more. I would even argue that creating users is not even the role of the package manager. Let's not even get into creating the user as a system user or not (useradd -r) which some packages do and some don't. I might not want /etc/skel copied over for that user.

Package manager's opinionated workflow are great for desktop systems. Debian packages are fairly decent at ensuring they never trample all over user changes. However servers are not desktops. The configuration is (or at least should be) a know state before the software is even installed. Default configuration files are useless for pretty much everyone.

Lay the bits on disk, ship well documented example configs and support storing persistent data in customizable locations. If you really want to make it easier for developers and the riak-curious to get started, ship a shell script that enables "developer-mode" - copy localhost-only configs over to real configs and start the service for the user (just don't add it to the system startup scripts!).

As someone who's going to be standing up a large multi-datacenter Riak cluster that will be automated with Chef in the near future, please don't make my job any harder ;)

include_recipe "xfs"
include_recipe "apt"
include_recipe "jdk::sun"
package "mdadm"
# Clean up ubuntu default mount on Natty AMI
if node[:platform] == "ubuntu" && node[:lsb][:release] == "11.04"
mount "/mnt" do
device "/dev/xvdb"
action [:umount, :disable]
mdadm "/dev/md0" do
devices [ "/dev/xvdb", "/dev/xvdc" ]
level 0
action [:create, :assemble]
execute "build cassandra filesystem" do
command "mkfs.xfs /dev/md0 -L data"
not_if { File.exists?("/mnt/va/data/cassandra") }
not_if "mount | grep '/mnt'"
mount "/mnt" do
device "/dev/md0"
action [:mount, :enable]
user "cassandra" do
system true
action [:create, :modify, :manage]
home "/var/lib/cassandra"
shell "/bin/bash"
supports :manage_home => true
directory "/mnt/va/"
directory "/mnt/va/data"
directory "/mnt/va/data/cassandra" do
owner "cassandra"
group "cassandra"
mode "0750"
directory "/var/log/cassandra" do
owner "cassandra"
group "cassandra"
mode "0755"
link "/var/lib/cassandra/data" do
to "/mnt/va/data/cassandra"
apt_repository "datastax" do
uri ""
components ["stable", "main"]
key ""
action [:add]
execute "apt-get update"
bash "cleanup_default_install" do
user "root"
cwd "/var/lib/cassandra/data"
code <<-EOH
rm -rf /var/lib/cassandra/data/system/*
action :nothing
not_if "test -f /var/lib/cassandra/.va"
service "dse" do
action :nothing
notifies :run, resources(:bash => "cleanup_default_install"), :immediately
package "dse-full" do
action [:install]
version "#{node[:dse][:cassandra_version]}"
notifies :stop, resources(:service => "dse"), :immediately
file "/var/lib/cassandra/.va" do
action :create_if_missing
if node.run_list.roles.include?("dse_hadoop_node")
include_recipe "dse::hadoop"
if node.run_list.roles.include?("dse_cassandra_node")
include_recipe "dse::cassandra"
okens = search(:cassandra, "id:tokens").first
cluster_name = node[:dse][:cluster_name]
node_token = tokens['nodes'][] || ""
endpoint_snitch = node[:dse][:endpoint_snitch]
template "/etc/dse/cassandra/cassandra.yaml" do
mode "0644"
owner "root"
group "root"
action :create
source "cassandra.yaml.erb"
variables({:endpoint_snitch => endpoint_snitch, :cluster_name => cluster_name, :node_token => node_token, :seed => node.dse.seed})
runit_service "cassandra-hadoop" do
action [ :enable, :start ]
subscribes :restart, "template[/etc/dse/cassandra/cassandra.yaml]"
Copy link

tels7ar commented Feb 2, 2012

Jared I think your proposed solution is just fine. Set up the framework in a way that allows people or automation to easily enable autostart if they chose. Set all the defaults to no autostart.

Those of us who deal with packaging all the time are a prickly bunch because software packaging sucks SO HARD. There are two factors here:

  1. the open ended nature of packaging tools
  2. the gap between devs and ops

Traditionally, developers (disconnected from operations) create software package. This is wrong. The purpose of packages is to move software in to production. How can you write package scripts if you don't have an understanding of the operational environment? This is further compounded by the open ended nature of packaging tools. They don't enforce good policy about things like autostart. That leaves the package creators (developers) to guess.

The result of this is broken packages that are not automatable.

Note that I'm not advocating for ops to create packages without developer involvement. I'm saying that the devops approach is needed here. You have to open up communication channels. Set up a culture where sysadmins are involved in the packaging process. Do cross-team codereviews. Have sysadmins give presentations to developers about how the operational environment works. There are lots more ways to encourage this sort of communication. The result of this will be much improved software packages.

Now I realize that much (most?) software isn't packaged with the intent of installing it in the same organization or company. If that's your use case, that's fine too. I'm sure you have ops folks in your company, right? Invite them in to the process. Even if they aren't directly installing your software, they will still be able to help with operability issues.

Bottom line: operations must be involved in the process of crafting software packages. Let's devops this whole thing, people!

edit Seeing Jordan's comment reminds me that fpm might often be the right way to go here. fpm concentrates on just creating the package, not implementing policy around it. I think a world where everyone used fpn to create packages would be just fine because we wouldn't end up with all these horrible package scripts. :)

I'm philiph on twitter.

Copy link

siebenmann commented Feb 2, 2012

As a sysadmin I prefer packages to not autostart things but to set themselves up so that fully enabling them is easy once they've been configured. Ideally it should take a single standard-for-the-system command to enable all of the daemon starting (update-rc.d on Debian, chkconfig or the like on CentOS, etc), and this enabling should definitely include starting on reboot.

Copy link

jaredmorrow commented Feb 2, 2012

@jordansissel I watched your FPM talk, and have looked at it heavily for inspiration. It was long past due and a great gift to the community btw! We have some internal workings of something similar that is more heavily based around erlang and rebar so we can sanely package multiple erlang apps in a similar fashion.

@tels7ar packaging is hard. That's it.

No seriously we have several different users of riak. We have paying customers who mostly go the route of using our Chef cookbook or our profserv people to do the install. Understandably, it is a big priority to get things 100% good there, and how we figured out that Something Changed (tm) in our packages this go around. We also have OSS users of riak who use it production, some use Chef, some take our packages and know exactly what to do with them, and some even take our source and build their own packages (the NetBSD community did this). Lastly we have people who download the OSS version to just try it out. They will most likely put it on an EC2 free micro instance that's 32bit ubuntu. Getting things working perfect all these places is hard, possibly impossible, but it still needs to be done. I think the latter group wants it to autostart and just work, but they are probably the only group who wants that. So understandably when there are clashes in wants and needs, we will lean towards what we think is sane and what will make sense for the majority of our users. Auto-start on install is not wanted by 98% of people, and it doesn't make sense for riak, so we won't do that. Creating rc.d startup scripts though is something I think many people DO want, but we need to put a stop gap in there to make sure it has been configured first. We have ops / profserv people and they are all involved, but I asked on twitter mostly to get a view from outside our company. IMHO that is a REALLY good thing to do if you have to make dogmatic decisions like what a package is going to do when installed and configured. I'm certainly glad I did and got the feedback that I did. So I appreciate your comments.

Copy link

tjake commented Feb 3, 2012

@lusis We did disable autostart in DSE 1.0.1 release. I'm not aware of anything using /tmp in DSE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment