This article introduces Fwissr, the configuration registry tool that we use to centralize all our configurations.
Photo by National Library of Norway
More than five years ago, when we started to build what became fotopedia, we had a main Rails application, a Merb application for user authentication and another Merb application for photo upload. These applications were deployed by Capistrano, and our servers were configured with Puppet with some scripts to glue everything together.
As the system evolved, we slowly got rid of many of these applications and started adding other components in the application stack. Puppet was replaced by Chef, and shortly thereafter, a new service landed in the stack, written in Java (and later, with parts in Scala).
All the pieces of this puzzle needed the same settings, and it became clear for our mental sanity that we needed a unified way to access these settings. These settings would be written by some authority (Chef, in our case, but YMMV), and all services would access these settings, independantly of their source language. This is how Fwissr was born, as a Ruby gem, a Python package and a Maven artifact.
The ruby gem is used in all our ruby applications, whether Rails based or console based. The Python package in our deploy scripts (using fabric), and the Java library in our java/Scala middlewares.
Chef is responsible for populating the registry in our MongoDB configuration cluster. Only the "admin" instance is allowed to update this part of the registry. This ensures configuration consistency across the servers.
This way, every time a new service needs to have access to the configuration of our platform, Fwissr is available for a quick and simple answer. Compared to using the Chef API, it provides much faster access and configuration-less usage. By using a simple "exporter", we serialize parts of the Chef registry directly into the MongoDB fwissr configuration.
[[MORE]]
This post is more about the Ruby version of Fwissr, but the usage is the same in Python and Java, please refer to their respective [pages]((http://search.maven.org/#artifactdetails%7Ccom.fotonauts%7Cfwissr%7C0.0.3%7Cjar) to find out about their particularities.
Fwissr is a ruby gem, so as usual, install it with:
$ [sudo] gem install fwissr
Then drop a JSON file to ~/.fwissr/ directory:
$ mkdir ~/.fwissr
$ cat <<EOS > ~/.fwissr/my_app.json
{
"aws": {
"access_key_id": "POpoOIHiyugUYF",
"secret_access_key": "OIJioHUhuiUIH",
"assets_bucket": "my_assets"
}
}
EOS
Now you can access these settings in your ruby code:
require 'fwissr'
bucket_name = Fwissr["/my_app/aws/assets_bucket"]
puts "Assets are stored to the #{bucket_name} bucket"
# => "Assets are stored to the my_assets bucket"Fwissr provides a tool that can be used in shell scripts:
for file in `find ./ -type f ! -name '.*' ! -mmin +60`; do
export AWS_ACCESS_KEY_ID=`fwissr /my_app/aws/access_key_id`
export AWS_SECRET_ACCESS_KEY=`fwissr /my_app/aws/secret_access_key`
s3cmd -v -d put `fwissr /my_app/aws/assets_bucket`:${file#./} ${file#./} "Cache-Control: max-age = 315576000, public" "Expires: Tue, 17 Sep 2019 11:40:58 GMT" "x-amz-acl: public-read"
done
Dump Fwissr registry with:
$ fwissr --dump -jp
{
"my_app": {
"aws": {
"secret_access_key": "OIJioHUhuiUIH",
"access_key_id": "POpoOIHiyugUYF",
"assets_bucket": "assets"
}
}
}
Note that one of the strengths of the system is that there is no need to configure it inside your framework, you can just use the Fwissr provider transparently and have access to its data at a very small cost.
Fwissr loads all json and yaml files found in the directories /etc/fwissr/ and ~/.fwissr/ then merges them to build a global dictionary (the registry).
File names are used to prefix keys in registry. For example all settings from configuration file my.great.app.json start with /my/great/app/ prefix.
The configuration file fwissr.json is a special one: its settings are not prefixed, and you can specify additional configuration sources inside this file thanks to the fwissr_sources setting.
For example, to load all configuration files in /etc/my_app/ directory, edit /etc/fwissr/fwissr.json to:
{
"fwissr_sources": [
{ "filepath": "/etc/my_app/" },
]
}
In addition to configuration files, you can setup configuration databases. The only database kind supported for now is MongoDB.
For example add that source to fwissr.json:
{
"fwissr_sources": [
{ "mongodb": "mongodb://db1.example.net/my_app", "collection": "production" },
]
}
Fwissr will now load all the documents in the production collection of the my_app database. In that collection, the document's _id field is the setting key and the value field is the setting value.
> db["production"].find()
{ "_id" : "foo", "value" : "bar" }
{ "_id" : "database", "value" : { "host": "db.my_app.com", "port": "1337" } }
Enable registry auto-update with the refresh source setting.
For example:
{
"fwissr_sources": [
{ "filepath": "/etc/my_app/my_app.json" },
{ "filepath": "/etc/my_app/stuff.json", "refresh": true },
{ "mongodb": "mongodb://db1.example.net/my_app", "collection": "production" },
{ "mongodb": "mongodb://db1.example.net/my_app", "collection": "config", "refresh": true }
]
}The /etc/my_app/my_app.json configuration file and the production mongodb collection are read once, whereas the settings holded by the /etc/my_app/stuff.json configuration file and the config mongodb collection expire periodically and re-fetched.
The default refresh period is 30 seconds, but you can change it with the fwissr_refresh_period setting:
{
"fwissr_sources": [
{ "filepath": "/etc/my_app/my_app.json" },
{ "filepath": "/etc/my_app/stuff.json", "refresh": true },
{ "mongodb": "mongodb://db1.example.net/my_app", "collection": "production" },
{ "mongodb": "mongodb://db1.example.net/my_app", "collection": "config", "refresh": true }
],
"fwissr_refresh_period": 60
}Everything is relased under the MIT License for the Ruby and Python version and ASF license for the Java version. Feel free to download and use them.
- Ruby source code at Github
- Ruby code documentation at rubydoc
- Python source code at Github
- Python code documentation at Read The Docs
- Java source code at Github
From Fotonauts: