There are some interesting database technologies emerging from the node.js arena. One that I have contributed to is leveldb.
leveldb integrates the leveldb key value store implemented by google with node.js bindings provided by leveldown and leveldown is then abstracted into a more javascript friendly and easier to use API in levelup.
You can obtain all the parts separately but level is the packaged distribution of all of these technologies in a nice and handy place.
One of the key questions I have heard is "how do we provide backup and restore functionality?"
When I discussed this on IRC (##leveldb) there was a couple of suggestions.
- replicate the data using level-master.
- Use Linux Volume Manager (LVM)
While 1. was an option it seemed to be a little heavy for plain backup purposes.
I was intrigued by option 2. the LVM option but I use joyent SmartOS and other illumOS systems for deployment so I decided to investigate what would be involved in using ZFS.
ZFS is a combined file system and logical volume manager designed by Sun Microsystems and is often described as a "modern" file system and given my experience with it I can see why it deserves such an epithet.
While my production deployments are on illumOS based systems I know that Linux is more readily installed so this article will describe how to use level and ZFS technology on Ubuntu and I will append the commands to the bottom of this post to use on illumOS based systems.
- node.js v 0.8 or above
- npm v 1.1.x or above 3, Ubuntu Raring 13.04 ZFS may be available for earlier versions but YMMV.
sudo apt-get install zfs-fuse
The above command worked for me on 13.04 but recent literature has the following instructions:
apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install debootstrap ubuntu-zfs
Which ever method you used for the install of ZFS the following command should work.
sudo zpool list
no pools available
In your home folder create a folder called zfslevel the create 4 files
mkdir zfslevel
cd zfslevel
dd if=/dev/zero of=level_test1 bs=134217728 count=1
dd if=/dev/zero of=level_test2 bs=134217728 count=1
dd if=/dev/zero of=level_test3 bs=134217728 count=1
dd if=/dev/zero of=level_test4 bs=134217728 count=1
sudo zpool create leveldata mirror ~/zfslevel/level_test1 ~/zfslevel/level_test2
And now the command sudo zpool status leveldata
Should give the output:
pool: leveldata
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
leveldata ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/home/anton/levelzfs/level_test1 ONLINE 0 0 0
/home/anton/levelzfs/level_test2 ONLINE 0 0 0
errors: No known data errors
stat /leveldata
File: ‘/leveldata/’
Size: 2 Blocks: 3 IO Block: 512 directory
Device: 19h/25d Inode: 1 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-05-29 17:01:46.000000000 +0100
Modify: 2013-05-29 16:50:23.000000000 +0100
Change: 2013-05-29 16:50:23.000000000 +0100
Birth: -
OK So now we have a mirrored file system lets create an application to use it.
cd to root of home make a directory called levapp and make an index.js and install level
cd ~
mkdir levapp
cd levapp
touch index.js
npm install level
Now edit index.js and enter the following code into it
var level = require('level')
// 1) Create our database, supply location and options.
// This will create or open the underlying LevelDB store.
var db = level('/leveldata/mydb')
// 2) put a key & value
db.put('name', 'Level number 2', function (err) {
if (err) return console.log('Ooops!', err) // some kind of I/O error
// 3) fetch by key
db.get('name', function (err, value) {
if (err) return console.log('Ooops!', err) // likely the key was not found
// ta da!
console.log('name=' + value)
})
})
now run the script with
sudo env PATH=$PATH node index.js
This will return with name=Level
I would recommend adding
alias sudo='sudo env PATH=$PATH'
to .bashrc
So we can now see our database has been created on the pool we created by running ls or zpool list
ls /leveldata/mydb/
000005.sst 000006.log CURRENT LOCK LOG LOG.old MANIFEST-000004
sudo zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
leveldata 123M 350K 123M 0% 1.00x ONLINE -
Snapshots create a point of reference that is only takes up space the more changes are applied to the current database.
To create a snapshot
sudo zfs snapshot leveldata@friday
change the line
db.put('name', 'Level', function (err) {
to
db.put('name', 'Level number 2', function (err) {
now run the script with
sudo env PATH=$PATH node index.js
This will return with name=Level
Lets look at the snapshot
sudo zfs list -r -t snapshot -o name,creation leveldata
NAME CREATION
leveldata@friday Wed May 29 20:00 2013
OK so lets roll this back
sudo zfs rollback leveldata@friday
Now lets create a script that just gets the data so we can see the rollback touch read.js
var level = require('level')
// 1) Create our database, supply location and options.
// This will create or open the underlying LevelDB store.
var db = level('/leveldata/mydb')
// 3) fetch by key
db.get('name', function (err, value) {
if (err) return console.log('Ooops!', err) // likely the key was not found
// ta da!
console.log('name=' + value)
})
now
sudo env PATH=$PATH node read.js
name=Level
For more info on snapshots see http://docs.huihoo.com/opensolaris/solaris-zfs-administration-guide/html/ch06.html
So we have seen snapshots but now we want to put the data somewhere else so we can restore it if there is an incident.
We will start off by creating a backup device in this case another ZFS pool. IRL this would be a different set of devices or an SSH connection but here we will use the other to file disks we created at the start
sudo zpool create levelbackup mirror ~/zfslevel/level_test3 ~/zfslevel/level_test4
And then we need to move the data from live to backup
# sudo zfs send leveldata@friday | sudo zfs receive levelbackup@friday
So lets over right the data
sudo env PATH=$PATH node index.js
name=Level number 2
then lets remove all the files in /leveldata/mydb
sudo rm -fr /leveldata/mydb
and the previous snapshot
sudo zfs destroy leveldata@friday
Now lets restore the backup
sudo zfs send levelbackup/test@friday | sudo zfs recv -F leveldata
You may find that the command returns cannot receive new filesystem stream: I/O error but this does not seem to effect the restore
Now we need to see the data that was restored using the read.js file
sudo env PATH=$PATH node read.js
for more info on backups see http://breden.org.uk/2008/05/12/home-fileserver-backups-from-zfs-snapshots/