A node-red flow which runs an archive routine, uploading data to Dropbox for safe keeping!

Node-red archive to dropbox flow

[{"id":"3857b636.3f0c9a","type":"comment","z":"4487e413.bb781c","name":"Nightly Backups","info":"","x":144,"y":1212,"wires":[]},{"id":"d7570d3d.383af","type":"inject","z":"4487e413.bb781c","name":"Start backup","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"00 03 * * *","once":false,"x":151,"y":1262,"wires":[["d3620081.6ce2a"]]},{"id":"86146d4a.44769","type":"function","z":"4487e413.bb781c","name":"Triggered queue","func":"// if queue doesn't exist, create it\ncontext.queue = context.queue || [];\ncontext.busy = context.busy || false;\n\n// if the msg is a trigger one release next message\nif (msg.hasOwnProperty(\"trigger\")) {\n    if (context.queue.length > 0) {\n        var m = context.queue.shift();\n        return {payload:m};\n    }\n    else {\n        context.busy = false;\n      //  node.send({payload:\"result\"});\n      var msg2 = { payload:\"OK\" };\n    }\n}\nelse {\n    if (context.busy) {\n        // if busy add to queue\n        context.queue.push(msg.payload);\n    }\n    else {\n        // otherwise we are empty so just pass through and set busy flag\n        context.busy = true;\n        return msg;\n    }\n}\n\nreturn [null,msg2];","outputs":"2","noerr":0,"x":330,"y":1317,"wires":[["37086894.134c48"],["bc2d8cea.d1e1d"]]},{"id":"f455a591.cf6ea8","type":"function","z":"4487e413.bb781c","name":"set trigger","func":"// handle the return from the exec in here \n// if all is good then set msg.trigger property to exist\nmsg.trigger = 1;\nreturn msg;","outputs":1,"noerr":0,"x":601,"y":1316,"wires":[["86146d4a.44769"]]},{"id":"37086894.134c48","type":"exec","z":"4487e413.bb781c","command":"","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"Execute commands","x":631,"y":1261,"wires":[["f455a591.cf6ea8"],[],["5fb57ced.cc8774"]]},{"id":"bc2d8cea.d1e1d","type":"function","z":"4487e413.bb781c","name":"Read filenames","func":"var path = ('/home/pi/.temp_nodered/');\nvar fs = (global.get('fse'));\n\nfs.readdirSync(path).forEach(file => {\n  node.send({localFilename: \"/home/pi/.temp_nodered/\"+(file)});\n});\nreturn null;","outputs":1,"noerr":0,"x":328,"y":1358,"wires":[["d12bd518.7b64b8"]]},{"id":"d12bd518.7b64b8","type":"dropbox out","z":"4487e413.bb781c","dropbox":"","filename":"","localFilename":"","name":"","x":603,"y":1358,"wires":[]},{"id":"d3620081.6ce2a","type":"function","z":"4487e413.bb781c","name":"Generate commands","func":"//--Edit these to your own installation\nvar emonpath = '/home/pi/data'; //location of emoncms datafile directory\nvar mysqlUsername = \"emoncms\"; //MYSQL username\nvar mysqlPassword = \"emonpiemoncmsmysql2016\"; //MYSQL password\n//Don't change these!\nvar path = '/home/pi/.temp_nodered/';\nvar tmp_path = '/tmp/mysql_backup/';\n\n//Generate days to add to archive title\nvar d = new Date();\nvar day = new Array(7);\nday[0]=  \"Sun\";\nday[1] = \"Mon\";\nday[2] = \"Tue\";\nday[3] = \"Wed\";\nday[4] = \"Thu\";\nday[5] = \"Fri\";\nday[6] = \"Sat\";\n\n//Create directory structure\nvar m1={payload: \"rm -rf \"+[path]};\nvar m2={payload: \"mkdir \"+[path]};\nvar m3={payload: \"mkdir \"+[tmp_path]};\n\n// ************************************************************************* //\n// *** The below commands will archive the entire node-red user directory,\n// *** emoncms data directories & mysql. To add further commands, use the\n// *** same format, and add the corresponding msg id to the 'return' statement.\n// ************************************************************************* //\n//---NODE-RED---//\n  //Create ~/.node-red archive\n  var m4={payload: \"tar -czvf \"+[path]+(day[d.getDay()])+\"_nodered\"+\".tar.gz /home/pi/.node-red\"};\n//---EMONCMS---//\n  //Stop apache to enable emoncms data backup\n  var m5={payload: \"sudo service apache2 stop\"};\n  //Archive emoncms feeddata\n  var m6={payload: \"tar -czvf \"+[path]+(day[d.getDay()])+\"_emoncms\"+\".tar.gz \"+[emonpath]};\n  //Restart apache\n  var m7={payload: \"sudo service apache2 start\"};\n  //export emoncms database to RAM, then archive it to .temp_nodered/\n  var m8={payload: \"mysqldump --lock-tables --user=\"+[mysqlUsername]+\" --password=\"+[mysqlPassword]+\" emoncms > \"+[tmp_path]+\"emoncms.sql\"};\n  var m9={payload: \"tar -czvf \"+[path]+(day[d.getDay()])+\"_mysql\"+\".tar.gz /tmp/mysql_backup\"};\n  var m10={payload: \"rm -rf \"+[tmp_path]}; //delete old tmp directory from RAM\n// ****************************************************** //\n\n// Output the commands for execution\nreturn [ [ m1, m2, m3, m4, m5, m6, m7, m8, m9, m10 ] ];","outputs":"1","noerr":0,"x":355,"y":1262,"wires":[["86146d4a.44769"]]},{"id":"5fb57ced.cc8774","type":"function","z":"4487e413.bb781c","name":"Report any errors","func":"var code = (msg.payload.code);\nif ([code] != \"0\") {\nmsg.payload = (msg.payload.message);\nmsg.topic = \"Raspberry backup process failed\";\nreturn msg;\n}\nelse {\nreturn null;\n}","outputs":1,"noerr":0,"x":337,"y":1401,"wires":[[]]},{"id":"e25b51a7.6cf8c","type":"comment","z":"4487e413.bb781c","name":"README (select & view in info panel!)","info":"**REQUIREMENTS**  \nRequires the library fs-extra installing & enabling\nglobally in settngs.js  \n(used in `Read Filenames` function node)\n\nTO INSTALL FS-EXTRA  \n```\ncd ./node-red  \nnpm install --save fs-extra\n```\nTO MAKE FS-EXTRA GLOBAL  \nAdd `fse:require('fs-extra')` to the\n`functionGlobalContext` section in NR settings.js file.\n\n**CONFIGURATION**  \nEdit the `Generate commands` function node with \nyour own;  \n- Emoncms data directory path\n- Emoncms MYSQL username (default = emoncms)\n- Emoncms MYSQL password  \n\nSetup the `dropbox` node, (details in the node info)  \nAlso edit the `Start backup` inject node with your preferred time to run.  \nIf you want to be alerted about archive problems or failures, you can use the `Report any errors` node output to dispatch the message via Pushover, email, twitter or whatever.","x":400,"y":1211,"wires":[]}]

PURPOSE
The flow was predominantly written to run a nightly archive of emoncms data directories, emoncms MYSQL database, and the entire node-red user directory, uploading them to dropbox for safe keeping, but can be easily adapted for other data sources.
The resultant tar.gz files will include the 'day of the week' in their title, so you can quickly identify the relevant archive to restore, and at the end of 7 days, the file will be overwritten with the current day's data. (However Dropbox retain 30 days worth of file versions - more on paid accounts, so it's easy to locate and restore archives up to 30 days old).
For emoncms/emonpi users - please note that by default, this flow assumes that all of your data sub-directories are included in one directory which will be archived . Also, further commands would need to be added to change the filesystem to read-write, and back to read-only (easy to do - just see examples in the Generate commands node.

REQUIREMENTS
Requires the library fs-extra installing & enabling
globally in settngs.js
(used in Read Filenames function node)

TO INSTALL FS-EXTRA

cd ./node-red  
npm install --save fs-extra

TO MAKE FS-EXTRA GLOBAL
Add fse:require('fs-extra') to the
functionGlobalContext section in NR settings.js file.

CONFIGURATION
Edit the Generate commands function node with
your own;

• Emoncms data directory path
• Emoncms MYSQL username (default = emoncms)
• Emoncms MYSQL password

Setup the dropbox node, (details in the node info)
Also edit the Start backup inject node with your preferred time to run.
If you want to be alerted about archive problems or failures, you can use the Report any errors node output to dispatch the message via Pushover, email, twitter or whatever. Currently, the msg.topic is the message title, and msg.payload is the error message.

Hi Paul,

I'm trying out your flow, (thanks BTW), but have a problem trying to locate the folder where the Emoncms data is stored.

/home/pi/data doesn't exist and I've spend several hours googling to try and locate it but no luck.

Any ideas about how I might find it?

Cheers

Shane.

You can find where your data files are stored by looking at your emoncms settings.php, which will tell you. For example;

        // Engines working folder. Default is /var/lib/phpfiwa,phpfina,phptimeseries
        // On windows or shared hosting you will likely need to specify a different data directory--
        // Make sure that emoncms has write permission's to the datadirectory folders
        'phpfiwa'=>array(
            'datadir' => '/mnt/data/emondata/phpfiwa/'
        ),
        'phpfina'=>array(
            'datadir' => '/mnt/data/emondata/phpfina/'
        ),
        'phptimeseries'=>array(
            'datadir' => '/mnt/data/emondata/phptimeseries/'
        ),
        'cassandra'=>array(
            'keyspace' => 'emoncms'
        )

Hi Paul
I've only just come to using node-red from using BBC Basic - years ago,

I'm having problems with this error on a node-red only backup installation.

29/05/2019, 10:15:16node: Read filenames
function : (error)
"TypeError: Cannot read property 'readdirSync' of undefined"

I'm also not getting any error messages sent to my e-mail, which I attached to the 'Report any errors' node

Paul,
Have you updated this to work with the new dropbox authorization process? I moved a node red install to a new pc and tried to use your wonderful code again with no luck. The access tokens only last 4 hours now.... It worked just once..
Thoughts?

@MuddyVT I think your question is more for the author of node-red-node-dropbox to answer, as my flow simply collates all of the required node-RED files, compresses them, and feeds the resulting archive into the default dropbox node for uploading.

Perhaps a post in the forum may prompt some feedback from the node-RED team, who maintain the dropbox node.