README.md

Riak-KV dumper

Export a riak-kv node's objects to a file tree.

History

Given that:

• A desire to have a tangible representation of a riak-kv's node's content á la mysqldump exists;
• Existing tools that could fulfil the above, all fail in one way or another. That is:

  • Every incantation that I could think of of

    riak-admin backup <node> <cookie> <filename> [node|all]

    ... produced, apart from this depreciation warning:

    riak-admin backup has been deprecated and its use is not recommended. Please see the documentation for more information on how to safely backup a cluster.

    ... exactly zero files.

  • https://github.com/basho-labs/riak-data-migrator

    git clone https://github.com/basho-labs/riak-data-migrator.git
    cd riak-data-migrator
    mvn clean
    mvn package
    cd target
    java -jar riak-data-migrator-0.2.9.jar -h

    Error: A JNI error has occurred, please check your installation and try again
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
          at java.lang.Class.getDeclaredMethods0(Native Method)
          at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    [...]
        ... 7 more
    

  • https://www.npmjs.com/package/riak-bucket-exporter

    npm install -g riak-bucket-exporter

    /usr/local/bin/riak-bucket-exporter -> /usr/local/lib/node_modules/riak-bucket-exporter/bin/riak-bucket-exporter.js
    
    > wtf8@0.2.0 install /usr/local/lib/node_modules/riak-bucket-exporter/node_modules/wtf8
    > node-gyp rebuild
    
      CXX(target) Release/obj.target/wtf8/wtf8.o
    In file included from ../wtf8.cc:4:
    In file included from ../../nan/nan.h:74:
    ../../nan/nan_new.h:23:43: error: no member named 'Handle' in namespace 'v8'
    template <typename T> v8::Local<T> To(v8::Handle<v8::Integer> i);
                                          ~~~~^
    [...]
    fatal error: too many errors emitted, stopping now [-ferror-limit=]
    20 errors generated.
    make: *** [Release/obj.target/wtf8/wtf8.o] Error 1
    gyp ERR! build error
    gyp ERR! stack Error: `make` failed with exit code: 2
    gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
    [...]
    npm ERR! Failed at the wtf8@0.2.0 install script.
    npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
    
    npm ERR! A complete log of this run can be found in:
    npm ERR!     /Users/david/.npm-cache/_logs/2020-01-24T12_11_00_858Z-debug.log
    

... I ultimately resorted to concoct the shell script included below;

Prerequisites

• curl;
• jq;
• An operational HTTP API on the riak node to export from;

Use

There are no command line arguments to supply; parameterization is purely driven by environment variables. These, and their defaults are:

HOST=localhost
PORT=8098
BUCKET_TYPE=default

So, a compeletely parameterized invocation could look like:

HOST=example.com PORT=1234 BUCKET_TYPE=cusomt-bt rkvd.sh

This will create a file tree in the current working directory, conforming to this pattern:

${BUCKET_TYPE}/${BUCKET}/${KEY}/{head,body}

... where head and body will contain the corresponding sections of the response to the HTTP request below:

GET /types/${BUCKET_TYPE}/buckets/${BUCKET}/keys/${KEY}

The script will simply walk the listed buckets and their keys to fetch each object and store it as described above;

Knowing available bucket types

Curiously, riak-kv's HTTP API does not seem to provide a means to list available bucket-types; so, to discover these, one could:

riak-admin bucket-type list

The script doesn't bother to do walk these as well but one could easily wrap a loop around the output of the above to achieve that:

( export BUCKET_TYPE;  while IFS='' read BUCKET_TYPE ;  do rkvd.sh ;  done < <( riak-admin bucket-type list | sed -e 's/\s*(active)//' ))

Note this implies that rkvd.sh be run on the machine that hosts the riak-kv node; (and you probably should sudo riak-admin ... too).

Limitations

The script is only as complete as I currently need it to be. There's probably "some" ground left to cover.

rkvd.sh

#!/usr/bin/env bash

##  Some UNIX-es do not allow arguments beyond the shebang line's first, so any
##  shell options should be set explicitly instead, like so:

##  Exit immediately when any command fails.
##
set -e


##  Define utils;
##
uri_encode ()
{
    jq --slurp --raw-input --raw-output '.[0:-1]|@uri' <<<"${1}"
}

list_fetch ()
{
    curl --silent -- "${1}?${2}=true" | jq --raw-output ".${2}[] | @uri"
}

object_fetch ()
{
    local OBJECT_URL="${2}/${1}"
    local OBJECT_PATH="${3}/${1}"

    (   mkdir -p -- "${OBJECT_PATH}" \
    &&  cd -- "${OBJECT_PATH}"       \
    &&  echo "${OBJECT_URL}"        \
    &&  curl --dump-header head --output body -- "${OBJECT_URL}"
    )
}


##  Set defaults;
##
HOST=${HOST:-localhost}
PORT=${PORT:-8098}
BUCKET_TYPE=${BUCKET_TYPE:-default}


##  Sanity checks;
##

##  These values must be defined and non-empty:
##
[[ -n "${HOST:?}" ]]
[[ -n "${PORT:?}" ]]
[[ -n "${BUCKET_TYPE:?}" ]]


##  Encode;
##
BUCKET_TYPE_ENCODED=$( uri_encode "${BUCKET_TYPE}" )
HOST_ENCODED=$( uri_encode "${HOST}" )
PORT_ENCODED=$( uri_encode "${PORT}" )


##  It would seem that no HTTP API exists for listing all bucket types; instead one can do:
##
##  ```bash
##  riak-admin bucket-type list
##  ```


##  Go:
##
URL_BUCKETS="http://${HOST_ENCODED}:${PORT_ENCODED}/types/${BUCKET_TYPE_ENCODED}/buckets"

while IFS='' read -r BUCKET_ENCODED; do

    URL_KEYS="${URL_BUCKETS}/${BUCKET_ENCODED}/keys"

    while IFS='' read -r KEY_ENCODED; do

        object_fetch            \
            "${KEY_ENCODED}"    \
            "${URL_KEYS}"       \
            "${BUCKET_TYPE_ENCODED}/${BUCKET_ENCODED}"

    done < <( list_fetch "${URL_KEYS}" keys )

done < <( list_fetch "${URL_BUCKETS}" buckets )