Dockerfile for running OpenTSDB

README.md

Dockerfile for OpenTSDB

This Dockerfile is generated, that is why it looks so ugly. The main reason was that I did not want to have any external dependencies. Instead of using ADD I decided to go for writing files using this technique

echo <base64decoded> | base64 -d > /tmp/<md5> && mv /tmp/<md5> <dst>

and just inline everything in the Dockerfile.

OpenTSDB is started with the --auto-metric flag so you do not need to create metrics before you write to them.

Test OpenTSDB 2.0

By removing the comment in line:

# RUN cd /opt/opentsdb && git checkout origin/next

you can already test OpenTSDB 2.0

Testing

You need to download the Dockerfile and build a new image:

docker build -t opentsdb .
id=$(docker run -d opentsdb)
port=$(docker port $id 4242)
echo "started opentsdb in container $id, listening on $port"

Now you can wait for OpenTSDB to be fully started (this can take a while)

for i in $(seq 1 30); do
  curl -sfI 127.0.0.1:$port > /dev/null 2>&1
  if [ "$?" == "0" ]; then
    echo "opentsdb running and listening on port $port"
    break
  fi
  echo -n "."
  sleep 1
done

The OpenTSDB frontend is now available on the displayed port.

With OpenTSDB running you can start to feed your first metrics:

while true; do
  line="put load.loadavg1m $(date +%s) $(cat /proc/loadavg  | cut -d ' ' -f 1) host=$(hostname)"
  echo $line
  echo $line | nc 127.0.0.1 $port
  sleep 1
done

and with that loop running you can have a look at your metrics:

 http://127.0.0.1:$port/q?png&m=avg:load.loadavg1m&start=10m-ago

Dockerfile

FROM ubuntu
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y curl build-essential git-core
RUN if [ ! $(grep universe /etc/apt/sources.list) ]; then sed 's/main$/main universe/' -i /etc/apt/sources.list && apt-get update; fi
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y openjdk-6-jdk
RUN mkdir -p /opt/downloads && cd /opt/downloads && curl -SsfLO "http://www.apache.org/dist/hbase/hbase-0.94.10/hbase-0.94.10.tar.gz"
RUN cd /opt && tar xvfz /opt/downloads/hbase-0.94.10.tar.gz
RUN mkdir -p /data/hbase
RUN mkdir -p /root/.profile.d
RUN mkdir -p /root && echo Zm9yIHByb2ZpbGVfcGF0aCBpbiAkKGZpbmQgL3Jvb3QvLnByb2ZpbGUuZC8gLXR5cGUgZik7IGRvIHNvdXJjZSAkcHJvZmlsZV9wYXRoOyBkb25lCg== | base64 -d > /tmp/wunderscale.e4e4b9a9fbb0a7813a4707407606ccb195999effc1d27eac75a879113a4a017b && mv /tmp/wunderscale.e4e4b9a9fbb0a7813a4707407606ccb195999effc1d27eac75a879113a4a017b /root/.profile
RUN mkdir -p /root/.profile.d && echo ZXhwb3J0IEpBVkFfSE9NRT0vdXNyL2xpYi9qdm0vamF2YS02LW9wZW5qZGstYW1kNjQK | base64 -d > /tmp/wunderscale.701a6427b3c95cf75c1ffbb200e2f5034c6ed0039d7810977c355596bdf7c62f && mv /tmp/wunderscale.701a6427b3c95cf75c1ffbb200e2f5034c6ed0039d7810977c355596bdf7c62f /root/.profile.d/java
RUN mkdir -p /opt && echo IyEvYmluL2Jhc2gKc291cmNlIC9yb290Ly5wcm9maWxlCmVjaG8gIjEyNy4wLjAuMSBsb2NhbGhvc3QgJChob3N0bmFtZSkiID4gL2V0Yy9ob3N0cwplY2hvICJzdGFydGluZyBoYmFzZSIKL29wdC9oYmFzZS0wLjk0LjEwL2Jpbi9oYmFzZSBtYXN0ZXIgc3RhcnQgPj4gL3Zhci9sb2cvaGJhc2UubG9nIDI+JjEgJgoKZWNobyAid2FpdGluZyBmb3IgaGJhc2UiCmZvciBpIGluICQoc2VxIDEgMTApOyBkbwoJdGltZW91dCAxMCBlY2hvIHN0YXR1cyB8IC9vcHQvaGJhc2UtMC45NC4xMC9iaW4vaGJhc2Ugc2hlbGwKCWlmIFsgJD8gPT0gIjAiIF07IHRoZW4KCQllY2hvICIiCgkJZWNobyAic3RhcnRlZCBoYmFzZSIKCQlleGl0IDAKCWZpCgllY2hvIC1uICIuIgpkb25lCmVjaG8gImVycm9yIHN0YXJ0aW5nIGhiYXNlIgpleGl0IDEK | base64 -d > /tmp/wunderscale.0ef0732d9abddce0ca3dc3f5adbf6a9fbea745de5f5ec3bdbb14ca6d6a7bf279 && chown root /tmp/wunderscale.0ef0732d9abddce0ca3dc3f5adbf6a9fbea745de5f5ec3bdbb14ca6d6a7bf279 && chmod 755 /tmp/wunderscale.0ef0732d9abddce0ca3dc3f5adbf6a9fbea745de5f5ec3bdbb14ca6d6a7bf279 && mv /tmp/wunderscale.0ef0732d9abddce0ca3dc3f5adbf6a9fbea745de5f5ec3bdbb14ca6d6a7bf279 /opt/start_hbase
RUN mkdir -p /opt/hbase-0.94.10/conf && echo PD94bWwtc3R5bGVzaGVldCB0eXBlPSJ0ZXh0L3hzbCIgaHJlZj0iY29uZmlndXJhdGlvbi54c2wiPz4KPGNvbmZpZ3VyYXRpb24+CgogIDxwcm9wZXJ0eT4KICAgIDxuYW1lPmhiYXNlLnJvb3RkaXI8L25hbWU+CiAgICA8dmFsdWU+ZmlsZTovLy8vZGF0YS9oYmFzZS9oYmFzZVwtJHt1c2VyLm5hbWV9L2hiYXNlPC92YWx1ZT4KICA8L3Byb3BlcnR5PgogIDxwcm9wZXJ0eT4KICAgIDxuYW1lPmhiYXNlLnpvb2tlZXBlci5kbnMuaW50ZXJmYWNlPC9uYW1lPgogICAgPHZhbHVlPmxvPC92YWx1ZT4KICA8L3Byb3BlcnR5PgogIDxwcm9wZXJ0eT4KICAgIDxuYW1lPmhiYXNlLnJlZ2lvbnNlcnZlci5kbnMuaW50ZXJmYWNlPC9uYW1lPgogICAgPHZhbHVlPmxvPC92YWx1ZT4KICA8L3Byb3BlcnR5PgogIDxwcm9wZXJ0eT4KICAgIDxuYW1lPmhiYXNlLm1hc3Rlci5kbnMuaW50ZXJmYWNlPC9uYW1lPgogICAgPHZhbHVlPmxvPC92YWx1ZT4KICA8L3Byb3BlcnR5Pgo8L2NvbmZpZ3VyYXRpb24+ | base64 -d > /tmp/wunderscale.aa1f175aa6faa7bc5c831d87944eae946a41b1b9dae619c64a7fddbc26f3233d && mv /tmp/wunderscale.aa1f175aa6faa7bc5c831d87944eae946a41b1b9dae619c64a7fddbc26f3233d /opt/hbase-0.94.10/conf/hbase-site.xml
EXPOSE 60000
EXPOSE 60010
EXPOSE 60030

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y automake gnuplot
RUN rm -Rf /opt/opentsdb && git clone git://github.com/OpenTSDB/opentsdb.git /opt/opentsdb
# RUN cd /opt/opentsdb && git checkout origin/next
RUN cd /opt/opentsdb && bash ./build.sh
RUN mkdir -p /opt && echo CiMhL2Jpbi9iYXNoCgpleHBvcnQgQ09NUFJFU1NJT049Ik5PTkUiCmV4cG9ydCBIQkFTRV9IT01FPS9vcHQvaGJhc2UtMC45NC4xMApleHBvcnQgSkFWQV9IT01FPS91c3IvbGliL2p2bS9qYXZhLTYtb3Blbmpkay1hbWQ2NAoKY2QgL29wdC9vcGVudHNkYgouL3NyYy9jcmVhdGVfdGFibGUuc2gKdG91Y2ggL29wdC9vcGVudHNkYl90YWJsZXNfY3JlYXRlZC50eHQK | base64 -d > /tmp/wunderscale.41f9f65722120150cd889ab932a345fef051d55d88fbd2791e5f95ec92e86ab0 && chown root /tmp/wunderscale.41f9f65722120150cd889ab932a345fef051d55d88fbd2791e5f95ec92e86ab0 && chmod 755 /tmp/wunderscale.41f9f65722120150cd889ab932a345fef051d55d88fbd2791e5f95ec92e86ab0 && mv /tmp/wunderscale.41f9f65722120150cd889ab932a345fef051d55d88fbd2791e5f95ec92e86ab0 /opt/create_tsdb_tables
RUN mkdir -p /opt && echo CiMhL3Vzci9iaW4vZW52IGJhc2gKCmlmIFsgISAtZSAvb3B0L29wZW50c2RiX3RhYmxlc19jcmVhdGVkLnR4dCBdOyB0aGVuCgllY2hvICJjcmVhdGluZyB0c2RiIHRhYmxlcyIKCWJhc2ggL29wdC9jcmVhdGVfdHNkYl90YWJsZXMKCWVjaG8gImNyZWF0ZWQgdHNkYiB0YWJsZXMiCmZpCgplY2hvICJzdGFydGluZyBvcGVudHNkYiIKL29wdC9vcGVudHNkYi9idWlsZC90c2RiIHRzZCAtLXBvcnQ9NDI0MiAtLXN0YXRpY3Jvb3Q9L29wdC9vcGVudHNkYi9idWlsZC9zdGF0aWNyb290IC0tY2FjaGVkaXI9L3RtcCAtLWF1dG8tbWV0cmljCg== | base64 -d > /tmp/wunderscale.d93b1768866fc8ed39d4c7f8c3ef5ddc49e70a2ec6de2cc81d9de5ed0b145f6c && chown root /tmp/wunderscale.d93b1768866fc8ed39d4c7f8c3ef5ddc49e70a2ec6de2cc81d9de5ed0b145f6c && chmod 755 /tmp/wunderscale.d93b1768866fc8ed39d4c7f8c3ef5ddc49e70a2ec6de2cc81d9de5ed0b145f6c && mv /tmp/wunderscale.d93b1768866fc8ed39d4c7f8c3ef5ddc49e70a2ec6de2cc81d9de5ed0b145f6c /opt/start_opentsdb
EXPOSE 4242
CMD bash -c "/opt/start_hbase && { /opt/create_tsdb_tables; /opt/start_opentsdb; }"

It seems that "echo status | hbase shell" succeeds even when the HBase cluster is not ready. This makes the script creating tables fails with org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error.

See HBASE-5534

I have used this instead:

for i in $(seq 1 10); do
    timeout 10 echo "status 'simple'" | \
        /opt/hbase-0.94.10/bin/hbase shell 2>/dev/null | \
        grep -q 'numberOfOnlineRegions=[1-9]'
    if [ $? == "0" ]; then
        log_end_msg 0
        exit 0
    fi
    log_progress_msg "."
done

Good point, I just ran into a error regarding this. I will have a look

Awesome, just playing around with OpenTSDB and ran into this… die Welt ist klein!