Paraphraser/IOTstack + InfluxDB 2 experiments.md

## IOTstack + InfluxDB 2 experiments.md

      
    Raw
  

              IOTstack + InfluxDB 2 experiments.md
            
          
    IOTstack: InfluxDB 2 experiments

Update #1:

Rewrite assuming user only has a single Raspberry Pi which is processing live data. Minimises the down time for InfluxDB 1.8.

 
This gist explains the steps I followed to:

Migrate my InfluxDB 1.8 databases to InfluxDB 2.
Change a Node-RED flow to append to a migrated database.
Build a single-panel dashboard in Grafana fetching from a migrated database.


contents


summary of findings


assumptions


terminology: database vs bucket


reference service definitions


InfluxDB 1.8


InfluxDB 2

required edits
migration phase summary


data-migration procedure


checking the migration


Flux queries via point-and-click


example: Node-RED


example: Grafana

defining an InfluxDB 2 data source
using an InfluxDB 2 data source in a dashboard


odds and ends


migration strategies


summary of findings


The process of migrating InfluxDB 1.8 databases to InfluxDB 2 is slightly convoluted but seems to be reliable.
Adjusting Node-RED to write to InfluxDB 2 databases is straightforward.
Getting Grafana to connect to InfluxDB 2 databases is easy.
InfluxDB 2 implies Flux but building Grafana dashboards with Flux queries is sub-optimal because Grafana does not have a point-and-click interface for constructing Flux queries. The InfluxDB 2 web GUI does have such an interface. A workaround is to copy and paste lines of Flux code between InfluxDB 2 and Grafana.
It is feasible to run both InfluxDB 1.8 and InfluxDB 2, side-by-side, where both are receiving the same data from Node-RED (ie the flow ends with two InfluxDB-out nodes, one for each database engine).


assumptions


Your Raspberry Pi is running full 64-bit Raspberry Pi OS Debian GNU/Linux 11 (bullseye).

DockerHub does not have 32-bit image for InfluxDB 2 so you can't follow this gist until you have upgraded.


You are using SensorsIot/IOTstack to manage your Docker containers.

This gist relies on SensorsIot/IOTstack conventions. You may still get useful hints and ideas from this gist if you are not running IOTstack but you will have to adapt it to your environment.


You are currently running InfluxDB 1.8.


Node-RED is your principal mechanism for feeding data to InfluxDB 1.8.

You may have other services feeding data to InfluxDB 1.8 (eg Telegraf). The steps in this gist will migrate that data but the gist does not discuss how to change other services to feed data to InfluxDB 2.


Grafana is your principle mechanism for creating dashboards based on data stored in InfluxDB 1.8.

You may have other visualisation tools. You may gain insights from studying how Grafana needs to be changed to run Flux queries against InfluxDB 2 buckets but this gist does not explore alternatives.


Node-RED, InfluxDB 1.8 and Grafana are all running in non-host mode on the same Docker instance.

If you are running any containers in host mode or have distributed the services across multiple Docker instances, you will have to adapt appropriately.


terminology: database vs bucket

InfluxDB 1.8 and InfluxDB 2 are both database management systems (DBMS), sometimes referred to as "engines", optimised for storage and retrieval of time-series data. InfluxDB 1.8 uses the term database to mean a collection of measurements. InfluxDB 2 uses the term bucket to mean the same thing.
When an InfluxDB 1.8 database is migrated, it becomes an InfluxDB 2 bucket. You will see this change in terminology in various places, such as the InfluxDB-out node in Node-RED. When that node is set to:


Version 1.x, the UI has a "Database" field which travels with the connection. For example:

[v1.x] influxdb:8086/power (set up in the connection sheet)

This implies that you need one connection per database.


Version 2.0, the UI has a "Bucket" field which is independent of the connection. For example:

[v2.0] influxdb:8086 (set up in the connection sheet)
Bucketpower/autogen (set up in the node)

This implies that you need one connection per engine. It is a subtle but important difference.


This gist treats database and bucket as synonyms.

reference service definitions


InfluxDB 1.8

  influxdb:
    container_name: influxdb
    image: "influxdb:1.8"
    restart: unless-stopped
    environment:
      - TZ=Etc/UTC
      - INFLUXDB_HTTP_AUTH_ENABLED=false
      - INFLUXDB_HTTP_FLUX_ENABLED=false
      - INFLUXDB_REPORTING_DISABLED=false
      - INFLUXDB_MONITOR_STORE_ENABLED=FALSE
    ports:
      - "8086:8086"
    volumes:
      - ./volumes/influxdb/data:/var/lib/influxdb
      - ./backups/influxdb/db:/var/lib/influxdb/backup
    healthcheck:
      test: ["CMD", "curl", "http://localhost:8086"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
Notes:

Don't worry if you have different environment variables and/or settings. Whatever you have running now is OK.
The healthcheck: directives are optional. You can adopt them if you wish. See also Issue 415. These changes are part of Pull Request 563.


InfluxDB 2

  influxdb2:
    container_name: influxdb2
    image: "influxdb:latest"
    restart: unless-stopped
    environment:
    - TZ=Etc/UTC
    - DOCKER_INFLUXDB_INIT_USERNAME=me
    - DOCKER_INFLUXDB_INIT_PASSWORD=mypassword
    - DOCKER_INFLUXDB_INIT_ORG=myorg
    - DOCKER_INFLUXDB_INIT_BUCKET=mybucket
    - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
    - DOCKER_INFLUXDB_INIT_MODE=setup
#   - DOCKER_INFLUXDB_INIT_MODE=upgrade
    ports:
    - "8087:8086"
    volumes:
    - ./volumes/influxdb2/data:/var/lib/influxdb2
    - ./volumes/influxdb2/config:/etc/influxdb2
    - ./volumes/influxdb2/backup:/var/lib/backup
#   - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro
    healthcheck:
      test: ["CMD", "influx", "ping"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

required edits

Copy the InfluxDB 2 service definition and paste it into your compose file. Then, change the following variables:


TZ=«country»/«city»


DOCKER_INFLUXDB_INIT_USERNAME=«username»
This name becomes the administrative user. It is associated with your «password» and «token».


DOCKER_INFLUXDB_INIT_PASSWORD=«password»
Your «username» and «password» form your login credentials when you access the web-based GUI for InfluxDB 2. The strength of your password is up to you.


DOCKER_INFLUXDB_INIT_ORG=«organisation»
An organisation name is required. Examples:

myorg
my-house
com.mydomain.myhouse


DOCKER_INFLUXDB_INIT_BUCKET=«bucket»
A default bucket name is required. The name does not matter because you won't actually be using it so you can accept the default of "mybucket". You can delete the bucket later if you want to be tidy.


DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=«token»
Although you can let InfluxDB 2 generate your access token for you, it will keep things simple if you generate your own. Here are some possibilities:


Use a universally-unique ID:
$ uuidgen
4fef85b4-2f56-480f-b143-fa5cb6e8f18a


Use GnuPG to generate a random string:
$ gpg --gen-random -a 0 25
bYS3EsnnY0AlRxJ2uk44Hzwm7GMKYu5unw==


Use a password-generator of your choosing.


Note:

Unless a container's documentation explicitly states that it is supported, you should never use quote marks to encapsulate the values you supply via environment variables. InfluxDB 2 treats quotes as being part of the value (eg a password of "fred" is the 6-character string that includes the quotes). If you put quote marks around anything as you were editing, please go back and remove them.


migration phase summary

Migrating your data from InfluxDB 1.8 databases to InfluxDB 2 buckets is performed in three steps:


phase 1 : where DOCKER_INFLUXDB_INIT_MODE=setup
Initialises the database engine structures.


phase 2 : where DOCKER_INFLUXDB_INIT_MODE=upgrade
InfluxDB 1.8 databases are migrated to InfluxDB 2 buckets.


phase 3 : where DOCKER_INFLUXDB_INIT_MODE is omitted
Normal operation.


The table below summarises which variables and directives need to be active in each phase:


service definition directive
phase 1
phase 2
phase 3


DOCKER_INFLUXDB_INIT_USERNAME
✔︎
✔︎
✘


DOCKER_INFLUXDB_INIT_PASSWORD
✔︎
✔︎
✘


DOCKER_INFLUXDB_INIT_ORG
✔︎
✔︎
✘


DOCKER_INFLUXDB_INIT_BUCKET
✔︎
✔︎
✘


DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
✔︎
✔︎
✘


DOCKER_INFLUXDB_INIT_MODE
setup
upgrade
✘


./volumes/influxdb.migrate/data:/var/lib/influxdb:ro
✘
✔︎
✘


data-migration procedure


We begin with the assumption that you have been using InfluxDB 1.8 with a service definition similar to the reference definition above. It is not necessary to have that exact service definition.


Be in the correct directory (assumed throughout):
$ cd ~/IOTstack


InfluxDB 1.8 runs as root and its persistent store is owned by root but not all files and folders in the persistent store are group or world readable. InfluxDB 2 runs as user ID 1000 (user "influxdb" inside the container). Because of this, you need to give InfluxDB 2 permission to read the InfluxDB 1.8 persistent store.
It is not a good idea to interfere with a persistent store while a container is running so the first step is to stop InfluxDB 1.8 for long enough to make a copy of its persistent store.
$ sudo rm -rf ./volumes/influxdb.migrate ./volumes/influxdb2
$ docker-compose rm --force --stop -v influxdb
$ sudo cp -a ./volumes/influxdb ./volumes/influxdb.migrate
$ docker-compose up -d influxdb
$ sudo chown -R 1000:1000 ./volumes/influxdb.migrate/data
In words:

Ensures any previous attempts at migration are removed and that you are starting from a clean slate.
Stop InfluxDB 1.8.
Make a copy of the InfluxDB 1.8 persistent store.
Start InfluxDB 1.8 again - downtime is limited to the time-to-copy.
Change ownership of the copy of the InfluxDB 1.8 persistent store so InfluxDB 2 can read it.


If you have not already done so, add the InfluxDB 2 service definition to your compose file and make the required edits.


Start the InfluxDB 2 container:
$ docker-compose up -d influxdb2


InfluxDB 2 will notice the following environment variable:
DOCKER_INFLUXDB_INIT_MODE=setup

This instructs the container to initialise the database engine structures based on a combination of defaults and the values you provide via the other environment variables. This is "phase 1" of the migration. You can use commands like:
$ docker ps
$ docker logs influxdb2
to confirm that the InfluxDB 2 container is not in a restart loop and isn't reporting errors.


InfluxDB 2 creates a "bolt" (lock) file to prevent accidental double-migrations. That file needs to be removed before moving to the next phase:
$ rm ./volumes/influxdb2/data/influxd.bolt


Edit your compose file. The changes you need to make are:


Change the initialisation mode from setup to upgrade:


before editing:
     - DOCKER_INFLUXDB_INIT_MODE=setup
 #   - DOCKER_INFLUXDB_INIT_MODE=upgrade


after editing:
 #   - DOCKER_INFLUXDB_INIT_MODE=setup
     - DOCKER_INFLUXDB_INIT_MODE=upgrade


Activate the volume mapping to give InfluxDB 2 read-only access to the InfluxDB 1.8 persistent store:


before editing:
 #   - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro


after editing:
     - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro


Save your work.


The InfluxDB 2 container is still running. The following command causes the container to be recreated with the edits you have just made:
$ docker-compose up -d influxdb2


InfluxDB 2 will notice the following environment variable:
DOCKER_INFLUXDB_INIT_MODE=upgrade

This, combined with the absence of the "bolt" file, starts the migration process. You need to wait until the migration is complete. The simplest way to do that is to just to watch the size of the persistent store for InfluxDB 2 until it stops increasing. My experience is that the persistent store for InfluxDB 2 is a bit larger than the persistent store for InfluxDB 1.8. For example:


reference size for my InfluxDB 1.8
 $ sudo du -sh ./volumes/influxdb
 633M	./volumes/influxdb


final migration size for my InfluxDB 2
 $ sudo du -sh ./volumes/influxdb2
 721M	./volumes/influxdb2


Phase 2 is complete once the folder size stops changing. Edit your compose file again. The changes you need to make are:


Deactivate all DOCKER_INFLUXDB_INIT_ environment variables:


before editing:
     - DOCKER_INFLUXDB_INIT_USERNAME=me
     - DOCKER_INFLUXDB_INIT_PASSWORD=mypassword
     - DOCKER_INFLUXDB_INIT_ORG=myorg
     - DOCKER_INFLUXDB_INIT_BUCKET=mybucket
     - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
 #   - DOCKER_INFLUXDB_INIT_MODE=setup
     - DOCKER_INFLUXDB_INIT_MODE=upgrade


after editing:
 #   - DOCKER_INFLUXDB_INIT_USERNAME=me
 #   - DOCKER_INFLUXDB_INIT_PASSWORD=mypassword
 #   - DOCKER_INFLUXDB_INIT_ORG=myorg
 #   - DOCKER_INFLUXDB_INIT_BUCKET=mybucket
 #   - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
 #   - DOCKER_INFLUXDB_INIT_MODE=setup
 #   - DOCKER_INFLUXDB_INIT_MODE=upgrade

If you wish, you can also remove those variables from your compose file. Just make sure you keep a copy of your credentials somewhere.


Deactivate the volume mapping:


before editing:
     - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro


after editing:
 #   - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro

If you wish, you can remove that mapping from your compose file.


Save your work.


The InfluxDB 2 container is still running. The following command causes the container to be recreated with the edits you have just made:
$ docker-compose up -d influxdb2


If you are short of disk space, it is safe to remove the copy of the InfluxDB 1.8 persistent store:
$ sudo rm -rf ./volumes/influxdb.migrate


The migration is now complete!


checking the migration


Launch a browser and connect it to port 8087 on your Raspberry Pi. For example:
http://raspberrypi.local:8087

You can also use the IP address or domain name of your Raspberry Pi. In this context, 8087 is the external port number.


Sign in to the InfluxDB 2 instance using your «username» and «password».


Click on "Explore" in the left-hand tool strip. That is marked [A] in the screen shot. In the area marked [B] you should be able to see a list of the buckets that were migrated from InfluxDB 1.8 databases.
In the screen shot, I clicked on other fields to create a query:

In area [B], I selected the "power/autogen" bucket;
In area [C], I selected the "hiking2" measurement;
In area [D], I selected the "voltage" field;
The bucket in this test is a migrated copy of an InfluxDB 1.8 database. It isn't ingesting live data so I also needed to change the duration popup menu [E] to a time-span that included the most-recent insertions;
Then I clicked the "Submit" button [F]; and
The result was the graph in [G].


You can explore your own tables using similar techniques.

Flux queries via point-and-click

Although this is skipping ahead a bit, Grafana does not (yet) seem to have the smarts to let you build Flux queries via point-and-click (like you can with InfluxQL queries). As an interim solution, it's probably a good idea to learn how to build Flux queries in InfluxDB and so you can copy-and-paste the Flux statements into Grafana.
Once you have constructed a query in the "Query Builder", click the "Script Editor" button [H] to switch to the editor view.
For this example, the query text is:
from(bucket: "power/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "hiking2")
  |> filter(fn: (r) => r["_field"] == "voltage")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

Two important things to note here are:

The bucket name: power/autogen; and
The measurement name: hiking2.


example: Node-RED


Assume you have an existing flow (eg a standard 3-node flow) which is logging to an InfluxDB 1.8 database.
Your goal is to modify the flow to log the same data to the recently-migrated InfluxDB 2 bucket.


If Node-RED is not running, start it:
$ docker-compose up -d nodered


Use a web browser to connect to your Node-RED instance.


Drag a new InfluxDB-out node onto the canvas:

This is exactly the same InfluxDB-out node that you have been using to write to your InfluxDB 1.8 databases.
Always drag a new InfluxDB-out node from the palette onto the canvas. Do not make the mistake of re-using an existing InfluxDB-out node (eg via copy and paste) because that is a very good way of breaking your flows.


Double-click the InfluxDB-out node to open it:


At [A], give the node a sensible name.


Click the pencil icon [B] adjacent to the Server field:


Leave the Name field [C] blank. This ensures that the title in the popup menu [D] automatically reflects the version and connection URL.


Change the Version popup menu [E] to "2.0".


Set the URL [F] to point to your InfluxDB 2 instance:
 http://influxdb2:8086


In this context, "influxdb2" is the container name and 8086 is the internal port. Node-RED is communicating with InfluxDB 2 across the internal bridged network (see assumptions).


Paste your «token» into the Token field [G].


Click "Update" [H].


Set the Organisation field [I] to your «organisation».


Set the Bucket [J] to the correct value. You can get that from either:

area [B] in the Influx Explorer screen shot; or
the bucket name from the saved Flux query.

In this example, the value is "power/autogen".


Set the Measurement [K] to the measurement name. You can get that from either:

area [C] in the Influx Explorer screen shot; or
the measurement name  from the saved Flux query.

In this example, the value is "hiking2".


Click Done [L].


Connect the outlet of the Change node to the inlet of the InfluxDB-out node.


Click Deploy.


Watch the debug panel to make sure no errors are being reported.


Go back to the InfluxDB 2 Data Explorer and click the refresh button "I". If everything has gone according to plan, you should see recent observations added to your graph.

You may need to wait until your sensor has sent new data.


example: Grafana


defining an InfluxDB 2 data source


Start Grafana if it is not running:
$ docker-compose up -d grafana


Use a web browser to connect to your Grafana instance and login as an administrator.


Hover your mouse over the gear icon in the tool-strip on the left hand side, and choose "Data sources".


Click the "Add data source" button.


Select the "InfluxDB" option.


Configure as follows:


Change the Name [A] to a  meaningful title that reflects the bucket you are going to query. For example, "InfluxDB 2.0 power".


Change the Query Language popup menu [B] to "Flux".

Ignore the advice about Flux support being in beta.


Change the URL [C] to point to your InfluxDB 2 instance:
 http://influxdb2:8086


In this context,  "influxdb2" is the container name and 8086 is the internal port. Grafana is communicating with InfluxDB 2 across the internal bridged network (see assumptions).


Turn off all the switches in the "Auth" group [D].


Set the Organisation [E] to your «organisation».


Paste your «token» into the Token field [F].

ignore the fact that the prompt text says "password" - you need the token!


Set the Default Bucket [G] to the bucket (database) you want to query. You can get that from either:

area [B] in the Influx Explorer screen shot; or
the bucket name from the saved Flux query.

In this example, the value is "power/autogen".


Click Save & Test [H].


using an InfluxDB 2 data source in a dashboard


Find the + icon in the tool-strip on the left hand side, hover your mouse over it and choose "Create » dashboard".
Click "Add a new panel".
Change the "Data source" popup to the bucket connection you created earlier ("InfluxDB 2.2 power").
The editor automatically switches into Flux mode.
Paste the query text you saved earlier from the InfluxDB 2 query inspector.
If necessary, change the duration to a period that is likely to contain some data to display.
Click the Refresh button.
Click Apply.

In the side-by-side screen shots below, observations before the straight-line segment were imported from InfluxDB 1.8 while observations after the straight-line segment were inserted by the new InfluxDB-out node in Node-RED.


odds and ends


Forgot your token?
$ docker exec influxdb2 influx auth ls


Create a new user, password and token:
$ docker exec influxdb2 influx user create --name «username» --password «password»
$ docker exec influxdb2 influx auth create --user «username» --all-access


Want to delete the default «bucket»?
$ influx bucket delete --org «organisation» --name «bucket»


migration strategies

From the fact that both InfluxDB 1.8 and InfluxDB 2 can run in parallel, with Node-RED feeding the same data to both, it should be self-evident that you can repeat the data-migration as often as necessary:


Get a clean slate:
$ cd ~/IOTstack
$ docker-compose rm --force --stop -v influxdb2
$ sudo rm -rf ./volumes/influxdb2


Reinstate the reference service definition for InfluxDB 2.


Perform the steps in the data-migration procedure.


This implies that you can concentrate on one database at a time, adjusting Node-RED so that it writes each row of data to both the old InfluxDB 1.8 database and corresponding InfluxDB 2 bucket.
Having the data going to both database engines also means you can take your time adjusting your Grafana dashboards to be based on Flux queries. You can either retrofit InfluxDB 2 bucket sources and Flux queries to existing dashboards, or build parallel dashboards from the ground up.
service definition directive	phase 1	phase 2	phase 3
`DOCKER_INFLUXDB_INIT_USERNAME`	✔︎	✔︎	✘
`DOCKER_INFLUXDB_INIT_PASSWORD`	✔︎	✔︎	✘
`DOCKER_INFLUXDB_INIT_ORG`	✔︎	✔︎	✘
`DOCKER_INFLUXDB_INIT_BUCKET`	✔︎	✔︎	✘
`DOCKER_INFLUXDB_INIT_ADMIN_TOKEN`	✔︎	✔︎	✘
`DOCKER_INFLUXDB_INIT_MODE`	setup	upgrade	✘
`./volumes/influxdb.migrate/data:/var/lib/influxdb:ro`	✘	✔︎	✘