philippkeller

Prepare SD Card

• flash raspbian lite on micro sd card using etcher
• setup wireless and ssh, desribed here

setup raspberry

• passwd # set new password
• set authorized keys
• sudo raspi-config: set password, timezone, locale, enable spi

philippkeller

pi@raspberrypi:~/sonos-lcd $ pip3 install -U --force-reinstall pillow-simd
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pillow-simd
  Using cached https://files.pythonhosted.org/packages/a0/6a/30d21c886293cca3755b8e55de34137a5068b77eba1c0644d3632080516b/Pillow-SIMD-7.0.0.post3.tar.gz
Building wheels for collected packages: pillow-simd
  Running setup.py bdist_wheel for pillow-simd ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-c7i2f9uv/pillow-simd/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\
n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-7fv32_j6 --python-tag cp37:
  running bdist_wheel
  running build

philippkeller

Generate ms-regionen json for metabase

1. go to https://blog.wuestpartner.com/2017/04/umzuge-schweiz/
2. find topojson file at https://blog.wuestpartner.com/wp-content/uploads/2017/04/chms.topojson
3. convert the topojson -> geojson on https://mygeodata.cloud/converter/topojson-to-geojson
4. upload to s3 into a public bucket (e.g. hg-data-public)
5. Metabase -> Admin -> Maps -> Add Region map, paste the link from step 4

philippkeller

#!/usr/bin/env python3

import flatten_json
import sys
import json

if __name__ == '__main__':
	if len(sys.argv) < 3:
		print(f'usage: {sys.argv[0]} in.csv out.csv')
	f_out = open(sys.argv[2], 'w')

philippkeller

Getting Started With Superset: Airbnb’s data exploration platform

These instructions are for Amazon Linux Version 2

Update Python and PIP versions on EC2 (Amazon AMI)

sudo yum update -y
sudo yum install python3 -y

philippkeller

Mappings:
  RegionMap:
    ap-south-1:
      "AMALINUX" : "ami-531a4c3c"
    eu-west-3:
      "AMALINUX" : "ami-8ee056f3"
    eu-west-2:
      "AMALINUX" : "ami-403e2524"
    eu-west-1:
      "AMALINUX" : "ami-d834aba1"

philippkeller

/*
 * $Id: scanbd.conf 213 2015-10-05 06:52:50Z wimalopaan $
 *
 *  scanbd - KMUX scanner button daemon
 *
 *  Copyright (C) 2008 - 2015  Wilhelm Meier (wilhelm.meier@fh-kl.de)
 *
 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation; either version 2 of the License, or

philippkeller

global {
        # log to console. put to 'false' once you're done with testing
        debug   = true

        # 1=error, 2=warn, 3=info, 4-7=debug
        # 3 was sufficient for me (shows when script is triggered)
        debug-level = 3
        scriptdir = /etc/scanbd/scripts

philippkeller

import time
import tracemalloc
import pympler.muppy, pympler.summary
import gc
import sys

tracemalloc.start()

collection = []
start = time.time()

philippkeller

import datetime

with open("myfile.csv") as f:
	window_from = None
	window_to = None
	warm = cold = 0
	for line in f:
		milliseconds, topics = line.split(",")
		if milliseconds == 'milliseconds':
			continue