Skip to content

Instantly share code, notes, and snippets.

View pnorman's full-sized avatar

Paul Norman pnorman

View GitHub Profile
@pnorman
pnorman / gist:a61c9fa4ddaa31a00603ab8f7daddc3d
Last active March 31, 2024 23:41
journalctl -u zfs-import-cache.service
Mar 31 15:20:50 merry systemd[1]: Starting zfs-import-cache.service - Import ZFS pools by cache file...
Mar 31 15:20:50 merry lsblk[1295]: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
Mar 31 15:20:50 merry lsblk[1295]: sda 8:0 1 1.9T 0 disk
Mar 31 15:20:50 merry lsblk[1295]: ├─sda1 8:1 1 1.9T 0 part
Mar 31 15:20:50 merry lsblk[1295]: └─sda9 8:9 1 8M 0 part
Mar 31 15:20:50 merry lsblk[1295]: sdb 8:16 1 1.9T 0 disk
Mar 31 15:20:50 merry lsblk[1295]: ├─sdb1 8:17 1 1.9T 0 part
Mar 31 15:20:50 merry lsblk[1295]: └─sdb9 8:25 1 8M 0 part
Mar 31 15:20:50 merry lsblk[1295]: sdc 8:32 1 223.6G 0 disk
Mar 31 15:20:50 merry lsblk[1295]: ├─sdc1 8:33 1 512M 0 part /boot/efi
@pnorman
pnorman / blog.md
Created February 29, 2024 21:05
tilekiln demo

Minutely Shortbread tiles

With this year being the year of OpenStreetMap vector maps I've been working on making vector tile maps that update minutely. Most maps don't need minutely updates and are fine with daily or, at most, weekly. Minutely updates on OpenStreetMap.org are a crucial part of the feedback cycle where mappers can see their edits right away and get inspired to map more often. Typically a mapper can make an edit and see their edit when reloading after 90-180 seconds, compared to the days or weeks of most OSM-based services, or the months or years of proprietary data sources.

Updating maps once a week can be done with a simple architecture that takes the OSM file for the planet and turns it into a single file containing all the tiles for the world. This can scale to daily updates, but not much faster. To do minutely updates we need to generate tiles one-by-one, since they change one-by-one.

@pnorman
pnorman / cuts.md
Last active April 4, 2024 16:42
cuts.md

The board is requiring that the OWG cut its budget from the 2024 proposal. Although we're still waiting to see what this means in terms of actual opex, capex and cash flow numbers, we already know that this will involve significant cuts. The OWG budget already had a cash flow about half of the previous two years budgets and we had already deferred as many capital expendutures as possible to bring the capex down by 75% over our 2023 request and by 80% of our 2022 request. This means there's nothing "optional" to trim and everything that can be deferred has been.

The only way to get any meaningful cuts to cash flow and opex is to go to operating a single primary site while maintaining current power usage or slightly higher.

Putting aside the technical details, this means loss of redundancy and cutting back some services. We would still maintain redundancy within a single site so if a single machine fails we don't have primary services go down. The savings given here are very rough as additional research a

from requests_oauthlib import OAuth2Session
client_id = r'0IrEz7VIm6zis60pP1jKW5ZXdLpVP0yojuG8Hy45z1o'
client_secret = r'Iejv415Rd-E0YqY2knXOrL8a1cmnJ-LtKMxPdmuvkyI'
redirect_uri = 'urn:ietf:wg:oauth:2.0:oob'
scope = ["read_prefs"]
oauth = OAuth2Session(client_id=client_id, redirect_uri=redirect_uri, scope=scope)
authorization_url, state = oauth.authorization_url('https://www.openstreetmap.org/oauth2/authorize')
iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAMAAABrrFhUAAAAvVBMVEX+/v7//wDm9NTp9ebM57LK5snI5qfS7LbL4aLX7ru74qaczoGe0oSl1Yu34J3X7czF0Zy00qzJsZK52pqGlIxodWt3d3dteoBdZ152i3S33bnK1afV7NfD3MpmZmbF2uy948Gbrpe2z853dwC74LzR4u+lurFXV1cBAQGHh4fNuKWXl5fHx8e70uiou82IiACoqKjX19d5hong3dguMDI+P0Lo6OhGRkZTTU2uuXM2NjdJRjonJye2trYXFxcc6TAJAAAeq0lEQVR42u1dCXubOBNmg880tw11c9A07RZRlBbWGFgK+v8/65tDAnxl28RJPyeofRJic2hezYyk0avBsrrSla50pStd6cqbL3+1Svvzg4MD2950gd3r9Q82fnFw8KoA6A2Gw8FovCKUPRwcvjs6Ojo+OV2R/nQ8Hr4mAKwBluFg0Du1myYejw8PD0+Ojt4BAoPmi9PecDwen44PXhMAB0OCYDAanQ1tkP0URTwdAgDnoADHI/hmMDxAUFD4Xm+M378mAGxWACgXcDAm+ca9QywAwKA3GEymjuO47x1n9uH8Es4Zno7tVwSANR5oFQAABgTAoS7Hx4PB1Tvnw3n/2rbt6+uby+nsw8lw3HtdABwYAM5QF0AHsPlPNAAn3ofLa+gnGIHrj7efZtPxae/0NQFgaQAuRgTAcDw8PJmcIAYnx+9m59dYEAE6uL65mzqD0/H+AfB5OwDDFgCIwHA8OOHy4cOX67+v69InBG6/+ld72BGKz1sBYBs4uzgDBM40Gij+8YcpN/91u/S/3H6dne+fExAtBKzNTuDs4mICIMC4aDhEAD59WBcfyu0t6MD1HgLQIGBtdAKj0WQCZjAkK8CB4OwWxUf/x+p/2seffQDg9pO7jwDUCKx+NyTrH01Gg4srrQ3D3qHD/g9VwNYugP4hALfv7/YRAIOAtcEGRoeDCQwGz05ORjgwGvXOP9yi1H9f
@pnorman
pnorman / pgconf.md
Created January 11, 2024 17:33
pgconf.dev 2024 submission

Title

OpenStreetMap and PostgreSQL

Abstract

As the largest crowd-sourced map project, OpenStreetMap uses PostgreSQL throughout its stack, and one which we use in unusual ways.

I will be taking you through

  • how we run a multi-TB database in a production service accessed by the public 24/7;
  • the challenges of schema changes when we have seven different programs using the database, all maintained by different people;
  • why we are using PostgreSQL Logical replication to produce the files to let users update their data every minute;
Label Visits Actions Maximum actions in one visit Total time spent by visitors (in seconds) Bounces Visits with Conversions Create account View Mapnik Edit Map Conversions Unique visitors (daily sum)
Facebook 120178 227871 435 6966080 89948 8331 737 204 7599 10199 118338
Instagram 61978 69629 84 504231 58085 1228 66 39 1205 1506 61646
www.immobiliare.it 20121 24202 34 370624 17564 1062 19 1059 4 1139 19863
leafletjs.com 3599 8625 83 400762 2127 889 32 870 27 1877 3549
accounts.google.com 1232 16351 240 758979 1 786 349 274 220 1106 1231
en.m.wikipedia.org 17434 20980 63 301387 15406 599 7 596 8 852 17183
en.wikipedia.org 4813 6771 45 181809 3906 483 12 481 16 800 4785
statuspnr.in 18718 21853 34 425968 16651 430 5 428 4 451 17888
TikTok 3303 7990 59 157051 2130 329 16 1 317 380 3247
@pnorman
pnorman / proposal.md
Last active February 4, 2024 02:34
vector tiles proposal 2023-11-24

Vector tiles proposal

Introduction

Background

There has been interest in a client-side rendered map on osm.org using vector tiles for a few years

Client-side rendering offers advantages for high-DPI devices, particularly mobile devices, with crisper rendering, rotation support, fractional zooms, and better labeling. Right now over half of osm.org traffic is mobile devices, where if they viewed the map, the standard layer would be blurry due to its resolution.

#!/usr/bin/env python3
import argparse
import csv
from math import log
from PIL import Image
ZOOM = 10
SCALE_MAX = 1_500_000
sudo fincore /store/database/nodes; uptime
odin: 14.4G/83.1G, 312d
ysera: 14.4G/83.1G, 411d
pyrene: 38.5G/83.1G, 414d
culebre: 6.1G/83.1G, 183d
nidhogg: 6.7G/83.1G, 223d
palulukon: 26.9G/83.1G, 291d
balerion: 43G/83.1G, 138d
bowser: 44.9G/83.1G, 335d