Skip to content

Instantly share code, notes, and snippets.

@derhuerst
Last active April 26, 2024 15:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save derhuerst/19fd358e17aa3ec3b093b1c134a1cbbc to your computer and use it in GitHub Desktop.
Save derhuerst/19fd358e17aa3ec3b093b1c134a1cbbc to your computer and use it in GitHub Desktop.
today i learned

2019-12-25

2019-12-09

how to use dtruss on macOS as an strace replacement

2019-12-05

finding AWDL devices:

# Start e.g. AirDrop from the GUI first!

# browse AirDrop devices:
dns-sd -i awdl0 -B _airdrop._tcp . | grep local.
# 21:50:45.009  Add        2   9 local.               _airdrop._tcp.       d6ca24342439
# 21:50:49.787  Add        2   9 local.               _airdrop._tcp.       c28e32bba720

# resolved the address of a specific AirDrop device:
dns-sd -i awdl0 -L d6ca24342439 _airdrop._tcp . | grep reached
# 21:51:27.767  d6ca24342439._airdrop._tcp.local. can be reached at jannis-ipad.local.:8770 (interface 9)

2019-12-04

how mDNS works

  • devices send DNS queries & responses to other local devices via IP multicast
  • "active" discovery by DNS PTR query
  • "passive" discovery by caching received announcements (SRV & TXT responses/records)

packet flow of a DNS-SD (used by mDNS a.k.a. Bonjour) address resolution:

  1. Alice actively searches for IPP devices: dig _ipp._tcp.local PTR -p 5353 @224.0.0.251
  2. Bob receives query, responds with announcement
  • bob._ipp._tcp.local. 10 IN SRV 0 0 631 bob.local.
  • `bob._ipp._tcp.local. 10 IN TXT "txtvers=1" "ty=Bobs Printer" "rp=ipp/print" …"

Record mDNS traffic using tcpdump -n host 224.0.0.251 and port 5353 and inspect DNS packets using dns-packet.

On macOS with e.g. AirDrop, AWDL is being used as a transport layer for DNS-SD packets.

2019-11-23

2019-10-22

2019-08-09

2019-07-24

2019-07-02

2020-11-25

2020-11-21

2020-11-06

2020-11-02

2020-10-25

2020-10-20

UNIX domain sockets (AF_UNIX) are pretty cool, because they guarantee the order of delivery, they don't lose data (under normal circumstances). They don't buffer data though, but propagate the "pressure" back, so the client will have to respect that.

// server
const {existsSync, unlinkSync} = require('fs')
const {createServer} = require('net')
const {Transform, pipeline} = require('stream')

const SOCKET = '/tmp/node-sock-stream'

if (existsSync(SOCKET)) unlinkSync(SOCKET)

const onError = (err) => {
	if (!err) return;
	console.error(err)
	process.exit(1)
}

createServer((con) => {
	pipeline(
		con,
		process.stdout,
		onError,
	)
})
.listen(SOCKET, (err) => {
	onError(err)
	console.info('listening on', SOCKET)
})
// client
const {connect} = require('net')
const {pipeline} = require('stream')

const onError = (err) => {
	if (!err) return;
	console.error(err)
	process.exit(1)
}

const con = connect('/tmp/node-sock-stream', (err) => {
	onError(err)

	// without flow control
	// The 1G of data will wait in this Node.js process.
	for (let i = 0; i < 1_000_000; i++) {
		con.write(Buffer.alloc(1000))
	}
	con.end()
	
	// with flow control
	pipeline(
		process.stdin,
		con,
		onError,
	)
})

They should also be supported on Windows by now.

2020-10-19

2020-10-18

performance of mterialized views in PostgreSQL

Sounds obvious, but if you have a query with several "nested" JOINs and WHERE clauses that apply to a table used by one of the "more nested" JOINs, it might not be worth using materialized views for speedups. The PostgreSQL query planner/optimizer is smart enough to turn the order of JOINs inside out, if it allows to start with a table a WHERE clause applies to. Materialized views, naively used to save PostgreSQL labor by allowing to pre-compute the inner JOINs, prevent this type of optimization, raising the runtime of my query from O(log(n) * m) to O(n m) or even O(n n m).

2020-10-04

cutoff=$(date +%s)
node -p "new Date(1000 * $cutoff).toISOString()" # 2020-11-27T22:33:22.000Z
journalctl -o cat -U @$cutoff -u some.service | gzip --best >some-service-2020-11-27T22:33:22Z.gz
journalctl --vacuum-size=2G

2020-09-23

2020-09-21

How to configure multiple IPv6 addresses for a VPS:

network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - 2a01:4f8:c2c:82fa:0:0:0:1/64
            - 2a01:4f8:c2c:82fa:0:0:0:2/64
            - 
            dhcp4: true
            gateway6: fe80::1
            set-name: eth0

It seems to fail with >5 though.

2020-08-14

2020-08-02

2020-06-29

2020-06-24

2020-06-22

  • use git checkout --ours file/git checkout --theirs file to resolve conflicts in a single file in favor of one side

2020-05-30

2020-05-28

2020-05-27

2020-05-26

2020-05-23

2020-05-21

2020-05-19

2020-05-12

2020-05-10

2020-05-09

2020-05-04

  • about the Server-Timing HTTP header (spec)
  • macOS: shared dir where guest should not be able to delete
    • chown -R jannis:everyone .
    • chmod -R g+w .
    • chmod -R +a 'user:guest deny delete,delete_child,file_inherit,directory_inherit . (how ACLs work)

2020-05-01

2020-04-29

2020-04-28

2020-04-27

2020-04-26

2020-04-25

2020-04-19

how to add a temporary port forwarding to your router:

brew install miniupnpc
# port-forward external port 443 to port 8443 of your device
upnpc -e 'mumble' -r 8443 443 tcp
# list port forwardings
upnpc -l
# remove port forwarding
upnpc -d 443 tcp

2020-03-16

Hwo to visualize BVG Wi-Fi scans using visualize-airport-util-wifi-scan:

cat scan.txt | parse-airport-util-wifi-scan \
	| ndjson-filter 'd.rssi >= -70' \
	| ndjson-filter 'd.name === "BVG Wi-Fi"' \
	| visualize-airport-util-wifi-scan >scan.html

2020-03-06

2020-02-29

2020-02-26

2020-02-17

how to fix node-gyp on macOS Catalina

2020-02-03

fetch only upper-cases the HTTP methods HEAD, OPTIONS, GET, PUT, DELETE & POST

2020-01-26

generate an HTML gallery using thumbsup:

brew install exiftool graphicsmagick dcraw
npx thumbsup --input src --include-raw-photos --photo-download copy --video-download copy --title foo --embed-exif --output dist

2021-12-14

2021-12-09

How does Tor change circuits?

Tor always serves a whole TCP connection over a single circuit otherwise a changed exit relay (new source IP address seen from the Internet) in a new circuit would cause the existing TCP connection to break.

By default Tor client stops using a circuit 10 minutes after its first use but existing connections will still be routed through their old circuits. It is explained here: How can I prevent my different activities carried out over Tor being linked?

– https://tor.stackexchange.com/questions/814/can-tor-change-circuit-while-accessing-a-webpage

2021-12-08

2021-10-17

The very first line cpu aggregates the numbers in all of the other cpuN lines.

These numbers identify the amount of time the CPU has spent performing different kinds of work. Time units are in USER_HZ or Jiffies (typically hundredths of a second).

The meanings of the columns are as follows, from left to right:

  • 1st column: user = normal processes executing in user mode
  • 2nd column: nice = niced processes executing in user mode
  • 3rd column: system = processes executing in kernel mode
  • 4th column: idle = twiddling thumbs
  • 5th column: iowait = waiting for I/O to complete
  • 6th column: irq = servicing interrupts
  • 7th column: softirq = servicing softirqs
  • Since Linux 2.6.11, there is an 8th column called steal - counts the ticks spent executing other virtual hosts (in virtualised environments like Xen)
  • Since Linux 2.6.24, there is a 9th column called guest - counts the time spent running a virtual CPU for guest operating systems under the control of the Linux kernel
  • Since Linux 2.6.33, there is a 10th column called guest_nice - Time spent running a niced guest (virtual CPU for guest operating systems under the control of the Linux kernel).

– How to read the Linux /proc/stat file

2021-10-09

The Grafana Agent is designed for easy installation and updates. It uses a subset of Prometheus code features to interact with hosted metrics, specifically:

  • Service discovery
  • Scraping
  • Write ahead log (WAL)
  • Remote writing
  • Along with this, the agent typically uses less memory than scraping metrics using Prometheus.

This optimization of Prometheus for remote write and resource reduction in the agent has resulted in some trade-offs:

  • Metrics are pushed rather than pulled
  • You can't query the agent directly; you can only query metrics from the remote write storage
  • Recording rules are not supported
  • Alerts from the agent are not supported

image – https://grafana.com/docs/grafana-cloud/fundamentals/gs-visualize/

2021-05-20

2021-05-14

2021-05-11

stuff about MobilityDB:

There are six built-in temporal types, namely tbool, tint, tfloat, ttext, tgeompoint, and tgeogpoint, which are, respectively, based on the base types bool, int, float, text, geometry, and geography (the last two types restricted to 2D or 3D points with Z dimension).

Temporal types based on discrete base types, that is the tbool, tint, or ttext evolve necesssarily in a stepwise manner. On the other hand, temporal types based on continuous base types, that is tfloat, tgeompoint, or tgeogpoint may evolve in a stepwise or linear manner.

The subtype of a temporal value states the temporal extent at which the evolution of values is recorded. Temporal values come in four subtypes, namely, instant, instant set, sequence, and sequence set.

A temporal value of instant subtype (briefly, an instant value) represents the value at a time instant, for example

SELECT tfloat '17@2018-01-01 08:00:00';

A temporal value of instant set subtype (briefly, an instant set value) represents the evolution of the value at a set of time instants, where the values between these instants are unknown. An example is as follows:

SELECT tfloat '{17@2018-01-01 08:00:00, 17.5@2018-01-01 08:05:00, 18@2018-01-01 08:10:00}';

A temporal value of sequence subtype (briefly, a sequence value) represents the evolution of the value during a sequence of time instants, where the values between these instants are interpolated […]. An example is as follows:

SELECT tint '(10@2018-01-01 08:00:00, 20@2018-01-01 08:05:00, 15@2018-01-01 08:10:00]';

As can be seen, a sequence value has a lower and an upper bound that can be inclusive (represented by [ and ]) or exclusive (represented by ( and )). […]

Finally, a temporal value of sequence set subtype (briefly, a sequence set value) represents the evolution of the value at a set of sequences, where the values between these sequences are unknown. An example is as follows:

SELECT tfloat '{[17@2018-01-01 08:00:00, 17.5@2018-01-01 08:05:00],
  [18@2018-01-01 08:10:00, 18@2018-01-01 08:15:00]}';

Temporal values with instant or sequence subtype are called temporal unit values, while temporal values with instant set or sequence set subtype are called temporal set values.

Temporal types support type modifiers (or typmod in PostgreSQL terminology), which specify additional information for a column definition. For example, in the following table definition:

CREATE TABLE Department(DeptNo integer, DeptName varchar(25), NoEmps tint(Sequence));

the type modifier for the type varchar is the value 25, which indicates the maximum length of the values of the column, while the type modifier for the type tint is the string Sequence, which restricts the subtype of the values of the column to be sequences. In the case of temporal alphanumeric types (that is, tbool, tint, tfloat, and ttext), the possible values for the type modifier are Instant, InstantSet, Sequence, and SequenceSet. If no type modifier is specified for a column, values of any subtype are allowed.

On the other hand, in the case of temporal point types (that is, tgeompoint or tgeogpoint) the type modifier may be used to specify specify the subtype, the dimensionality, and/or the spatial reference identifier (SRID). For example, in the following table definition:

CREATE TABLE Flight(FlightNo integer, Route tgeogpoint(Sequence, PointZ, 4326));

the type modifier for the type tgeogpoint is composed of three values, the first one indicating the subtype as above, the second one the spatial type of the geographies composing the temporal point, and the last one the SRID of the composing geographies. For temporal points, the possible values for the first argument of the type modifier are as above, those for the second argument are either Point or PointZ, and those for the third argument are valid SRIDs. All the three arguments are optional and if any of them is not specified for a column, values of any subtype, dimensionality, and/or SRID are allowed.

Each temporal type is associated to another type, referred to as its bounding box, which represent its extent in the value and/or the time dimension. The bounding box of the various temporal types are as follows:

  • The period type for the tbool and ttext types, where only the temporal extent is considered.
  • A tbox (temporal box) type for the tint and tfloat types, where the value extent is defined in the X dimension and the temporal extent in the T dimension.
  • A stbox (spatiotemporal box) type for the tgeompoint and tgeogpoint types, where the spatial extent is defined in the X, Y, and Z dimensions, and the temporal extent in the T dimension.

Chapter 3. Temporal Types – MobilityDB documentation

2021-05-07

As a convenience, the pipe &| redirects both stdout and stderr to the same process. Note that this is different from bash, which uses |&.

fish-shell introduction – Redirects 

2021-05-03

2021-04-30

2021-04-20

2021-04-06

2021-03-29

2021-03-27

2021-03-22

2021-03-16

2021-03-10

PostgreSQL query planning node types:

  • Scan Types
    • Sequential Scan
      • Basically a brute-force retrieval from disk
      • Scans the whole table
      • Fast for small tables
    • Index Scan
      • Scan all/some rows in an index; look up rows in heap
      • Causes random seek, which can be costly for old-school spindle-based disks
      • Faster than a Sequential Scan when extracting a small number of rows for large tables
    • Index Only Scan
      • Scan all/some rows in index
      • No need to lookup rows in the table because the values we want are already stored in the index itself
    • Bitmap Heap Scan
      • Scan index, building a bitmap of pages to visit
      • Then, look up only relevant pages in the table for desired rows
  • Join Types
    • Nested Loops
      • For each row in the outer table, scan for matching rows in the inner table
      • Fast to start, best for small tables
    • Merge Join
      • Zipper-operation on sorted data sets
      • Good for large tables
      • High startup cost if an additional sort is required
    • Hash Join
      • Build hash of inner table values, scan outer table for matches
      • Only usable for equality conditions
      • High startup cost, but fast execution

https://www.enterprisedb.com/blog/postgresql-query-optimization-performance-tuning-with-explain-analyze

2021-02-27

A single point in time can be represented by concatenating a complete date expression, the letter T as a delimiter, and a valid time expression. For example, 2007-04-05T14:30. In ISO 8601:2004 it was permitted to omit the T character by mutual agreement as in 200704051430, but this provision was removed in ISO 8601-1:2019. Separating date and time parts with other characters such as space is not allowed in ISO 8601, but allowed in its profile RFC 3339.

– ISO 8601: Combined date and time representations

2021-02-15

2021-02-14

2021-02-09

fish --private:

-P or --private enables private mode, so fish will not access old or store new history.

2021-02-08

2021-01-24

2021-01-22

2021-01-03

2022-12-30

2022-12-03

2022-11-30

2022-10-26

2022-10-24

  • Caddy v2 path matchers are not prefix-based except if you add *, so if you want to match any request starting with /, you must use /*.

2022-10-20

2022-10-14

2022-10-08

22022-09-26

2022-09-13

2022-09-08

2022-09-02

2022-08-30

about SQLite's "type affinity":

The type affinity of a column is the recommended type for data stored in that column. The important idea here is that the type is recommended, not required. Any column can still store any type of data.

about SQLite's "strict tables":

In a CREATE TABLE statement, if the STRICT table-option keyword is added to the end, after the closing ), then strict typing rules apply to that table.

2022-08-26

2022-08-18

2022-08-08

2022-08-07

2022-08-04

2022-08-01

2022-07-31

2022-07-30

2022-07-15

Bash 3.2, as used on macOS 12.2 🙄, does not support the -n flag of the wait builtin (check page 87 of the reference manual). Therefore, setting a trap that kills all child processes (kill -- $(jobs -p); exit 1) as soon as any of them fails is not possible using wait -n || exit 1. The -n flag has AFAICT been introduced in Bash 5.0.

On macOS, in order to use the Homebrew-installed Bash 5.0, use #!/usr/bin/env bash instead of #!/bin/bash.

2022-07-09

2022-06-26

2022-06-15

2022-05-31

2022-05-09

2022-04-29

2022-04-19

2022-04-15

2022-02-19

keep in mind that you can also use compressed CSV files [...], e.g. a gzipped file such as test.csv.gz will work just fine

– DuckDB CSV loading docs

SELECT * FROM read_csv_auto('my_csv.gz');
CREATE TABLE my_table AS SELECT * FROM read_csv_auto('my_csv.gz');

https://twitter.com/iamchriswalker/status/1494368784961789975

2022-02-17

npm install <alias>@npm:<name>:

Install a package under a custom alias. Allows multiple versions of a same-name package side-by-side, more convenient import names for packages with otherwise long ones, and using git forks replacements or forked npm packages as replacements. Aliasing works only on your project and does not rename packages in transitive dependencies. […]

Examples:

npm install my-react@npm:react
npm install jquery2@npm:jquery@2
npm install jquery3@npm:jquery@3
npm install npa@npm:npm-package-arg

– npm install docs

2022-01-18

2022-01-04

2023-12-17

2023-11-23

In OpenSSH config files (e.g. ~/.ssh/config), one can match a user/host pair (not just a host) by using the Match Host example.org User jane-doe notation.

2023-11-15

2023-10-31

2023-10-30

  • about RFC 2369, which enables the email unsubscribe feature

2023-10-12

2023-10-07

2023-09-19

2023-09-11

2023-08-31

2023-08-27

2023-08-09

2023-08-06

2023-08-02

2023-07-26

2023-07-11

2023-06-20

2023-06-03

2023-05-21

2023-04-30

2023-03-21

2023-03-13

  • that CSS' var() optionally accepts a fallback value

2023-02-10

  • the terrible hack that apparently is the only reasonable way to let yarn upgrade a nested dependency (in a controlled way)

2023-01-28

2023-01-23

  • If you have manually pushed a Docker image into the GitHub Container Registry of an orginsation, and attempt to push to it from a repo's GitHub Actions using github.token afterwards, it won't work. You need to explicitly allow write access (or read access, for that matter) to the repo in the package's settings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment