Skip to content

Instantly share code, notes, and snippets.

Avatar

phiresky

View GitHub Profile
@phiresky
phiresky / amazon-sort-by-price-per-unit.js
Last active Oct 26, 2020
reorders amazon search results to sort by price per weight to find the actually cheapest item (only tested on amazon.de)
View amazon-sort-by-price-per-unit.js
// reorders amazon search results to sort by price per weight to find the actually cheapest item
// 1. install https://addons.mozilla.org/en-US/firefox/addon/weautopagerize/
// 2. search for something
// 3. scroll down all the way
// 4. run this script
// "featured by amazon"
for (let crap of document.getElementsByClassName(
"template=FEATURED_ASINS_LIST"
))
View sql-libs-for-typescript.md

Using SQL databases in a typed language is a pain unless you have great libraries to support you. There's a lot of different libraries for TypeScript, but they all have flaws.

This is complete overview of SQL libraries for TypeScript. If I'm missing a library, please let me know.

Object Relation Mappers (ORMs)

In an ORM you declare the schema completely in the host language (TypeScript). The ORM then completely manages synchronization between your objects / classes and the corresponding database tables.

ORMs always have the same issues: If you have somewhat complex queries, you will get to the limit of the ORM and not be able to represent that query in it without escape hatching. You also lose direct control over how the queries are handled, and thus may get surprising performance issues when the ORM uses dumb SQL queries in the background.

@phiresky
phiresky / tune.md
Last active Jan 21, 2021
SQLite performance tuning
View tune.md

You can scale a SQLite database to multiple GByte in size and many concurrent readers by applying the below optimizations.

Run these every time you connect to the db

(some are applied permanently, but others are reset on new connection)

pragma journal_mode = WAL;

Instead of writing directly to the db file, write to a write-ahead-log instead and regularily commit the changes. Allows multiple concurrent readers, and can significantly improve performance.

@phiresky
phiresky / .gitignore
Last active Jun 5, 2020
parity auto-kill script
View .gitignore
/node_modules
*.log
@phiresky
phiresky / pdfextract.sh
Created May 29, 2020
ripgrep pdf text extractor with caching that is much faster than pdfgrep
View pdfextract.sh
#!/bin/bash
# usage: `rg --no-line-number --sort-files --pre pdfextract "$@"`
# better and much faster solution: https://github.com/phiresky/ripgrep-all
fname="$1"
cachedir=/tmp/pdfextract
mkdir -p "$cachedir"
View newscatcher-json.html
<meta charset="utf-8">
<script src="https://unpkg.com/sql.js@1.2.2/dist/sql-asm.js"></script>
<script>
async function go() {
const SQL = await initSqlJs();
const dbres = await fetch("https://rawcdn.githack.com/kotartemiy/newscatcher/b30358cf57c9f8f4a481b51c0a0884a64e0b85b2/newscatcher/data/package_rss.db");
@phiresky
phiresky / README.md
Last active Dec 30, 2020
guitar synthesizer in 96 characters of C
View README.md

tiny guitar synth in 96 chars of C

works by starting with a array filled with white noise (from /dev/urandom), then continuously modulating it with a low pass filter of the desired frequency.

this results in a sound pretty similar to a guitar with steel or nylon strings.

@phiresky
phiresky / client.ts
Last active Apr 9, 2020
koa example of a somewhat sane typed http server
View client.ts
import { makeClient } from "./makeTypedApi";
import { Api } from "./common";
const api = makeClient(Api);
// has all the HTTP methods like normal methods, e.g.
const results = await api.byDate()
@phiresky
phiresky / nginx.conf
Last active Dec 2, 2020
Log as much information as possible in nginx to json format
View nginx.conf
http {
log_format json_combined escape=json
'{'
'"time_iso8601":"$time_iso8601", "remote_addr":"$remote_addr", "remote_user":"$remote_user", "request":"$request",'
'"status": "$status", "body_bytes_sent":"$body_bytes_sent", "request_time":"$request_time","http_host":"$http_host","host":"$host",'
'"args":"$args",'
'"connection":"$connection","content_length":"$content_length","content_type":"$content_type","uri":"$uri","request_filename":"$request_filename",'
'"http_referrer":"$http_referer", "http_user_agent":"$http_user_agent",'
'"upstream_connect_time": "$upstream_connect_time", "upstream_response_time":"$upstream_response_time"'
'}';
@phiresky
phiresky / README.md
Last active Jun 6, 2019
zsh histdb example data filler
View README.md

run in a somewhat interesting dir (e.g. checkout of a linux kernel) with <100k files (otherwise too slow).

mv -i ~/.histdb/zsh-history.db ~/.histdb/realhistory

./makedb.sh | sqlite3 ~/.histdb/zsh-history.db