Skip to content

Instantly share code, notes, and snippets.

View dkam's full-sized avatar

Dan Milne dkam

View GitHub Profile
@dkam
dkam / longest_wp_tag.rb
Created January 30, 2024 02:10
Find longest Wordpress Tag / Category
Feedjira.parse(URI.open("https://blog.booko.com.au/feed/").read).entries.max_by {|e| e.categories.max_by {|c| c.length }.then {|c| c.length } }
@dkam
dkam / gist:5725c01173a6fa71f7f80c0e08605f96
Last active January 22, 2024 05:54
Convert ISNI_persons.jsonld.gz into a JSONL file using command line tools sed and jq.
# https://isni.org/page/linked-data/
# https://isni.oclc.org:2443/isni/public_export/ISNI_persons.jsonld.gz
wget https://isni.oclc.org:2443/isni/public_export/ISNI_persons.jsonld.gz
# The file I downloaded was full of the 0x1E character, or ^^ in ASCII. This will strip that
sed 's/\x1E//g' ISNI_persons.jsonld > cleaned_ISNI_persons.jsonld
# Then use JQ to convert the file into the way more sane JSONL format. By default, it tries to read it all into
@dkam
dkam / random_isbn13.rb
Created December 28, 2023 04:07
Random ISBN13
def random_isbn13
isbn12 = rand(978_000_000_000..979_999_999_999).to_s
chksum = (10 - isbn12.each_char.with_index.sum { |digit, i| digit.to_i * (i.even? ? 1 : 3) } % 10 ) % 10
"#{isbn12}#{chksum}"
end
def risbn13 = rand(978_000_000_000..979_999_999_999).then {|isbn12| "#{isbn12}#{(10 - isbn12.to_s.each_char.with_index.sum { |digit, i| digit.to_i * (i.even? ? 1 : 3) } % 10 ) % 10}" }
@dkam
dkam / favicon.rb
Last active December 15, 2023 07:51
Get a site's favicon
class Favicon
def initialize(doc, url: nil)
@doc = doc
@url = if url
@url = URI(url.to_s)
elsif canonical
URI(canonical)
else
@dkam
dkam / gist:37d7a41d3c4ab00278580531832130bd
Last active November 27, 2023 02:07
Postgres table, index & toast size.
# This ( https://gist.github.com/hatarist/675dc3debf6cf5f825b5c15aed4cbac0 ) with TOAST size formatting
SELECT
table_name,
pg_size_pretty(table_bytes) AS table,
pg_size_pretty(index_bytes) AS index,
pg_size_pretty(toast_bytes) AS toast,
pg_size_pretty(total_bytes) AS total
FROM (
SELECT
@dkam
dkam / songlink.rb
Created October 31, 2023 02:41
Lookup albums on song.link for sharing
@dkam
dkam / add_query_parameters_to_a_url.rb
Created September 15, 2023 05:15
Add query parameter with URI
URI(url).tap {|u| u.query = URI.encode_www_form(CGI.parse(u.query || "").merge({ref: 123})) }
@dkam
dkam / misc.rb
Created September 5, 2023 06:08
Logging in rails
# Make add tags to the normal rails logger, accessed like `hc_logger.info("my log message")`
def hc_logger = @hc_logger ||= logger.tagged("HttpClients").tagged(self.class.name)
# Create a new logger as an instance variable for a class:
class MyServiceClass
# ....
def my_logger = @my_logger ||= create_logger.tagged(self.class.name)
@dkam
dkam / bot_shedding.rb
Last active August 10, 2023 02:06
A Rails before_action to request bots retry-after X seconds when normalised load is high
def bot_shedding(wait: 600, threshold: 1)
require "sys/cpu"
require "etc"
# normalize the 1-minute load average based on processor count
return unless Sys::CPU.load_avg[0] / Etc.nprocessors > threshold
return unless DeviceDetector.new(request.user_agent).bot?
logger.info("Shedding bots: Returning 503 / Retry-After: #{wait} for #{request.remote_ip} / #{request.user_agent}")
/\b(?:97[89]-?)?(?:\d{1,5}-?)?(?:\d{1,7}-?)?(?:\d{1,6}-?)?\d{1,3}\b/