Skip to content

Instantly share code, notes, and snippets.

View line-reader-transform-stream.js
lineReader = () => {
let buffer = "";
return new TransformStream({
transform(chunk, controller) {
buffer += chunk;
const parts = buffer.split("\n");
parts.slice(0, -1).forEach((part) => controller.enqueue(part));
buffer = parts[parts.length - 1];
},
@hubgit
hubgit / genbank-to-sqlite.ts
Last active Sep 5, 2022
A ReadableStream created from an async iterator which fetches paginated data, piped into a WritableStream which inserts items into an SQLite database.
View genbank-to-sqlite.ts
import { parse } from 'https://deno.land/x/xml@2.0.4/mod.ts'
import { readableStreamFromIterable } from 'https://deno.land/std@0.96.0/io/streams.ts'
import { Database } from 'https://deno.land/x/sqlite3@0.5.2/mod.ts'
import ProgressBar from 'https://deno.land/x/progress@v1.2.7/mod.ts'
let counter = 0
const progress = new ProgressBar({
title: 'processing:',
interval: 100,
@hubgit
hubgit / README.md
Last active Sep 2, 2022
Processing the Crossref Public Data File
View README.md

First, download the data files using a BitTorrent client:

aria2c https://academictorrents.com/download/4dcfdf804775f2d92b7a030305fa0350ebef6f3e.torrent

Next, convert the data files to a single newline-delimited JSON file:

deno run process.ts
@hubgit
hubgit / deno-cloud-storage-web-streams.ts
Last active Aug 27, 2022
Write to a file in a Google Cloud Storage bucket by piping a Web Stream to stdin of a `gcloud alpha storage cp` process.
View deno-cloud-storage-web-streams.ts
export const cloudStorageJsonLinesWriter = (url: string) => {
// gcloud components install alpha
const process = Deno.run({
cmd: [
'gcloud',
'alpha',
'storage',
'cp',
'-',
url,
@hubgit
hubgit / deno-web-streams.ts
Last active Sep 4, 2022
Reader and Writer web streams for Deno
View deno-web-streams.ts
import { TextLineStream } from 'https://deno.land/std@0.153.0/streams/mod.ts'
// const input = await jsonLinesReader('input.jsonl.gz')
// const output = await jsonLinesWriter('output.jsonl.gz')
// for await (const item of input) {
//// do something
// await output.write(item)
// }
@hubgit
hubgit / deno-compressed-json-lines-reader.ts
Last active Aug 26, 2022
Read/write a gzipped JSON lines file with Deno
View deno-compressed-json-lines-reader.ts
const createInputReader = async (path: string) => {
const file = await Deno.open(path, {
read: true,
})
return file.readable
.pipeThrough(new DecompressionStream('gzip'))
.pipeThrough(new TextDecoderStream())
.pipeThrough(new TextLineStream())
.pipeThrough(
@hubgit
hubgit / docker-image-metadata.mjs
Last active Jul 7, 2022
Fetch Docker image metadata
View docker-image-metadata.mjs
// adapted from https://github.com/distribution/distribution/issues/1252#issuecomment-274944254
const IMAGE = 'example/foo'
const TAG = 'latest'
const authorise = async () => {
const url = new URL('https://auth.docker.io/token')
url.searchParams.set('scope', `repository:${IMAGE}:pull`)
url.searchParams.set('service', 'registry.docker.io')
const response = await fetch(url)
View get-iplayer-clip.sh
get_iplayer --file-prefix="<pid>" --output=$PWD --pid $PID --tv-quality sd --start $START --stop $STOP
@hubgit
hubgit / _headlines.txt
Created Feb 13, 2022
8 years of headlines from the Daily Express' "World War 3" page
View _headlines.txt
Paxman the Peacemaker as he grills feuding Japan and China ambassadors in SEPARATE rooms
Russian President Vladimir Putin wants to start World War Three, says Ukraine PM
Putin's nuclear threat: Russian leader nearly started World War Three over the Crimea war
Putin has started a 'war with the whole civilised world' which could leave MILLIONS dead
Russia's chilling THREAT to the West: Putin's army chief warns of 'how it will all END'
World War Three in SPACE? Fears over rise in anti-satellite weapons created by Russia
Migrant crisis and Euro tensions threaten to trigger catastrophic conflict claim experts
World War Three could begin TOMORROW - and UK's armed forces 'are woefully unprepared'
WW3 THREAT: Ex-Navy chief calls presence of Russia's SAMs in Syria 'extremely dangerous'
Turkey threatens to SHOOT DOWN Putin's planes as it drags West closer to war with Russia
@hubgit
hubgit / wordle-tweets.js
Last active Jan 20, 2022
Fetch and parse Wordle lines from recent tweets
View wordle-tweets.js
import fs from 'fs-extra'
import Twitter from 'twitter-v2'
const n = '213'
const query = `"wordle ${n}" -RT`
const auth = fs.readJSONSync('auth-node.json')
const client = new Twitter(auth)