Skip to content

Instantly share code, notes, and snippets.

@hubgit
Last active August 27, 2022 20:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hubgit/76fab549127e4a94c10632d1242983c6 to your computer and use it in GitHub Desktop.
Save hubgit/76fab549127e4a94c10632d1242983c6 to your computer and use it in GitHub Desktop.
Write to a file in a Google Cloud Storage bucket by piping a Web Stream to stdin of a `gcloud alpha storage cp` process.
export const cloudStorageJsonLinesWriter = (url: string) => {
// gcloud components install alpha
const process = Deno.run({
cmd: [
'gcloud',
'alpha',
'storage',
'cp',
'-',
url,
'--cache-control=no-transform',
'--content-encoding=gzip',
'--content-type=application/x-ndjson',
'--quiet',
],
stdin: 'piped',
})
const stream = new TransformStream()
stream.readable
.pipeThrough(
new TransformStream({
transform(chunk, controller) {
controller.enqueue(JSON.stringify(chunk))
controller.enqueue('\n')
},
})
)
.pipeThrough(new TextEncoderStream())
.pipeThrough(new CompressionStream('gzip'))
.pipeTo(process.stdin!.writable)
.finally(() => process.status())
return stream.writable.getWriter()
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment