Skip to content

Instantly share code, notes, and snippets.

@jackabox
Last active March 1, 2024 05:35
Show Gist options
  • Save jackabox/60a5343eba05f3cfcc3d3886e6c85acf to your computer and use it in GitHub Desktop.
Save jackabox/60a5343eba05f3cfcc3d3886e6c85acf to your computer and use it in GitHub Desktop.
Example of Node / JSONStream / Event-stream parser
import hyperquest from 'hyperquest'
import JSONStream from 'JSONStream'
import es from 'event-stream'
import { format } from 'date-fns'
const scraper = async () => {
const scryfallJSONUrl =
'https://archive.scryfall.com/json/scryfall-default-cards.json'
await hyperquest(scryfallJSONUrl)
.pipe(JSONStream.parse('*')) // Parse each row
.pipe(
es.map(async (data, callback) => {
const date = await format(Date.now(), 'yyyy-MM-dd HH:mm:ss')
const mappedData = {
scryfall_id: data.id,
name: data.name,
mana_cost: data.mana_cost,
cmc: data.cmc,
type_line: data.type_line,
released_at: data.released_at,
image_uris: data.image_uris,
colors: data.colors,
legalities: data.legalities,
set: data.set,
set_name: data.set_name,
reserved: data.reserved,
last_modified: date
}
console.log(mappedData)
callback(null, data)
})
)
.on('finish', () => console.info('stream done'))
}
scraper()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment