Skip to content

Instantly share code, notes, and snippets.

@rusintez
Created April 17, 2017 11:25
Show Gist options
  • Save rusintez/46cb0209d7bc6b64b16233042c85d822 to your computer and use it in GitHub Desktop.
Save rusintez/46cb0209d7bc6b64b16233042c85d822 to your computer and use it in GitHub Desktop.
ffffound is shutting down
const fs = require('fs');
const hyperquest = require('hyperquest');
const pump = require('pump');
const xray = require('x-ray');
const parse = require('json-parse-stream');
const through = require('through2').obj;
const x = xray().throttle(1, 300); // be considerate
const schema = [{
title: '.title a',
link: '.title a@title',
description: '.description',
url: 'table a img@src'
}];
x('http://ffffound.com', '#assets .asset', schema)
.paginate('#paging-next@href')
.limit(Infinity)
.stream()
.pipe(parse())
.pipe(through(({ type, value }, enc, cb) => {
if (type !== 'object') return cb();
value.filename = value.url.split('/').pop();
// make sure `img` folder exists
const path = `${__dirname}/img/${value.filename}`;
pump(hyperquest(value.url), fs.createWriteStream(path), (err) => {
if (!err) console.log('Downloaded', value.url);
cb(err, JSON.stringify(value) + '\n');
});
}))
.pipe(fs.createWriteStream(`${__dirname}/meta.json`))
.on('finish', process.exit);
@ecrecover
Copy link

@vslbsl Hey man, do you still have the dump file? The torrent is dead after these years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment