Skip to content

Instantly share code, notes, and snippets.

@voodooattack
Forked from irony/encode-cluster.js
Last active April 27, 2016 21:35
Show Gist options
  • Save voodooattack/45711ace6ffc8a57ec93b346601b33a2 to your computer and use it in GitHub Desktop.
Save voodooattack/45711ace6ffc8a57ec93b346601b33a2 to your computer and use it in GitHub Desktop.
var startTime = Date.now()
var fs = require('fs')
var Iconv = require('iconv').Iconv
var cluster = require('cluster')
var workers;
if (cluster.isMaster) workers = [1, 2, 3, 4].map(_ => cluster.fork())
if (!cluster.isMaster)
{
var converter = new Iconv('UTF-8', 'UTF-16LE')
var input = fs.createReadStream('./enwik8')
var out = fs.createWriteStream('enwik16_' + Date.now())
input.pipe(converter).pipe(out)
out.on('close', _ => process.exit())
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment