Skip to content

Instantly share code, notes, and snippets.

View MichalCz's full-sized avatar
😏

Budleigh Salterton MichalCz

😏
View GitHub Profile
#!/bin/bash
mnt=$1
host=$2
user=$3
red=$'\e[31m'
yellow=$'\e[33m'
green=$'\e[32m'
cyan=$'\e[36m'
michal@miura:/ccc/apps/worker-4.2.0⟫ docker run -it travisci/ci-nodejs:packer-1494866191 /bin/bash
root@3bb11d58773a:/# su -l travis
travis@3bb11d58773a:~$ git clone --depth=50 --branch=master https://github.com/signicode/scramjet-core.git signicode/scramjet-core
Cloning into 'signicode/scramjet-core'...
remote: Enumerating objects: 816, done.
remote: Counting objects: 100% (816/816), done.
remote: Compressing objects: 100% (311/311), done.
remote: Total 816 (delta 549), reused 735 (delta 494), pack-reused 0
Receiving objects: 100% (816/816), 245.59 KiB | 0 bytes/s, done.
Resolving deltas: 100% (549/549), done.
const scramjet = require("scramjet");
const {getAPI,insertDB} = require('./io-simulators');
const ROWS = 1000;
class APIReadable extends scramjet.DataStream {
constructor(options) {
super(options);
this.offset = 0;
}
@MichalCz
MichalCz / simple-rest-to-db.js
Created April 6, 2017 15:40
How to crawl a rest API and save response to DB with Scramjet.
const scramjet = require("scramjet");
const stream = require("stream");
const request = require("request-promise");
const config = require(process.env.CONFIG);
let cnt = 0;
// build a stream of indexes
new scramjet.DataStream({
read() {