Skip to content

Instantly share code, notes, and snippets.

@mbostock mbostock/.block
Last active Oct 11, 2019

Embed
What would you like to do?
The Gist to Clone All Gists
license: gpl-3.0

Run like so:

chmod u+x gist-clone-all
./gist-clone-all username OAUTH_TOKEN

You'll want to replace “username” with your own username and “OAUTH_TOKEN” with an OAuth token.

This script clones using the push URL, so you should probably be the owner of the gists. You could also use this to clone someone else's gists, but in that case you may wish to edit the code to use gist_pull_url instead.

#!/usr/bin/env node
var fs = require("fs"),
https = require("https"),
exec = require("child_process").exec;
// TODO --pull or --push
var user = process.argv[2],
token = process.argv[3];
fetchAndClone(1, function callback(error, nextPage) {
if (error) throw error;
if (nextPage > 0) fetchAndClone(nextPage, callback);
});
function fetchAndClone(page, callback) {
fetch(page, function(error, gists) {
if (error) return callback(error);
if (gists.length) ++page; else page = -1;
cloneNext(gists.pop());
function cloneNext(gist) {
if (!gist) return callback(null, page);
if (directoryExists(gist.id)) return cloneNext(gists.pop());
console.log("cloning " + gist.id);
exec("git clone git@gist.github.com:" + gist.id + ".git", function(error, stdout, stderr) {
if (error) return callback(error);
cloneNext(gists.pop());
});
}
});
}
function fetch(page, callback) {
var request = https.request({
hostname: "api.github.com",
port: 443,
path: "/users/" + encodeURIComponent(user) + "/gists?page=" + page,
method: "GET",
headers: {
"Accept": "application/vnd.github.v3+json",
"Authorization": "token " + token,
"User-Agent": "mbostock/gist-clone-all"
}
}, function(response) {
var chunks = [];
response.setEncoding("utf8");
response.on("data", function(chunk) { chunks.push(chunk); });
response.on("end", function() { callback(null, JSON.parse(chunks.join(""))); });
});
request.on("error", callback);
request.end();
}
function directoryExists(path) {
try {
return fs.lstatSync(path).isDirectory();
} catch (ignored) {
return false;
}
}
#!/bin/bash
for i in *; do
if [ -d $i ]; then
pushd $i
git commit -am '' --allow-empty-message && git push
popd
fi
done
@batandwa

This comment has been minimized.

Copy link

commented Apr 18, 2013

Superb gist. Thank you.

@weakish

This comment has been minimized.

Copy link

commented Oct 21, 2014

https://gist.github.com/mbostock/3883098#file-gist-clone-all-js-L1

#!/usr/local/bin/node

Why not #!/usr/bin/env node?


@batandwa This only clones public gists. If you wnt to clone all your gists (including private ones), you can have a look at gister sync.

@nanda-malve

This comment has been minimized.

Copy link

commented Feb 24, 2016

I am unable to use the above script, please help me in this

root@dev-node:~/git# node gist-clone-all

/root/git/gist-clone-all:21
cloneNext(gists.pop());
^
TypeError: Object # has no method 'pop'
at /root/git/gist-clone-all:21:21
at IncomingMessage. (/root/git/gist-clone-all:50:37)
at IncomingMessage.emit (events.js:88:20)
at HTTPParser.onMessageComplete (http.js:137:23)
at CleartextStream.ondata (http.js:1150:24)
at CleartextStream._push (tls.js:375:27)
at SecurePair.cycle (tls.js:734:20)
at EncryptedStream.write (tls.js:130:13)
at Socket.ondata (stream.js:38:26)
at Socket.emit (events.js:67:17)

@maowug

This comment has been minimized.

Copy link

commented Mar 26, 2016

@nanda-malve

I'v made one much simpler to use. No OAUTH_TOKEN, for both github.com and github enterprise.
https://github.com/maowug/backup-gists

@tianchaijz

This comment has been minimized.

Copy link

commented May 10, 2016

@vetras

This comment has been minimized.

Copy link

commented May 18, 2016

The error:

root@dev-node:~/git# node gist-clone-all

/root/git/gist-clone-all:21
cloneNext(gists.pop());
^
TypeError: Object # has no method 'pop'
(...)

Is due to a invalid token.
If you add a log to the GIST object:

(...)
console.log(gists);
cloneNext(gists.pop());
(...)

It comes with the GitHub message:

{ message: 'Bad credentials',
documentation_url: 'https://developer.github.com/v3'
}

My solution was to generate another token with more permissions.

Great stuff, many thanks!

@hyang0

This comment has been minimized.

Copy link

commented Sep 2, 2016

I tried. It works. The environment is ubuntu 16.04 64bit.

First of all, you should install nodejs.

➜ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -

Installing the NodeSource Node.js v4.x LTS Argon repo...

➜ sudo apt-get install nodejs
Reading package lists... Done

Then you should generate a ssh public key and add it to you github account.

➜ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hyang0/.ssh/id_rsa):

Finnally, retry the script to sync your gists.

➜ ./gist-clone-all.sh GIST-ACCOUNT Personal-access-token
cloning 7d5e4348f307f7cb2fbe
cloning 6986436
cloning 8572463
cloning 8572515
cloning 8572569
cloning a59175560823bd3cf383
cloning 5234efc9170aa3302dd9
cloning 75c39d5a6186e5bae304
cloning 159e719feb8b230a63fa
cloning 1eeb1affd91e91ba73c7
cloning 2498e1494bf7af5503b9

Cool job. Thanks!

@lmj0011

This comment has been minimized.

Copy link

commented Dec 30, 2016

I only had to set the gist permission for the OAuth token, but you may not even need that set.

@mbostock
Would be nice to have, which permission should be set for the OAuth token, in the README.md .

@lmj0011

This comment has been minimized.

Copy link

commented Dec 30, 2016

I only had to set the gist permission for the OAuth token, but you may not even need that set.

correction: Only the "gist" scope is needed for the OAuth token.

@tianyiii

This comment has been minimized.

Copy link

commented Jan 30, 2017

what is gist_pull_url?

@nlevick

This comment has been minimized.

Copy link

commented Mar 30, 2017

@maowug
Great work! Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.