The Gist to Clone All Gists

.block

license: gpl-3.0

README.md

Run like so:

chmod u+x gist-clone-all
./gist-clone-all username OAUTH_TOKEN

You'll want to replace “username” with your own username and “OAUTH_TOKEN” with an OAuth token.

This script clones using the push URL, so you should probably be the owner of the gists. You could also use this to clone someone else's gists, but in that case you may wish to edit the code to use gist_pull_url instead.

gist-clone-all

#!/usr/bin/env node

var fs = require("fs"),
    https = require("https"),
    exec = require("child_process").exec;

// TODO --pull or --push

var user = process.argv[2],
    token = process.argv[3];

fetchAndClone(1, function callback(error, nextPage) {
  if (error) throw error;
  if (nextPage > 0) fetchAndClone(nextPage, callback);
});

function fetchAndClone(page, callback) {
  fetch(page, function(error, gists) {
    if (error) return callback(error);
    if (gists.length) ++page; else page = -1;
    cloneNext(gists.pop());

    function cloneNext(gist) {
      if (!gist) return callback(null, page);
      if (directoryExists(gist.id)) return cloneNext(gists.pop());
      console.log("cloning " + gist.id);
      exec("git clone git@gist.github.com:" + gist.id + ".git", function(error, stdout, stderr) {
        if (error) return callback(error);
        cloneNext(gists.pop());
      });
    }
  });
}

function fetch(page, callback) {
  var request = https.request({
    hostname: "api.github.com",
    port: 443,
    path: "/users/" + encodeURIComponent(user) + "/gists?page=" + page,
    method: "GET",
    headers: {
      "Accept": "application/vnd.github.v3+json",
      "Authorization": "token " + token,
      "User-Agent": "mbostock/gist-clone-all"
    }
  }, function(response) {
    var chunks = [];
    response.setEncoding("utf8");
    response.on("data", function(chunk) { chunks.push(chunk); });
    response.on("end", function() { callback(null, JSON.parse(chunks.join(""))); });
  });
  request.on("error", callback);
  request.end();
}

function directoryExists(path) {
  try {
    return fs.lstatSync(path).isDirectory();
  } catch (ignored) {
    return false;
  }
}

gist-push-all.sh

#!/bin/bash

for i in *; do
  if [ -d $i ]; then
    pushd $i
    git commit -am '' --allow-empty-message && git push
    popd
  fi
done

Superb gist. Thank you.

https://gist.github.com/mbostock/3883098#file-gist-clone-all-js-L1

#!/usr/local/bin/node

Why not #!/usr/bin/env node?

---

@batandwa This only clones public gists. If you wnt to clone all your gists (including private ones), you can have a look at gister sync.

I am unable to use the above script, please help me in this

root@dev-node:~/git# node gist-clone-all

/root/git/gist-clone-all:21
cloneNext(gists.pop());
^
TypeError: Object # has no method 'pop'
at /root/git/gist-clone-all:21:21
at IncomingMessage. (/root/git/gist-clone-all:50:37)
at IncomingMessage.emit (events.js:88:20)
at HTTPParser.onMessageComplete (http.js:137:23)
at CleartextStream.ondata (http.js:1150:24)
at CleartextStream._push (tls.js:375:27)
at SecurePair.cycle (tls.js:734:20)
at EncryptedStream.write (tls.js:130:13)
at Socket.ondata (stream.js:38:26)
at Socket.emit (events.js:67:17)

@nanda-malve

I'v made one much simpler to use. No OAUTH_TOKEN, for both github.com and github enterprise.
https://github.com/maowug/backup-gists

A Python version:
https://gist.github.com/tianchaijz/cb30e71d6f1fb85e954bca8454942c87

The error:

root@dev-node:~/git# node gist-clone-all

/root/git/gist-clone-all:21
cloneNext(gists.pop());
^
TypeError: Object # has no method 'pop'
(...)

Is due to a invalid token.
If you add a log to the GIST object:

(...)
console.log(gists);
cloneNext(gists.pop());
(...)

It comes with the GitHub message:

{ message: 'Bad credentials',
documentation_url: 'https://developer.github.com/v3'
}

My solution was to generate another token with more permissions.

Great stuff, many thanks!

I tried. It works. The environment is ubuntu 16.04 64bit.

First of all, you should install nodejs.

➜ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -

Installing the NodeSource Node.js v4.x LTS Argon repo...

➜ sudo apt-get install nodejs
Reading package lists... Done

Then you should generate a ssh public key and add it to you github account.

➜ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hyang0/.ssh/id_rsa):

Finnally, retry the script to sync your gists.

➜ ./gist-clone-all.sh GIST-ACCOUNT Personal-access-token
cloning 7d5e4348f307f7cb2fbe
cloning 6986436
cloning 8572463
cloning 8572515
cloning 8572569
cloning a59175560823bd3cf383
cloning 5234efc9170aa3302dd9
cloning 75c39d5a6186e5bae304
cloning 159e719feb8b230a63fa
cloning 1eeb1affd91e91ba73c7
cloning 2498e1494bf7af5503b9

Cool job. Thanks!

I only had to set the gist permission for the OAuth token, but you may not even need that set.

@mbostock
Would be nice to have, which permission should be set for the OAuth token, in the README.md .

I only had to set the gist permission for the OAuth token, but you may not even need that set.

correction: Only the "gist" scope is needed for the OAuth token.

what is gist_pull_url?

@maowug
Great work! Thank you!

I've been using https://github.com/aprescott/gist-backup but I was wondering if there was a way to clone all other people's forks of my gists, too, so I can easily see what changes have been made. I've been adding them manually with username as remote name.

I have failed many times with this. but after searching the internet I found an easy one to do the same job, ghcloneall 1.10.1, enjoy!