janosgyerik/draft-watch-unwatch-2.md

## draft-watch-unwatch-2.md

      
    Raw
  

              draft-watch-unwatch-2.md
            
          
    Watch or unwatch GitHub repositories using GitHub API + curl + jq + Bash
When you need to perform an operation on a large number of GitHub repositories, their REST API and some command line magic comes in handy. This short talk will demonstrate a technique to watch and unwatch GitHub repositories using ad-hoc scripts. The example should give a good idea how to perform other bulk operations too.
Background...
I have this setting in my personal GitHub account:

(See https://github.com/settings/notifications)
This works fine most of the time:
when I gain push access to a repository,
I usually want to watch it.
... except, if I'm added to a GitHub organization with many dozens of repositories, then no, I don't want to watch all of them...
The user interface to unwatch many repositories is a bit limited:

(See https://github.com/watching)
You can either unwatch everything, or you can click click click.
If you want to unwatch several dozen repositories, the click click click method is not so much fun. Not for me anyway.
Ok so let's see what the famous GitHub API can do for us...
https://developer.github.com/
Let's search for "watching", for example:

Hm, sweet, on "w" already a promising match.
Indeed, this exactly the documentation we're looking for.
Apparently "watching" has been renamed to "subscriptions".
Let's try to use this API to verify our subscription to a repository,
any one on our watched list, let's say sonar-java:
$ curl -s https://api.github.com/repos/janosgyerik/sonar-java/subscription
{"message":"Not Found","documentation_url":"https://developer.github.com/v3"}

Ok that doesn't seem to work.
The documentation does shed some light, if we read carefully:

Response if you are subscribed to the repository

The key word there is "you".
We must authenticate.
Let's search in the docs for "auth",
and let's go for Other Authentication Methods,
which explains a simple solution using tokens.
See https://developer.github.com/v3/auth/
With a token, authenticated requests can be as simple as:
curl -u username:token https://api.github.com/user

We can generate tokens on the settings page,
enter whatever Token description,
and select notifications,
that's enough for our purposes:

(See https://github.com/settings/tokens/new)
After the token is generated,
the next screen shows its value.
Be sure to save it,
as this value won't be visible again anywhere.

Let's save the authentication info in a variable for easy reuse:
auth=janosgyerik:614d36ea4fab840b4b33ee8af68fe45e3eaa57f5

Ok let's try the earlier request again,
but this time authenticating using the token:
$ curl -su $auth https://api.github.com/repos/janosgyerik/sonar-java/subscription
{"subscribed":true,"ignored":false,"reason":null,"created_at":"2016-08-05T11:40:30Z","url":"https://api.github.com/repos/janosgyerik/sonar-java/subscription","repository_url":"https://api.github.com/repos/janosgyerik/sonar-java"}

Much better!
Except that, it's not very easy to read in this format...
jq to the rescue!
jq is a command line tool to filter and transform JSON strings.
In the most basic use of doing nothing just passing data through it,
it pretty-prints, like this:
$ curl -su $auth https://api.github.com/repos/janosgyerik/sonar-java/subscription | jq .
{
  "subscribed": true,
  "ignored": false,
  "reason": null,
  "created_at": "2016-08-05T11:40:30Z",
  "url": "https://api.github.com/repos/janosgyerik/sonar-java/subscription",
  "repository_url": "https://api.github.com/repos/janosgyerik/sonar-java"
}

Ok so let's test that we can unsubscribe to this repository.
According to the docs,
we just need to do a PUT request,
with subscribed and/or ignored parameter.
It's not really clear if we need one or both.
In fact, at the time of this writing,
only setting ignored works,
and it flips the value of subscribed:
$ curl -su $auth https://api.github.com/repos/janosgyerik/sonar-java/subscription \
-X PUT -d '{"ignored": true}' | jq .
{
  "subscribed": false,
  "ignored": true,
  "reason": null,
  "created_at": "2016-08-05T11:40:30Z",
  "url": "https://api.github.com/repos/janosgyerik/sonar-java/subscription",
  "repository_url": "https://api.github.com/repos/janosgyerik/sonar-java"
}

The response is the updated subscription info,
and it looks as we wanted.
Let's verify on the list of watched pages:
indeed sonar-java is now gone from the list.
A better way:
$ curl -su $auth https://api.github.com/repos/janosgyerik/sonar-java/subscription -X DELETE

Alright, so let's do this in bulk, to many repositories at once.
There's no specific API for such operation.
What we can do is get a list of repository names we want to unwatch,
and repeat this request for each.
So let's first get the list of repositories we are currently watching.
Following the docs:
https://developer.github.com/v3/activity/watching/#list-repositories-being-watched
$ curl -su $auth https://api.github.com/user/subscriptions | jq . | less

Ok that's a lot of JSON data.
What we need from this is actually just the full_name field of each repository.
To do that, we need a bit more jq magic.
The . we've been using so far means "the current object".
In this reponse,
the object is an array.
What we need is,
for each object in the array,
we want to get to the full_name field.
The syntax goes like this:
$ curl -su $auth https://api.github.com/user/subscriptions | jq '.[] | .full_name'
"janosgyerik/bashoneliners"
"janosgyerik/updatesite-mobile"
"janosgyerik/autossh-tunnel"
"janosgyerik/shellscripts"
"janosgyerik/gtd-cli-py"
"janosgyerik/stacktools"
...

That's better.
Just one little thing,
it will be easier for scripting if the double-quotes are not there.
The --raw-output or simply -r flag can do that:
$ curl -su $auth https://api.github.com/user/subscriptions | jq '.[] | .full_name' -r
janosgyerik/bashoneliners
janosgyerik/updatesite-mobile
janosgyerik/autossh-tunnel
janosgyerik/shellscripts
janosgyerik/gtd-cli-py
janosgyerik/stacktools
...

Now we can loop over this output to unwatch each repository one by one:
$ curl -su $auth https://api.github.com/user/subscriptions | \
    jq '.[] | .full_name' -r | \
    while read repo; do \
        echo curl -su $auth https://api.github.com/repos/$repo/subscription \
            -X DELETE; done

Note that most GitHub API responses are paginated,
returning at most 30 results:
$ curl -s https://api.github.com/users/janosgyerik/repos | jq -r '.[] | .full_name' | wc -l
     30

To get more results, you can try to use the per_page query parameter,
for example:
$ curl -s https://api.github.com/users/janosgyerik/repos?per_page=200 | jq -r '.[] | .full_name'  | wc -l
     100

This helps getting more results, but not all results.
The secondary limit of 100 overrides the specified per_page=200.
To get more results,
you need to traverse pages using the page query parameter,
for example:
$ curl -s https://api.github.com/users/janosgyerik/repos?per_page=100'&page=2' | jq -r '.[] | .full_name' | wc -l
       1