Issues by month -- Open Source Deep Learning Platforms

download_issue_dates.sh

project="paddlepaddle/paddle"
if [[ $# -ne 0 ]]; then
   project=$1
fi

file=${project/\//-}
> $file.issues

issues_per_page=25
page=1
until [ $issues_per_page -lt 25 ]; do
    echo Parsing page: $page >&2

    url=$(printf 'https://github.com/%s/issues?page=%d' $project $page)
    url=$url+'&q=is%3Aissue'

    echo $url >&2

    issues_per_page=$(curl -s "$url" | grep 'relative-time' | sed "s/^.*datetime=\"\([^T]*\).*$/\1/" | tee -a $file.issues | wc -l)
    if [[ $issues_per_page -eq 0 ]]; then
        sleep 61 # work around the Github abuse detection mechanism.
        issues_per_page=$(curl -s "$url" | grep 'relative-time' | sed "s/^.*datetime=\"\([^T]*\).*$/\1/" | tee -a $file.issues | wc -l)
    fi
    
    echo issues per page: $issues_per_page >&2
    let page=page+1

    sleep 2 # work around the Github abuse detection mechanism.
done

cat $file.issues | sort | cut -f1,2 -d '-' | uniq -c | gawk \
'BEGIN {
    year=2015;
    month=6;
    current=sprintf("%04d-%02d", year, month);
    cum=0;
}
{
    while (current < $2) {
	printf("%s-30,%d\n", current, cum);
	month++;
	if (month > 12) {
	    month=1;
	    year++;
	}
	current=sprintf("%04d-%02d", year, month);
    }
    if (current > $2) {
	cum+=$1;
    }
    if (current == $2) {
	cum+=$1;
	printf("%s-30,%d\n", current, cum);
	month++;
	if (month > 12) {
	    month=1;
	    year++;
	}
        current=sprintf("%04d-%02d", year, month);
    }
}
'

Above script can download all issues of an open source repo and count the cumulative number of issues by month. It outputs a .csv file that can be imported to Google Docs and be visualized there.

Here is an example that compares the number of issues of paddlepaddle/paddle and dmlc/mxnet. The figure was drawn using Google Docs is as follows:

To run the script with a Github repo, say dmlc/mxnet:

./github-issues-tracker.sh dmlc/mxnet > dmlc-mxnet.csv