Round-trip integration between GitHub/GitLab and git-issue

GSoCFinalReport.md

Project: git-issue

Organization: Open Technologies Aliance — GFOSS

Mentors: Diomidis Spinellis(main mentor), Kostas Papadimas, Alexios Zavras(backup contracts)

(This document was adapted from my final GSoC Report hosted on Medium, which you can find here)

Introduction

My project for Google Summer of Code 2019 was to extend git-issue, a minimalist Git-based decentralized issue management system, with issue import/export functionality towards GitHub and GitLab. The intent is to be able to import an issue, edit/comment on it, and export it without ever leaving git-issue’s command line interface.

What was done

My work for the duration of the program was done on a dedicated GitHub repository, cloned from the original. The first commit I made in the context of GSoC was d7d499e. My work spans from that commit, until a47e475.

My first priority when the work period officially began, was to lay the groundwork for my implementation by getting the attributes supported by git-issue more in tune with the ones supported by the aforementioned platforms. At this point, git-issue had no support for attributes such as due date, issue weight or time estimates, which GitLab used.

An invaluable tool during the development was automated regression testing, which was emphasized by my mentor from day one. Online multi-platform testing was provided by Travis CI. At this point, testing revealed a problem: Some of the features of date my implementation relied on(in order to parse dates), were only present in GNU date, not installed by default on OS X. Since those features were necessary , a decision was made to add GNU date as a dependency.

Once this was done and tested, I moved on to the main part: importing/exporting issues. I started with GitHub export (GitHub importing was already implemented in git-issue), and then continued with GitLab import/export. One challenge I faced were the subtle ways in which the handling of issues between GitLab and GitHub differ. A compromise often had to be made: For example, GitLab may ignore newlines in issue descriptions unless they are preceded by two or more spaces, which meant that in order to preserve formatting, git-issue needed to append those spaces.

jq and curl were used for JSON parsing and interacting with GitHub’s/GitLab’s REST APIs respectively. Fortunately, both utilities were already used by git-issue, so no new dependencies had to be added.

Another noteworthy challenge was handling importing and reexporting whole repositories, possibly across platforms. Author usernames might not be preserved, and references to other issues might no longer apply. After some discussion with my mentor, the decision was made that the least intrusive way to handle this would be with the introduction of a new subcommand : filter-apply. Along with user-defined or even built-in scripts, it can be used to fix those inconsistencies, as well as edit the database for any other purpose.

Using the new features

All in all, 5 new subcommands were added (excluding the ones used for the new attributes): create, export, exportall, filter-apply, dump.

To create a single issue on a remote repo based on a local one, you can use create:

$ git issue create 1f2 gitlab vyrondrosos demo

To export all already existing issues to a remote repository,one could use:

$ git issue export <provider> <user> <repo>
Issue b83d92872dc16440402516a5f4ce1b8cc6436344 hasn't been modified, skipping...
Comment a93764f32179e93493ceb0a7060efce1e980aff1 hasn't been modified, skipping...
Exporting issue 9179d381135273220301f175c03b101b3e9c703d as #15

To export all open issues, exportall can be used:

$ git issue exportall <provider> <user> <repo>
Creating issue 9179d38...
Couldn't add assignee user1. Skipping...
Creating issue 3651dd3...
Creating new Milestone ver3...
Creating comment d72c68d0177b500a91ea37548e6594f84457fd5b...

To dump the whole database to JSON(useful for debugging purposes), use dump:

$ git issue dump
{
  "issues": [
    {
      "sha": "4fb7a9f",
      "tags": [
        "closed"
      ],
...
...

Lastly, filter-apply can be used with any command, in order to make modifications en masse. For example :

$ git issue filter-apply editdesc.sh
Filtering issue 4fb7a9ffcfc9f79b87b497e137f097c643ec3332...
Filtering issue cdaf1208bdf9c6dd839e15ea05e2d007903f4a09...
Filtering issue f3dc62b500455fccc8174fd400aabc9e561a7d19...Filter applied. To commit the results, run "gi git commit -a".

For more information, please consult git-issue’s README.md.

Moving forward

Although the main goals of my proposal were accomplished, there is still some room for improvement. Notably, as I mentioned above there are some challenges when exporting issues that you are not the original author of, or that contain internal references to other issues. In dealing with such issues the foundation has been laid with filter-apply, but the relevant scripts to be used with it still need to be developed.

Minor performance improvements could also be made in the API code by batching together some requests, and by caching some intermediate results.

Conclusion

My participation in GSoC was above all, a learning experience for me(even more than I expected). During those three months, I used — for the first time in a real-world context — and got acquainted with the following skills:

• Automated regression testing
• Writing maintainable and portable UNIX shell code
• Writing documentation
• Working with RESTful APIs
• Parsing and generating JSON
• Advanced management of a codebase with Git(branching, merging etc)

And of course other non-technical skills, such as:

• creating and adhering to a timetable, while maintaining some flexibility and making adjustments on the fly
• Working with other people, effectively communicating my progress or asking for help

Participating in GSoC has given me the confidence, the resolve and skills to get more involved in the Open Source Community moving forward. I’d also like to continue contributing to git-issue, by both iterating upon the features I implemented and unrelated ones.