The Open Science Framework is an online tool built to facilitate the public sharing of scientific effort. One feature we provide to our users is the ability to connect external storage providers (e.g. Dropbox, GoogleDrive, GitHub) to their projects to make it easier to create, move, and share data with others. To abstract this process, we have written an asynchronous storage provider abstraction service called WaterButler. WaterButler allows us to interact with our supported providers through a single consistent interface. For this grant, we want a WaterButler provider that will implement this interface for git repositories hosted on GitLab.com as well as an OSF add-on that will allow users to connect their GitLab account to their OSF account and link individual repositories to their OSF projects. The user should then be able to use the file browser within a project to create new folders, upload files, move files between storage providers, and delete files and folders.
1.) A pull-request against the OSF repository that implements the GitLab add-on. This add-on will allow users to specify a GitLab instance to communicate with, and then connect their GitLab account to their OSF account via an OAuth token. Users may then connect a project for which they have admin rights to one of their GitLab-hosted repositories. Users should also be able to change which repository is linked to a given project at any time and revoke permissions from any or all projects as desired.
2.) A pull-request against the WaterButler repository that implements the GitLab storage provider. This provider must conform to the WaterButler API v1 interface. The provider must issue all requests to the GitLab API asynchronously. An asynchronous library may be used if available, otherwise the aiohttp library should be used for all requests. Unit tests are required, but do not necessarily have to be exhaustive. They should cover the same basic features as the tests for the other providers (basic metadata fetching, file/folder creation, deletion, etc.). The tests should intercept all HTTP requests and mock their response. If the provider uses the aiohttp library, then the aiohttpretty library may be used to intercept and mock test requests. The provider should also be well-documented, with docstrings for all methods.
Both pull-requests will be reviewed by COS developers for code correctness and quality. The grantee should respond to feedback until both pull-requests are accepted.
There will be several check-in steps along the way.
1.) The implementor should open Work-In-Progress pull requests as soon as development begins. The title of the PR should begin with "[WIP]" to indicate that it is not yet ready for review. This PR should be updated regularly as development proceeds.
2.) Once a substantial body of the work has been completed, the implementor should schedule a Product Demo with the Grant Manager.
3.) After accomodating the feedback from Product Demo, the implementor should open the PRs for code review by changing the "[WIP]" in the title to "[CR]" and requesting review in a comment on the PR.
4.) Once the code has passed code review, it will be merged and go through a QA cycle internally within COS. The Code Reviewer or Grant Manager will contact the implementor if further changes are needed.
-
PRs against the OSF should follow the COS development guidelines. See also: COSDev: Developing An Addon
-
PRs against WaterButler should conform to its CONTRIBUTING.md.
-
Current WaterButler (as of 2016-03-16) requires python 3.4, but we will soon be transitioning to a python 3.5 implementation. Development should be targeted against the
feature/python-3.5branch until that is merged intodevelop. -
Implementors should consult the existing Github OSF add-on and WaterButler provider for guidance and inspiration. The OSF Dataverse addon allows users to specify the Dataverse instance they wish to connect to.
-
The WaterButler provider is expected to behave as if it were a filesystem. Deviations from this behavior should be documented in the provider class's docstring, in a section labeled Quirks.
-
git doesn't have a concept of empty folders. To respect filesystem-like behavior, folder creation requests should create an empty
.gitkeepfile inside the folder. -
Since gitlab can be locally hosted the provider should be written to suppoort as many versions of the gitlab API as possible. Please include a section in the provider docstring explicitly stating which major versions of GitLab CE are supported. If a major version cannot be supported or would require significant overhead (in lines of code or additional requests), please document this and let the Grant Manager know as soon as possible.
There is a #cos-integrations Slack channel where assistance may be requested. Help can also be requested over email or via the GitHub pull-requests, once submitted.