Lessons learned on Zenodo development

README.md

Zenodo local dev environment

Usage

NB: The environment is manged by virtualenvwrapper.

1. start required services

cd zenodo
docker-compose up db es cache mq

2. start celery worker (in new terminal)

workon zenodo
celery worker -A zenodo.celery -l INFO --purge

3. start Zenodo development server (debug mode)

workon zenodo
nvm use 7.4
FLASK_DEBUG=1 zenodo run

Now open Zenodo at the URL shown in the console.

Changes in source code files are detected and service is restarted automatically.

More useful docs for running additional features (e.g. ORCID or GitHub login locally) see https://zenodo.readthedocs.io/en/latest/usage.html.

Notes

Installation

• only use pip2 (= Python 2, /usr/bin/python)

Trying to manipulate configuration

• FLASK_DEBUG=True APP_THEME_TWITTERHANDLE=@o2r_project zenodo run does not have an effect in the UI, and the config variable using the right hand side debug menu on the page does have the value.
• using instance configuration file as described in https://zenodo.readthedocs.io/en/latest/api/config.html

cdvirtualenv
touch var/instance/invenio.cfg
nano var/instance/invenio.cfg

and add

SUPPORT_EMAIL = "daniel.nuest@uni-muenster.de"
THEME_TWITTERHANDLE = "@o2r_project"
THEME_TAG = "GeoSoft II"

at least results in the the configuration available in the right hand side config debug menu "Config". 🎉

• changes in theme .html files at zenodo/modules/theme/templates/zenodo_theme/ are available after a page refresh
• changes ininvenio.cfg require a service restart and Shift-refresh
• THEME_TWITTERHANDLE does not change the Twitter stream loaded on the start page, see /home/daniel/git/zenodo-dev/zenodo/zenodo/modules/frontpage/templates/zenodo_frontpage/index.html
• I can create an Email-login (daniel.nuest@wwu.de, daniel) and create a deposit, but I cannot upload a file... just says "Error" in "Progress" column, the PUT request has a 404 HTTP status (NOT FOUND) for the URL http://localhost:5000/api/files/1850cd43-281d-4f01-b853-992844224e81/GeoSoft_2__WS1819_v2.pdf
• works after a restart, and at FIXTURES_FILES_LOCATION '/home/daniel/.virtualenvs/zenodo/var/instance/data' there are files!

Debugging in vscode

Material:

• https://code.visualstudio.com/docs/python/tutorial-flask
  • How does this work with virtualenvwrapper? Probably not...
  • https://code.visualstudio.com/docs/python/environments#_where-the-extension-looks-for-environments and https://code.visualstudio.com/docs/python/settings-reference#_general-settings
  • configure in workspace settings:

    {
        "folders": [
            {
                "path": "zenodo"
            },
            {
                "path": "zenodo-migrator"
            }
        ],
        "settings": {
            "cSpell.words": [
                "jsonify"
            ],
            "python.venvFolders": [
                "~/.virtualenvs",
                "envs",
                ".pyenv",
                ".direnv"
            ],
            "python.pythonPath": "~/.virtualenvs/zenodo/bin/python"
        }
    }
    

  • add launch configuration:

    {
        "version": "0.2.0",
        "configurations": [
            {
                // based on https://github.com/DonJayamanne/pythonVSCode/wiki/Debugging:-Flask#linuxos-x
                "name": "Debug Zenodo",
                "type": "python",
                "request": "launch",
                "stopOnEntry": false,
                "program": "${env:HOME}/.virtualenvs/zenodo/bin/zenodo",
                "env": {
                    "FLASK_DEBUG": "1"
                },
                "args": [
                    "run"
                ],
                //"debugOptions": [
                //    "WaitOnAbnormalExit",
                //    "WaitOnNormalExit",
                //    "RedirectOutput"
                //]
            }
        ]
    }
    

  • now you can set debug points in VSCode

Adding endpoints

1. Add a blueprint to a module in Zenodo - searching for @blueprint.route or @blueprint delivers useful examples
2. If the blueprint is in a new file, add the file to entry_points in setup.py
3. Important: you must run pip install -e . if you added the blueprint to a new file/new module. It is then "found" and subsequent changes in that file will be automatically deployed during development

Try out the testing endpoints:

• http://127.0.0.1:5000/api/similarity/
• http://127.0.0.1:5000/api/otherendpoint/1234/
• http://127.0.0.1:5000/api/records/1234/similar

Other stuff

• Searching Zenodo source code for "endpoints" then "RECORDS_REST_ENDPOINTS": endpoints are configured in zenodo/config.py
• In views (e.g. zenodo/modules/deposit/views.py) you can see who the config is accessed to get the right URL, e.g.

if latest_record:
            links['latest'] = current_app.config['RECORDS_API'].format(
                pid_value=latest_record.pid_value)

• More on the architecture: https://invenio.readthedocs.io/en/latest/application-architecture.html
• for adding something to records, extend zenodo/modules/records/views.py

Installation

Based on https://zenodo.readthedocs.io/en/latest/installation.html#development-installation

Virtual environment was created with

mkvirtualenv zenodo

using https://virtualenvwrapper.readthedocs.io/en/latest/ which is sourced via ~/.bashrc

Dependencies were installed using

pip2 install -r requirements.txt --src ~/git/zenodo-dev/ --pre --upgrade;
pip2 install -e .[all,postgresql,elasticsearch2]

i.e. switching out the suggested location for --src

Installed npm deps with nvm, not globally.

Tasks

• Developing Invenio locally, see https://zenodo.readthedocs.io/en/latest/development/guide.html, i.e. also run (parts of) Invenio from source