ODS 3.0

ODS3.md

Work Packages for OpenDevStack 3.0

Release timeline: May 2020. Theme: Quality

Shared Library Testing

Owner(s): ? / Size: M / Benefit: Reliable pipelines, more confidence in changes / Ticket(s): #149, #146

Write automated tests for the shared library. Could be a mixed of tests with mocks and E2E tests. Run e.g. in Travis or Github Actions.

Project Provisioning Testing

Owner(s): @hugowschneider, ? / Size: M / Benefit: Reliable provisioning, more confidence in changes / Ticket(s): #342, #343, #341

Write automated tests for project creation. Should probably be E2E tests mostly. Run e.g. in Travis or Github Actions.

Quickstarter Provisioning Testing

Owner(s): @hugowschneider, ? / Size: M / Benefit: Reliable quickstarters, more confidence in changes / Ticket(s): ?

Write automated tests for each quickstarter creation. Should probably be E2E tests mostly, ideally all the way from provisioning through to running component. Run e.g. in Travis or Github Actions.

Image Building Testing

Owner(s): @henrjk / Size: S / Benefit: Reliable images, more confidence in changes / Ticket(s): ?

Write automated tests for all images (Jenkins master/slave/webhook-proxy, SonarQube). Ideally for CentOS and RHEL. Run e.g. in Travis or Github Actions.

SonarQube Pull Request integration

Owner(s): @michaelsauter / Size: M / Benefit: Improve code quality, bring SonarQube closer into feedback loop / Ticket(s): #174

Annotate pull requests with feedback from SonarQube.

Component Health Checks

Owner(s): ? / Size: M / Benefit: Zero-downtime rolling deployments / Ticket(s): #11

Add a /health endpoint to each quickstarter and configure a readiness check against it.

Documentation re-organisation, extension, cleanup ...

Owner(s): ? / Size: M / Benefit: Easier to find relevant information; clearer for contributors / Ticket(s): #20, #25, #24, ...

The documentation has actually three main target groups: users (people who create projects and build components using quickstarters), administrators (maintaining an OpenDevStack installation) and contributors (developers working on OpenDevStack itself). The documentation should reflect this. Further, where documentation is written (location of the files, how they are linked etc.) is still a bit unclear and can be improved.

Move provisioning app to central CD namespace

Owner(s): ? / Size: M / Benefit: Simplify ODS installation / Ticket(s): ?

Most ODS users do not develop on the provisioning app, therefore ODS should ship with just one provisioning app, located in the central CD namespace by default. That would also mean we have one running Jenkins in the central CD namespace which will make it easier to build custom images (such as docgen) easily. Users who also want to develop the provisioning app should be provided an alternative way to still develop on the prov app. This change requires a modification of the shared library as well to allow deployment into the cd namespace.

Refactor provisioning app authentication

Owner(s): @oalyman, @stitakis / Size: L / Benefit: Better security / Ticket(s): ?

Users should not need to enter their password in the prov-app, but rather authenticate via OpenID Connect. Further, the prov-app should use a technical user to communicate with other services such as Jira, Bitbucket, etc. if possible

Gateway quickstarter

Owner(s): @gerardcl / Size: M / Benefit: Plug-and-play gateway quickstarter / Ticket(s): #56

Investigate multiple options (e.g. Kong, Gloo, Nginx, ...) and provide one good default.

Good Scala quickstarter

Owner(s): @oalyman / Size: S / Benefit: More useful Scala quickstarter / Ticket(s): ?

Using Play framework. Retirement of Akka based quickstarter.

Proper Quota Support for build environment pods

Owner(s): @michaelsauter / Size: S / Benefit: Less resources used, easier to adapt to different needs for users / Ticket(s): #173

Build slaves should have CPU and memory constraints assigned (fitting to the requirements of the language/framework used). The shared library should expose an easy way to customise those constraints.

GitHub Changelog Automation

Owner(s): ? / Size: S / Benefit: Quicker to create a release / Ticket(s): #197

The changelog should be auto-generated from pull requests so that no manual process is required.

Proper templating for quickstarters

Owner(s): ? / Size: S / Benefit: Easier to write and modify quickstarters / Ticket(s): #15

Instead of sed replacements, use a proper templating approach to render e.g. Jenkinsfile.template or sonar-project.properties.template.

Quickstarter ownership and quality improvements

Every quickstarter should have at least one maintainer. There could be more maintainers, and maintainers might rotate for every ODS release cylce (half-year). A maintainer should:

• ensure that used dependencies are up-to-date
• ensure that all features (e.g. SonarQube) work out-of-the-box
• ensure that Nexus is configured correctly as a proxy if applicable
• ensure completeness of documentation
• keep an eye on issues on GitHub related to the quickstarter, and answer questions as much as possible

Being an owner does not mean that you need to provide support to users! Users needing help need to ask on GitHub and/or within BI platforms

Having no maintainer for a quickstarter means that this quickstarter is not suitable to be shipped with ODS. As a consequence, that quickstarter should not be part of ods-quickstarters going forward, and will have to be moved to some other repository which users of ODS may still use in their installation, but at their own risk.

Owners:

• airflow-cluster: @hugowschneider @gerardcl
• be-golang-plain: @michaelsauter, @henrjk
• be-java-springboot: @stitakis
• be-python-flask: @buegelbeatz, @henrjk
• be-scala-akka: discontinued, will be replaced with be-scala-play (owned by @oalyman)
• be-typescript-express:
• docker-plain: does not need a specific owner
• ds-jupyter-notebook: @hugowschneider @gerardcl
• ds-ml-service: @sklingel @hugowschneider @gerardcl
• ds-rshiny:
• e2e-cypress: @herrkoch
• fe-angular: @cschweikert
• fe-ionic: @rianet, @bljubisic
• fe-react:
• fe-vue:
• release-manager: @metmajer

Horizon / Other potential topics

• Refactoring of provisioning app frontend (unclear what it should be, might depend on BE refactor)
• Image Scanning and runtime protection (Aqua Security?) -> @rdupont
• Installation automation (might be easier after testing is complete)
• OpenShift 4 support (should come, but no urgent need right now, maybe in second half of 2020)
• Project specific Nexus and SonarQube access (currently feels like it needs more time/input)
• Tailor deploy (depending on MRO input concept and general approach to infrastructure-as-code approach for OpenShift)

I would be like to own "Refactor provisioning app authentication"

I would like to take ownership on fe-angular.

Testing

I agree that test coverage should be a top priority. In particular all quickstarters should be E2E tested regularly.

Has anyone looked at community/sig-testing at master · kubernetes/community yet? One thing that seemed interesting to me is "Gopherage is a tool for manipulating Go coverage files. We use it on the Kubernetes project to report on code coverage due to e2e tests".
Would be cool to have something similar for our e2e tests.

One thing I have on my radar is to keep an eye on covering issues already fixed so they will not regress in the future. Some examples:

• Avoid disclosure of tokens in logs - ods-project-quickstarters - 340
• Using bad names for projects caused failure during rundeck but as far as the provisioning app is concerned it worked and the name cannot be reused anymore. This should never happen. It is possible this is fixed at the moment, but E2E tests should catch regressions on this one.
• auto clone environment failing - ods-project-quickstarters - 331
• Components previously deleted reappear in cloned environment - ods-core - 318

I also remember tailor having trouble applying patches in certain edge cases, perhaps an E2E tests would make sense to make this more robust.

Image Build Testing

Do you have some more input on what this might entail?

Quickstarters

I would like to become a second maintainer for be-golang-plain.

Perhaps we can introduce labels for each quickstarter which would be part of the issue template and could be used to filter issues.

Perhaps we can utilize badges to show dependency staleness or other quality issues on the quickstarters.

In be-python-flask I see quite a few issues - at least as used in ASAP - perhaps working together with Hugo and Gerard we can improve the situation for the quickstarter, but I don't see myself as a maintainer in this arena at the moment as I am somewhat a newbie in Python.

Proper templating for quickstarters

I could work on this, but I think the testing field is more important at the moment and I have doubt I will have the cycles until May.

Unfortunately I can already anticipate that I will only be able to work on some of these items, but I definitely like to help out as much as time allows!

Cheers,

Henrich

I would like to take ownership of be-java-springboot.
Apart from that I'll work with Jan on "Refactor provisioning app authentication"

@henrjk thanks for the detailed response! Regarding image building: the idea would be to build the container images for every commit. E.g. sometimes the image building fails because some step in the Dockerfile downloads a dependency which no longer exists ... or someone updated the Dockerfile with some new code that actually does not work due to a typo. Things like this would be caught if we'd simply run docker build in GitHub Actions or Travis. Problem is that the RHEL based images might not work outside a RedHat-based host environment.

I would like to add myself as second to fe-ionic

hi!
regarding quickstarters:

• for ds-ml-service: @sklingel @hugowschneider @gerardcl
• for airflow: @hugowschneider @gerardcl
• for ds-jupyter-notebook and ds-rshiny: @hugowschneider @gerardcl

regarding workpackages:

• I can go over the gateway package too