Issue tracker and RPU metadata issues and solutions

notes.txt

Issue Tracker metadata requirements

* Components generally have 1 issue tracker in the Jenkins project infra (GH issues, Jira), but some have 2 in parallel
	- GH issues may (rarely) be in a different GH repo?
	- Jira component names may not match artifact ID
		- Explicitly decoupling these may be useful to allow renames of Jira components without breaking metadata (e.g. via ID)
* We do not want to support issue trackers outside Jenkins project infra for plugins published by the Jenkins project
* Some plugins have issue tracker metadata in pom.xml but it's often outdated or outright wrong.
* Allow for future integration in Jenkins plugin manager UI
	- Similar to URL, prefer UC data over plugin metadata, but support plugin metadata (nice for private plugins)
	- Existing pom.xml indicates plugin metadata is often trash
* Needs publishing via update-center; which in turn needs a static source to read from because computing would take too long
	- Alt 1: Separate process publishing issue tracker metadata? Still difficult to determine automatically due to inconsistencies
	- Alt 2: Manually maintain metadata e.g. via RPU

General RPU issues

* Unit of release is distributed across many files (multi-module projects), can be a problem for permissions and (new) CD tokens, e.g. all of Blue Ocean
* Separate components that share a ton of redundant metadata (uploaders, security contacts); e.g. much of Pipeline
* Files are error-prone to edit between 'path' and 'name'
* No concept of logical components ("plugins"), it's just raw Maven stuff
* Flat directory structure makes it unwieldy, too many individual files
* RPU is now doing a lot more than just permissions; should be renamed and the RPU tool moved into a separate repo.
* Conceptually decouple "ownership" and "release permission"
	- Not everyone who is allowed to release a component would be allowed to make decisions about the component
	- OTOH granting every account who is allowed to make decisions about a component the permission to release is potentially unnecessary

Solutions

* RPU: Add issue tracker metadata as a list (sorted and/or allow explicitly specifying preferred issue tracker)
* UC: Consume new RPU metadata and add it to output
* (Optionally) HPI Plugin: Write issue tracker metadata to META-INF to make it consumable in Jenkins without UC
* (Optionally) Core: Consume new UC/HPI metadata
* RPU: Collapse multiple files into one, e.g. single file per logical component (unit that is released), either a new format or just allowing multi-doc YAML files with still redundant metadata
* Initial issue tracker metadata import: Manually add new metadata from Jira components:
	- map between plugins, Jira components (misnamed?), GH repos (w/ issue tracker?)
	- Do we need to support other repos in metadata? Would be trivial depending on format; but we would not want to support issue trackers outside jenkinsci
	- determine preferred issue tracker if both exist
	- ignore empty issue trackers?

Proposed steps to a solution

A. Issue Tracker

1. Create Inventory of issue trackers (Jira, GH) and how many open/closed issues there are (optionally: Last issue filed date?). Map Jira components to plugins.
2. RPU: Batch edit all RPU files to add a list of issue trackers as appropriate. Support "jira" and "github".
2a. Jira/GH: Optionally disable unused issue trackers (when empty? when no issues are open?)
3. Consume new metadata
	UC: Add support for RPU metadata, document output format
	Plugin site: Can add direct support for RPU metadata, or wait for UC output support and consume from there
3a. UC: Get maintainer information from RPU metadata instead of pom.xml

B. RPU Metadata Overhaul

1. Define a new format and apply it to all components
2. Add support in RPU tool

Problems
* Should these two processes combined or done independently?
  - Separating:
    . May be confusing for those submitting and reviewing changes as multiple changes in succession.
    . Doing issue tracker metadata first, file format overhaul second will create more work as we need to adapt newly added consumers (UC2, plugin site API)
  - Combining:
    . Will take longer overall to have the first result.
    . Conflicts during (assumed longer) review periods are less likely.

RPU: Add issue tracker metadata as a list (sorted and/or allow explicitly specifying preferred issue tracker)

Not a single person on the mailing list talked about the list. I asked in that thread. I think people would be happy with a single one? A list would be better future proofing, but I don't see having multiple is ever a good thing. Maybe one active, and a list of extra urls or something for a sidebar link?

Jira components (misnamed?)

I've renamed most of the broken components, there are some with -plugin on the end, and some without, but otherwise they match up.

I'm worried about how large of scope this is

GH issues may (rarely) be in a different GH repo?

I've not seen this yet so far.

• Unit of release is distributed across many files (multi-module projects), can be a problem for permissions and (new) CD tokens, e.g. all of Blue Ocean
• Separate components that share a ton of redundant metadata (uploaders, security contacts); e.g. much of Pipeline

+1000

Maybe one active, and a list of extra urls or something for a sidebar link?

Yup, which is why I want one to be "primary" (explicit or implicitly through list order): Use that one for "report issues" link. Use all of them to build a list of open issues, or count open issues, or similar.

It depends on the use case. If it is to have a "report a bug" link in isolation (which is what I expect the first Jenkins core integration would be), just having one is probably fine. For listing open issues, like https://plugins.jenkins.io/git/#issues does, especially transition phases could be a problem.

I wrote something similar in jenkins-infra/plugin-site-api#97 (comment)

Adding support for multiple issue trackers in the data store and intermediate APIs is basically free even if it's almost always a single entry. Trying to add it afterwards will be more difficult, syncing up changes in multiple components, or adding compat code, etc.

(And FWIW if we improve support for tracking issues via GH, I expect we'll see some more components migrating; which in turn makes having list support more useful.)

It could be as simple as

issues:
- jira: git-plugin
- github: jenkinsci/git-plugin

Now the above is a bit redundant; but it would support explicitly declaring which is the primary tracker, e.g.

issues:
- github: jenkinsci/git-plugin
  primary: true

and the following simpler format wouldn't allow that:

issues:
- github

Perhaps a special syntax could be used to indicate "same repo as github", e.g.

issues:
- github: .
  primary: true

I should probably read up on YAML and may well find a saner representation, but I think it should be readable, concise, easy to understand, and ideally is easy to process.

---

Not a single person on the mailing list talked about the list. I asked in that thread.

Which thread? I looked at https://groups.google.com/g/jenkinsci-dev/c/LnvX3mX5Gek and there's just a few people responding, some not even discussing the metadata (e.g. one just brought up a good point of too long dir listings in RPU, which I added to the doc above).

Which thread?

Yea thats the one, nobody talked about having multiple trackers when i brought it up

I'm happy with any of those the first two, with the second being my preference.
I rather not do the "simplier" ones, i want to be explicit, so its easy to read and not worry about having to fix multiple systems if defaults ever change

I rather not do the "simplier" ones, i want to be explicit, so its easy to read and not worry about having to fix multiple systems if defaults ever change

Makes sense. Also means a list of >1 entry with none indicating as preferred would need to fail validation.

---

Regarding containing the scope of things, another option would be to add additional output to the RPU tool, similar to github.index.json which is used for incrementals: https://github.com/jenkins-infra/incrementals-publisher/blob/81b70396bfcfc396b0a1d2239c7c7620d03a34cc/README.adoc#environment-variables

ci.j.io URLs aren't great or trustworthy, so I would store the file on reports.jenkins.io, uploaded from trusted.ci. This would decouple the issue tracker metadata consumers (update-center2 and possibly plugin-site-api) from plugin metadata storage formats. The file could look similar to https://updates.jenkins.io/current/plugin-documentation-urls.json except it would have a list for each component.

So the minimal steps to add metadata to RPU YAML files and getting it to plugin site and/or Jenkins:

• Add new metadata to RPU YAML files similar to your open PR.
• Change RPU tool output to generate issues.index.json or whatever.
• Add upload from trusted.ci to reports.j.io to RPU Jenkinsfile
• Consume the periodically uploaded file in update-center2 (for Jenkins) and possibly plugin-site-api (unless going through uc2).

Thoughts?