shundhammer/libyui-packaging.md

## libyui-packaging.md

      
    Raw
  

              libyui-packaging.md
            
          
    Libyui Packaging with the New CMake Build Environment

Current Situation


Lots of different libyui*.rpm packages from lots of individual, independent GitHub repos


Most of those packages have a -devel subpackage for the header files and a -doc subpackage for auto-generated doxygen documentation


The main package of each has a .spec file and a .changes file in the repo's package/ subdirectory


The -doc subpackage has its own -doc.spec file and its own .changes file


The -devel subpackage is defined in the main package's .spec file; there is no separate .changes file


Both the -devel and the -doc subpackage have the same version number as the main package


The version numbering of all the individual libyui*.rpm packages is wildly different between them.


Requirements in SUSE Distros


Each distro or product may or may not contain some of the libyui*.rpm packages


Each software pattern of a distro or product may or may not contain some of the libyui*.rpm packages


Upon package upgrade (zypper dup), the resulting system needs to be consistent so YaST can be started and used; i.e. dependencies need to be set up to ensure that only compatible libyui*.rpm packages are on the target system on any given time.


Requirements by OBS


No build dependency cycles; there needs to be a clear starting and end point for building a set of packages

Requirements by the YaST CI (Jenkins, Travis)

Some of those requirements are not fulfilled right now; that's why we are working on this topic.


When there are changes in libyui, the CI needs to be able to bootstrap itself.
Currently, SO number changes introduce a vicious cycle: The docker image used for building libyui and dependent packages still contains an older version, so all packages other than libyui.rpm itself inevitably cause a ton of build problems, so both Travis and Jenkins keep failing until all libyui*.rpm packages make it to that docker image, but that doesn't happen until all PRs are test-built, reviewed and merged; but a successful Travis build is a prerequisite to review a PR. This cycle can only be broken by brute force and praying that everything will be alright.


Jenkins should be able to detect a change in the libyui source repo, check out the updated sources, test-build (at least) the changed packages and submit (at least) the changed packages to OBS if the test-build was successful.
In an ideal world, Jenkins would only submit any libyui*.rpm packages to OBS when they ALL build without problems.


The Simplistic Approach: Keep Packaging as it Is

We could just merge all our libyui* GitHub repos to a single big one containing a subdirectory for each and iterate manually or with simple scripts over them:
libyui
├── libyui
│   ├── doc
│   ├── examples
│   ├── legacy-buildtools
│   ├── package
│   └── src
├── libyui-ncurses
│   ├── doc
│   ├── package
│   └── src
├── libyui-ncurses-pkg
│   ├── package
│   └── src
├── libyui-ncurses-rest-api
│   ├── package
│   └── src
├── libyui-qt
│   ├── doc
│   ├── package
│   └── src
├── libyui-qt-graph
│   ├── examples
│   ├── package
│   └── src
├── libyui-qt-pkg
│   ├── examples
│   ├── package
│   └── src
├── libyui-qt-rest-api
│   ├── package
│   └── src
└── libyui-rest-api
    ├── package
    └── src

Each libyui/libyui* directory would be a self-sufficient package in its own right with a .spec file and a .changes file in its package/ subdirectory (and possibly one more for its -doc subpackage if it has one).
We'd have to update the version number in each one separately (rake version:bump), write a separate change log entry in its packages/*.changes file, build each one separately (mkdir build; cd build; cmake ..; make), create a tarball in each one separately (rake tarball). After a PR is reviewed and merged, Jenkins would do a test build and, if successful, submit a new package to OBS.
A number of simple scripts at the toplevel directory (libyui/) could support that manual work for bulk operations: for dir in libyui*; rake version:bump; done.
Pros


Simple


Not very disruptive for our CI, OBS and the distros


Keep using existing tools and workflows


Atomic source code changes: One PR for all libyui* packages (because it's now one single GitHub repo)


Cons


Responsibility rests heavily on the shoulders of the developer: There is no way to make sure that everything builds without problem, so partial submissions would remain a problem: Some libyui* packages build fine and are submitted, some may fail to build and cannot be submitted, again causing a lot of disruption for the release manager, OBS and QA.


The vicious circle in Travis and Jenkins remains: The libyui docker image cannot be updated until all packages are submitted, and that's not possible because Travis keeps failing for each libyui* package except libyui.rpm.


The Complex Approach: One Huge .spec File Containing all libyui* Packages

Instead of having a separate package/ subdirectory for each libyui* package, we would have one huge unified libyui.spec file in a toplevel package/ subdirectory.
That libyui.spec file would contain all the libyui*rpm packages; all except libyui.rpm would be subpackages.
There would be one single build command that builds everything, and the created files would be packaged by separate file lists in the huge libyui.spec file into all the subpackages. There would be one single libyui.changes file and one common version number for each of the subpackages.
We would have to add all the Requires and BuildRequires to the huge libyui.spec file, separated by subpackages.
Pros


Atomic source code changes: One PR for all libyui* packages (because it's now one single GitHub repo)


Atomic package submission because now it's all or nothing.


Cons


The one unified libyui.spec file would be incredibly huge and incredibly complex, thus very hard to maintain.


There are zero chances to submit any libyui* package if any subpackage fails to build, no matter if that subpackage is relevant to the target distro. If it fails because of changes that are outside our responsibility, we'd have to come up with workarounds.


All target distros would need to provide all Requires and BuildRequires, even if they are intended to be minimalistic: If it doesn't contain, say, graphviz (for libyui-qt-graph), it couldn't build, so we'd have to come up with workarounds. Similar with libyui-*rest-api: Some target distros may not even want to provide them for security considerations.


We'd probably have to adapt our auto-submission logic in our Jenkins CI jobs.


One Single Big Fat Tarball, Individual .spec Files

This was first inspired by Qt5 packages; initially I had thought they all share the same large tarball. On closer inspection, that is not the case at all. Still, the idea has merit.
Most libqt5*.rpm packages in Factory and/or SLE use the single large tarball from the qt-everywhere repo. While that tarball contains all the sources, only a part of them is actually used for building the resulting libqt5*rpm binary packages; yet they all build with a consistent set of header files that ensure that everything is binary compatible.
All the libqt5* packages still have their own .spec file, and they are submitted individually.


OBS Factory libqt5-qtbase


OBS Factory libqt5-qtx11extras


OBS Factory libqt5-qtsvg


...


We could do the same thing with the libyui* packages.
Pros


Atomic source code changes: One PR for all libyui* packages (because it's now one single GitHub repo)


We can simply add one layer of CMake CMakeLists.txt in the toplevel directory to enable or disable individual parts to be built (with or without Qt, with or without libzypp (libyui*-pkg)


Consistency / binary compatibility between libyui* packages: We can easily ensure that dependent packages use the headers from ../libyui/src in the same tarball (the proposed new CMake configuration in the huha-cmake branch already does this).


Cons


The source tarball becomes considerably bigger; it would be the same large one for each of the libyui* packages.

To Clarify


Enforce the same package version between our libyui* packages? (Not the community-maintained libyui-gtk*, libyui-mga*, of course!)


Enforce the exact same package number as Requires for all our libyui* packages, very much like between kernel and kernel module packages?


Automate common tasks like adding a new change log entry for each libyui* package when the version number changes? Provide a script that accepts a bsc# to be added along with the automated change log entry? E.g.
---------------------------------------------------------------
Wed Nov 11 17:17:45 UTC 2020 - MyRealName <myusername@suse.com>

- Related to bsc#4711
- 4.42.1
---------------------------------------------------------------

(The user name and mail address can be obtained with git config --show --no-show-origin user.name and git config --show --no-show-origin user.email)