gistfile1.txt

# Everything you need to know about static KF5 applications

Hello world!

It's been a long time since my last blog. In the coming weeks, I will publish
a series of post about Qt5/KF5 application minification. I spent almost a month
fixing all issues related to this and had literally no documentation or prior
work to follow. As a long time Gentoo users, I accumulated a large number of
tips and tricks about how to bend GCC to your will.

## What is a (semi) static KF5 application?

It is an application built with static libraries and plugins. It isn't a "real"
static executable since some libraries like OpenGl are provided by the system
and there is no way to avoid them unless you target a single system with
open Mesa compatible drivers. It also has to use the system GLibC since OpenGl
uses it. Using musl is only possible with Mesa.

## Why would you want to ever do this to your application?

Mostly for AppImage and FlatPak or for embedded systems. It is superior to copy
pasting system libraries into an AppImage and call it a day because:

 * It create much, much smaller payloads. Over 90% smaller in most case.
 * Once your pipeline is in place, it is easy to automate using the CI.
 * It allows to turn on extra optimizations such as LTO and PGO
 * It loads faster
 * It is less prone to changes to the host system
 * It has some security advantages (but also some drawbacks)
 * It unifies the Linux, Android, mingw and macOS build pipeline
 * It integrates well with CMake and Docker.io
 * It uncovers extra bugs in everything since no one tested this in ages...
 * It is easier to deploy on embedded systems

It is important to know the history here. Once upon a time, static libraries
ruled. In fact, they still rule on other OSes. But Linux came with the wonderful
concept of a package manager. With shared libraries, you could update the
library without updating anything else. It is great until we have to come to
reason that the new bundled format are not as granular. Sure, in theory they
can support bundled dependency, but it might not be worth doing. As Docker has
shown, sometime just replacing the whole container gets rid of some useless
complexity.

So static libraries are not "backward thinking". Shared libraries were
prevalent because it solved a problem we no longer have.

## Will I need a commercial Qt license?

All components have to be compatible with the GPLv3 as-is. GPLv2 or LGPL isn't
going to be enough. If you can't meet those requirements, then yes, you need
a commercial license for Qt and probably just about everything else.

## Can it creates single file portable binaries

Yes.

## Will it keep working forever?

As long as the system ships `libGl`, yes. There is no other factors to take into
accounts. GLibC is backward compatible and will hopefully always be.

## How to get started?

I use a `Dockerfile`, but this is out of scope of this blog so I will explain
the steps from scratch.

## How to add existing Qt plugins to your application

You need to add a `Q_IMPORT_PLUGIN` macro in `main.cpp` or a subproject you are
100% certain wont get touched by the dead code elimination pass.

I don't see any logic in them and it's undocumented. Sometime it's prefixed
`QtQuick`, sometime `QtQuick2` for no apparent reason. Sometime there is even
typos in them such as libdialogplugin.a being QtQuick2Dialog**s**Plugin

The best way to find the name is to us the find command:

    find /opt/usr/ -iname '*plugin.a' | grep -i dialog

Then `gcc-nm`:

    gcc-nm -C /opt/usr/qml/QtQuick/Dialogs/libdialogplugin.a | grep qt_static_plugin_

This will give:

    00000000 T qt_static_plugin_QtQuick2DialogsPlugin()

A more advanced bash function (but not compatible with all systems) is:

    function getpluginname() {
        gcc-nm -C $1 | grep -Po '(?<=qt_static_plugin_)([^(]+)'
    }

In this case, `Q_IMPORT_PLUGIN(QtQuick2DialogsPlugin)` will be the right macro.
Note that I am not sure if this is stable or unstable over time. At some point
a CMake module could automate this properly.

To get the list of all plugins:

    function getallpluginnames() {
        find $QT_DIR -iname 'lib*plugin.a' | xargs gcc-nm -C \
            | grep -Po '(?<=qt_static_plugin_)([^(]+)'
    }

## I stil get "undefined references" when linking

QMake add some extra linking flags. The easiest way to get the exact list is to
keep a sample project with the same plugins and copy-pasting the linker
arguments. However it can also be done manually with bash functions like:

    function findsymbol() {
        for file in $(find $QT_DIR -iname '*.a'); do
            armv7a-hardfloat-linux-gnueabi-gcc-nm $file -C 2> /dev/null \
                | grep $1 | grep -v " U " > /dev/null && echo $file
        done
    }


## How to handle sound

I recommend building against ALSA only. PulseAudio and JACK have a
compatibility mode and it will "just work". If you *need* Pulse or JACK features,
then either detect the host at runtime or accept that your app isn't fully
portable.

## How to correctly create Qt plugins with CMake

The only documentation about this is a StackOverflow entry saying it's
impossible. This is wrong, it *is* possible, but you have to know how `qmake`
does it. At first, I did read `qmake` code until I reverse engineered the
correct magic arguments.


## Creating new static plugins using CMake

There is little documentation of, but it can be done. Your own plugins can
either by internal libraries of QML and/or logic and real exported components
installed by your project.

### Getting stated

The following options needs to be set:

    set(CMAKE_AUTOMOC ON)
    set(AUTOMOC_MOC_OPTIONS -Muri=com.my.namespace)
    add_definitions(-DQT_PLUGIN)
    add_definitions(-DQT_STATICPLUGIN=1)
    set_target_properties(myplugin PROPERTIES AUTOMOC_MOC_OPTIONS -Muri=com.my.namespace)

It is the only way CMake will produce working Qt plugins. Note that the URI
needs to exist and if it is a QML plugin, it needs to be used. Do not try
to have QML plugins with multiple non-canonical namespaces. For example, if
the `Muri` is set to `com.foo.bar`, your plugin should not contain a new
`com.foo.baz`. It's asking for trouble and may have unintended side effects.
`com.foo.bar.baz` is fine.

### Creating a `.a` instead of `.so`

Many CMake projects hardcode shared libraries assumptions in their build. This
needs to be fixed. The most common is forcing the library type.

Do **not** use:

    add_library(mylib SHARED ${mylib_SRCS})

Use:

    option(BUILD_SHARED_LIBS "Build a shared module" ON)
    add_library(mylib ${mylib_SRCS})

### Limiting symbols

To avoid dependency hell, make sure not to export too many internal symbols
and let inlining work early when generating the plugin. Symbol collision is
much more common in static libraries than shared ones.
Use:

    set(CMAKE_C_VISIBILITY_PRESET hidden)
    set(CMAKE_CXX_VISIBILITY_PRESET hidden)
    set(CMAKE_VISIBILITY_INLINES_HIDDEN 1)

When using `target_link_library` **use** the private and public style instead
of the undefined visibility style.

### Putting QML files in a QRC.

If you know the Qt version in advance, you can compile the QML files and put
them into a resource file. Even if you wish the plugin to (try to) works across
many versions, you should still put the QML files (not compiled) in the plugin.

Unless your `qmldir` are perfect and the plugin not using the C++ format, some
files will need to be explicitly registered:

    qmlRegisterType(QStringLiteral("qrc:/com.my.namespace.private/MyPrivateFile.qml"), uri, 1, 0)

Note that if your QML module has alternate files to handle multiple Qt versions,
it is a bit useless to ship all variants in the QRC as it makes the binary
bigger for no reasons. The Qt plugins likely wont work across versions anyway.
To save space, the QML and QRC can be preprocessed:

The myplugin.qrc.in file:

    <RCC>
        <qresource prefix="com.my.namespace">
            <file>@myplugin_QML_DIR@/MyFile.qml</file>
            <file alias="MyAltFile.qml">@myplugin_QML_DIR@/@myplugin_myaltfile@.qml</file>
        </qresource>
        <qresource prefix="com.my.namespace.private">
            <file>@myplugin_QML_PRIVATE_DIR@/MyPrivateFile.qml</file>
        </qresource>
    </RCC>

CMake:

    if(STATIC_LIBRARY)
        # Set some variables to insert the right files in the QRC
        if(Qt5Qml_VERSION VERSION_LESS 5.10)
            set(myplugin_myaltfile MyAltFile_Qt59.qml)
        else()
            set(myplugin_myaltfile MyAltFile_Qt510.qml)
        endif()

        # `rcc` is a bit dumb and isn't designed to use auto generated files, to
        # avoid poluting the source directory, use absolute paths
        set(myplugin_QML_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../)
        set(myplugin_QML_PRIVATE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../private)

        # First, pre-process the QRC to add the files associated with the right Qt
        # version.
        configure_file(
            ${CMAKE_CURRENT_SOURCE_DIR}/../myplugin.qrc.in
            ${CMAKE_CURRENT_BINARY_DIR}/../myplugin.qrc
            @ONLY
        )

        # When using the static library, all QML files need to be shipped within the
        # .a file.
        qt5_add_resources(
            RESOURCES ${CMAKE_CURRENT_BINARY_DIR}/../myplugin.qrc
        )
    endif(STATIC_LIBRARY)

Sometime, adding an explicit import of you QRC is required to prevent the dead
code elimination from getting rid of them. There is probably a cleaner way, but
this seems to work:

in myplugin.cpp:

    #ifdef QT_STATICPLUGIN
     #include <qrc_myresourcename.cpp>
    #endif

Also, if your QML file do `import "a_subfolder"`, replace that  with
`import com.my.namespace.a_subfolder 1.0`.