j9ac9k/macOS_app_creation.md

## macOS_app_creation.md

      
    Raw
  

              macOS_app_creation.md
            
          
    Building a PyQt5/PySide2 macOS Application

Preface

As part of my professional duties, I have been tasked with developing a desktop application to analyze audio files.  This desktop application would need to be work across platforms (Windows, macOS and Ubuntu), and I cannot rely on dependencies having already been installed (such as Python itself).   Naturally I turned to Qt (and more specifically, Qt for Python) due to its capabilities for creating GUI applications that work across OSs.  As I could not be dependent on the end user having Python installed (so no deploying a wheel and calling it good), I decided it may be worth trying to make native installers (deb packages for linux, .app bundles and .dmg files for macOS, and setup.exe files for Windows).
In short, here were the objectives I aimed to have

Cross-Platform compatible GUI application
Deployment to users via native installers (no assumption of python being on the target machine)

This post details the steps I took to create the installer and some issues I came across the way.   I will be focusing on the macOS component as I did some extra steps such as code-signing my application, as well as notarizing it.
What this blog post will not contain is instructions on building and running a Qt application in Python in general.  To get a sample minimum working example after you install fbs you can run fbs startproject, and one will be generated for you.
Code Signing - What Is It

Code signing is the process of telling your end users that, you, or your organization is the one that developed the application.  To code-sign an application, you will need an Apple Developer ID Application Certificate which costs $100/year (you get more than the cert with the subscription, but the cert is what is needed here).  While code-signing is not a requirement for an end user to install an application, it is a requirement to get apple to let you by GateKeeper.

Note: fbs does have some code-sign support but at the time of this writing it is only for .deb packages on linux.  For macOS (and Windows) we will have to code-sign ourselves.

Shoulders of Giants

There are a number of libraries that are used along the way.  I wanted my application to be installable as a python binary wheel, and I wanted to deploy native installers for end users.  The python library that has an easy to use API and fantastic documentation generate the installers is fbs.  fbs makes heavy use of pyinstaller under the hood but provides a nice, cross-platform compatible API, but it requires some specific directory structure that is not intuitive (which I will get to later).
The work here could not be done without the hard work of maintainers and contributors of setuptools, pyinstaller, and fbs libraries.  Furthermore, the pyinstaller issue tracker has been a gold mine of information when I am getting unexplained behavior.
I also want to point out the Qt for Python developer gitter channel is a great place to ask for help if needed (these are the PySide2 developers, keep in mind they're pretty busy so don't expect a prompt reply).  The folks I have interacted with there have been very helpful, and have verified bugs I have come across.
Considerations As We Start


TIP: Pin your dependencies

Pin your dependencies folks.  pyinstaller is fragile (or I should say, the product pyinstaller creates is fragile), having pinned dependency versions lets you backtrack in your git history (bonus points if you use git bisect) to isolate what change in dependency suddenly broke your app bundle.  I have had version changes in numpy (1.16.4 > 1.17.0) and fbs (0.8.2 > 0.8.3), that completely break the pyinstaller generated bundles.
With pinned dependencies, you can go back in git history and recreate the environment at the time of the commit to a very precise level.  You can pin dependencies via pipenv or making use of the output of pip freeze and requirements.txt files. I currently use pipenv but using the output of pip freeze is sufficient.  With pipenv I ran into my first gotcha, you have to be careful  with which python binary it grabs.

GOTCHA: PySide 5.12 (and maybe more recent versions) has an issue where if you are using a conda distributed version of python, your application will not respect macOS's dark mode setting (tl;dr no dark mode for your app).  This is problematic with pipenv as when pipenv creates a virtual environment, it searches through $PATH for the first python binary that matches the version requirements, if that version of python comes from a conda distribution, you have silently broken dark mode on your application.  In short, if you use pipenv and have a conda distribution of python on your system, be very careful.


GOTCHA: pipenv does not handle OS specific dependencies well and you will have OS specific build dependencies.  If you're going to use pipenv I urge you to be careful with dependencies you will need come build time (like pypiwin32 for Windows), and store those dependencies outside of your Pipfile.

Project Directory Structure

fbs is very particular about the directory structure of your project.  Key here is to follow this structure precisely, and we will make our setup.py work with it.  I would suggest having your project code inside the src/main/python/my_app/ directory, as this makes the name-space a bit easier to work with.  I will not repeat the contents of fbs's documentation here, just follow their recommended structure.

TIP  Do not just pack all your requirements into setup.py's install_requires, use a requirements.txt file.   The requirements.txt file is the same as ./requirements/base.txt, so you can just copy it over as part of your build process.  The trick will be to keep only runtime dependencies in your requirements.txt file, so you will need to put developer dependencies elsewhere (like requirements-dev.txt or setup.py's extra_requires).

To make setup.py conform to this directory structure we just need to add the following lines inside our setup call.
from setuptools import find_packages, setup

...

setup(
    ...
    packages=find_packages("src/main/python"),
    package_dir={"": "src/main/python"},
    ...
)
The main reason I wanted to make my application into an installable package is that makes things much easier come testing, I can import my modules via import MyApp instead of from src.main.python import MyApp.
Steps Before Building the .app Bundle

Before building the app bundle, there are a few checks we want to do to ensure things will run correctly.


Ensure fbs run actually launches the application.
This has been somewhat of an issue for me as I want to keep fbs as a non-runtime dependency due to it's GPLv3 license, and from the fbs manual

When you use fbs, you typically add references to fbs_runtime to your source code. For example, the default main.py does this with ApplicationContext from this package.

What I do here is I have a src/main/python/main.py file that includes the following command:
import sys
from fbs_runtime.application_context.PySide2 import (
    ApplicationContext,
    cached_property,
)

from my_app.App import MyApp

class AppContext(ApplicationContext):
    @cached_property
    def app(self) -> MyApp:
        app = MyApp()
        return app.qtapp

    def run(self) -> int:
        self.app.showMainWindow()
        return self.app.exec_()

def main() -> None:
    appctxt = AppContext()
    exit_code = appctxt.run()
    sys.exit(exit_code)


if __name__ == "__main__":
    main()
Also from the fbs documentation:

What you usually don't want to do is to refer to fbs from your application's implementation (src/main/). If at all, you should only refer to fbs from build scripts (build.py and/or src/build/python/).

What this lets you do is you can have fbs run launch your application, but we are not required to have fbs as part of runtime dependencies.  The alternative being launching your application via setuptools entry point, which can be of the form of of running python src/main/python/my_app/__main__.py, and having a corresponding file with something along the lines of
import os
import sys


def main() -> None:
    os.environ["QT_API"] = "pyside2"
    from App import MyApp

    app = MyApp()
    app.display()
    sys.exit(app.qtapp.exec_())


if __name__ == "__main__":
    main()


Update ./src/build/settings/base.json
Here are the basic contents of base.json for my app:
{
  "app_name": "my_app_name",
  "author": "my_company_name",
  "version": "0.1.0",
  "main_module": "src/main/python/main.py",
  "public_settings": ["app_name", "author", "version"],
  "hidden_imports": []
}

TIP: The hidden_imports field is the place you will add modules to that pyinstaller did not grab on its own during the bundling process.  For my applications, I had to include PySide2.QtMultimediaWidgets when I added audio playback capability to my application.


Create an Info.plist file
... and place it in ./src/freeze/mac/Contents/Info.plist.  The following is what I created, mostly getting tips from google.  The ${} fields are grabbed from the build.json file above under the public_settings key.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleExecutable</key>
    <string>${app_name}</string>
    <key>CFBundleDisplayName</key>
    <string>${app_name}</string>
    <key>CFBundlePackageType</key>
    <string>APPL</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleIconFile</key>
    <string>Icon.icns</string>
    <key>CFBundleIdentifier</key>
    <string>com.mycompany.mygroup.app_name</string>
    <key>CFBundleDevelopmentRegion</key>
    <string>English</string>
    <key>LSBackgroundOnly</key>
    <string>0</string>
    <key>CFBundleShortVersionString</key>
    <string>${version}</string>
    <key>CFBundleVersion</key>
    <string>${version}</string>
    <key>CFBundleName</key>
    <string>${app_name}</string>
    <!-- Enable Retina support on OS X: -->
    <key>NSPrincipalClass</key>
    <string>NSApplication</string>
    <key>NSRequiresAquaSystemAppearance</key>
    <string>NO</string>
    <key>NSHighResolutionCapable</key>
    <string>YES</string>
</dict>
</plist>


With that done, you should be ready to move onto actually building the .app bundle!
Making the Bundle and Installer

The commands to run here are straight forward
cp -f requirements.txt ./requirements/base.txt
fbs freeze
The first copy command is to ensure fbs will grab the relevant dependencies, the second command will generate the ./target/MyApp.app bundle.  This process may take a minute or so, and the terminal may output a bunch of errors/warnings along the lines of
ERROR: Can not find path ./libshiboken2.abi3.5.12.dylib (needed by ....)

but if the .app bundle built, best thing to do is just attempt to open that and see what happens
open ./target/MyApp.app`
If this executes your application, congratulations you've done it.  The next step is to generate the .dmg installer, which can be done by running:
fbs installer
That command will produce a MyApp.dmg file that you can distribute to others.
If you do not have an Apple Developer account/subscription, or if you just have no interest in code-signing your application, you can stop here.
Code-signing and Notarizing

If you want to code-sign your application, it can be done, but fair warning, you are proceeding into mostly uncharted territory.  Apple's documentation written with Xcode projects in mind, so a lot of it will not be relevant.  That said, the issue tracker on pyinstaller has been very helpful for debugging issues.
Fix Generated .app Bundle

Before code-signing your application, we actually need to "fix" the app bundle that pyinstaller generated.  Here is a link to the issue on the pyinstaller repo, and the associated pyinstaller wiki entry.  The issue tracker goes into what the issue is, but it fixes a couple of problems, one of which is macOS treating directories with periods in them as separate bundles.  The tl;dr is that you should run the script against your fbs generates .app bundle.
The modifications I made removed unneeded files that would not preserve their extended attributes during a move, and would then invalidate the code-signing.
Create Application Entitlements (optional)

One of the requirements for application notarization (which will be required starting macOS 10.15) is code-signing with a hardened runtime.  There may be "entitlements" you want to enable your application to have (check the link above for what's available), but if you are bundling numba or anything else that wants to execute JIT-compiled code, you should consider adding the Allow Execution of JIT-compiled Code Entitlement.
To add entitlements is fairly straight forward, in your project root folder create a file .entitlements, and put the following contents in it (of course add entitlements you think are needed).
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
</dict>
</plist>
Finally To the Code-signing Part

After running this step, you can now code-sign your .app bundle!
codesign -fs "$CODESIGN_ID" --deep --force --verbose -o runtime --entitlements .entitlements --preserve-metadata=identifier,entitlements,requirements,runtime --timestamp ./target/MyApp.app
Some explanation of the options

-o runtime enables hardened runtime (required for app notarization)
--entitlements .entitlements passes the entitlements you created to your application
--timestamp enables secure timestamp option (also required for notarization)
--deep does a recursive code-sign through the bundle.  The Apple documentation recommends using this only for troubleshooting, however the documentation is aimed toward Xcode projects, and not particularly relevant in our use case.
"$CODESIGN_ID is an env variable for my "Developer ID Application" certificate name that I have stored in my system keychain

Verifying the codesign operation


GOTCHA: Doing a verification of the codesign command ran as expected after you do run codesign sounds reasonable, right?  WRONG queue evil laugh.  Part of the codesign operation involves verifying the "extended attributes" of the files inside the .app bundle.  When you copy/move the .app bundle, the extended attributes change, but if you never touch the file, the extended attributes will remain fixed, and a codesign verification step may falsely "pass" (don't ask me how long it took me to troubleshoot this).

To address this, before verifying the codesign operation, we need to modify the extended attributes, which can be done by running the following command.
xattr -cr ./target/MyApp.app
Once that is run, then we can execute a codesign verification step.
codesign --verify --verbose ./target/MyApp.app
That should return
./target/MyApp.app: valid on disk
./target/MyApp.app: satisfies its Designated Requirement

Feeling paranoid? I know I have been during this process, there is a second way we can verify the code-signing operation
spctl --assess --verbose ./target/MyApp.app
This should return
./target/MyApp.app: accepted
source=Developer ID

At this point, the application is code-signed, and we should generate our .dmg file.
fbs installer
That should return something along the lines of:
waited 1 seconds for .DS_STORE to be created.
hdiutil: internet-enable: enable succeeded
Created target/MyApp.dmg.

So now we have a .app bundle that is code-signed and a .dmg file that is made from the code-signed bundle; next step, we need to notarize the application.
Notarizing your Application

This is a case where the Apple Documentation is actually quite helpful.
First Time Only Commands

The following commands only need to be run once on a system.
First, we need to select the appropriate Xcode.app
sudo xcode-select -s /path/to/Xcode10.app
Next, we need to ensure our Apple developer credentials, with a application-specific password are set in our keychain.
security add-generic-password -a $APPLE_DEVELOPER_EMAIL -w $PASSWORD -s "AC_PASSWORD"
Once that is done, we can move on with actually performing the notarization.
Do the Notarization

To perform the notarization run the following command
xcrun altool --notarize-app --primary-bundle-id "com.my_company.my_group.my_app" --username $APPLE_DEVELOPER_EMAIL --password "@keychain:AC_PASSWORD" --file ./target/MyApp.dmg

for the argument to --primary-bundle-id place what you have in your Info.plist file earlier
replace $APPLE_DEVELOPER_EMAIL with your apple developer email address

Once the upload succeeds, you should get an identifier back along the lines of
RequestUUID = 2EFE2717-52EF-43A5-96DC-0797E4CA1041

You can query the status of the notarization by running the following command (there is a modification to the command you can can run by the RequestUUID parameter you got earlier).
xcrun altool --notarization-history 0 -u $APPLE_DEVELOPER_EMAIL -p "@keychain:AC_PASSWORD"
If there is an error, it will return a URL you can look at and see more specific issues.

GOTCHA: PySide 5.12.0, 5.12.1, 5.12.2, 5.12.3, 5.12.4 and 5.13.0 on PyPi had some of their binaries build linking to an incompatible SDK.  They are fully functional, but if you use the wheels them, notarization will fail.  I reported the issue to their bug-tracker here and they have resolved it in time for the next release (5.12.5 and 5.13.1).  If you are using any of these libraries you will need to build the pyside2 wheels yourself, and when installing dependencies of your app you will need to install those custom built wheels, not the ones from PyPI.  See the Appendix for instructions

If it the query status says the package is accepted, you can "staple" the notarization with the following command
xcrun stapler staple "MyApp.app"
xcrun stapler staple "MyApp.dmg"
Appendix

Import Error During Runtime

Periodically you get an import error during runtime.  I got this when I tried to play audio through QAudioOutput configured device.  Error was along the lines of an import error seeking PySide2.QtMultimediaWidgets.
The solution to this issue, is to add that import to the list of hidden_imports in your ./src/build/settings/base.json file.
Similarly, when I noticed my .app bundle wouldn't run after I upgraded from numpy 1.16.4 to 1.17.0, I had to add some modules to the hidden_imports list to solve the issue, specifically:

numpy.random.common
numpy.random.bounded_integers
numpy.random.entropy

For the record, I do not think these needed to be explicitly defined prior as of numpy 1.17.1.  Once those imports/modules were added, and I rebuilt my .app bundle, it ran successfully.
That hidden_imports feature will often be the first place to look to patch things up, especially when you know it is an issue with part of a dependency that is missing.
Building Custom PySide2 Wheels

If you are using PySide 5.12.0, 5.12.1, 5.12.2, 5.12.3, 5.12.4, or 5.13.0, and plan to notarize your application, you will need to build your own wheels, as there is an issue with the SDK linked.  Luckily for us, the Qt for Python folks have provided instructions on building PySide from source.
I won't repeat the instructions on the wiki here, but the dependencies needed to build the wheels involve

Xcode
Qt (matching version) installed

Should contain qmake binary


CMake
setuptools and wheel python libraries

The command to build the wheel is
python setup.py bdist_wheel --qmake=/path/to/qmake --ignore-git --parallel=8 --standalone -macos-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

--qmake= should contain the path to your qmake binary that should will be in your associated version of Qt you installed earlier as part of the dependencies
--macos-sysroot= should be the path to the MacOSX10.X.sdk, I put the path for it on my machine, but it may be different on yours, verify this is accurate.

The build time on my MacBook Air was approximately 30 minutes.  3 wheels will be generated, pyside2, shiboken2 and shiboken2-generator.  When building your application, you can install the wheels by running
pip install --no-index --find-links=path_to_wheel_directory pyside2
This will install the pyside2 and shiboken2 wheels you just created instead of the ones on PyPI.
Querying Git Commits for Versioning

This was a "gotcha" that was an issue for almost a month before I figured out what happened.  A coworker submitted a merge request that modifies the version of the application by appending the first 6 characters of the current commit hash to the version number.
Perfectly sensible.
What the issue was, that it did this at runtime, and when the application is in bundle form, it's no longer in a git repository, thus querying the git status must generate an error.
The frustrating part here is that the failure mode was when opening MyApp.app, the icon would bounce for a few seconds, and then just close out, with no meaningful error of any kind.
The way I isolated this issue was via git bisect, and I tracked down the commit with the diff, which incorporated the feature I just described.
Logging

While I am still working on debugging this, what I feel comfortable saying right now is that logging.FileHandler('any_file.log') will cause MyApp.app bundle to fail on launch silently (just like the previous git issue).
My working theory here is that logging.FileHandler() takes only relative paths, not absolute paths, and it's attempting to create a log file in a write protected area of the .app bundle.
After some googling, the main suggestion seems to be to create your own logging handler.  I have not implemented this myself yet, so I cannot speak to it, so for now the only advice I'll give you, is to not write logs via FileHandler.
If any readers have a good solution to the logging issue with python app bundles, please let me know, I am all ears.
Homebrew coreutils Package

If you use homebrew and have the coreutils package, make sure you do not have your PATH modified so that the g-prefixed utilities under their normal names.  Modifying PATH so that the coreutils utilities are available under their normal names will break the fbs installer command.  If you run into this issue, you likely just need to remove this line in your .bashrc or .zshrc.
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"

PyInstaller Virtualenv and Distutils

Talk about distutils bundling issue with pyinstaller