Proprietary Audio Plugins for Linux, but the right way

README.md

Proprietary Audio Plugins for Linux, but the right way

Why the current approach to VST and VST3 does not work for proprietary plugins on Linux ?

When building plugins for Linux there are two options and one problem:

1. Link to the system libraries
2. Build all your dependencies and statically link to them
3. Can't use gtk or qt.

What's wrong with solution 1.?

• It forces to build using an old system in order to maintain a maximum level of compatibility. Which prevents from using modern compiler and library features.
• There is no guarentee that the plugin will load on the user system.

What's wrong with solution 2.?

• It requires work to maintain your development libraries, some might not even be working on the user system (paths with fontconfig?, configuration files format change, ...).
• More work to ensure maximum compatiblity toward libm, libc, ...
• The host will probably use dlopen(RTLD_LOCAL), because RTLD_DEEPBIND creates various issues, breaks most plugins and is not used in the end. RTLD_LOCAL means that the plugin will use the symbols from the host before its own, which might turn out to be bad for multiple reasons. While if everyone uses the symbols from the system's runtime there is no conflict at all. One problematic scenario is:
  • host is staticaly linked against libX version N
  • plugin is dynamicaly linked against libX version N+1
  • libX_blah_create() returns an opaque pointer libX_ptr, which has a different data layout in version N+1
  • libX_doY(libX_ptr) is a new symbol introduced in version N+1 which takes this opaque pointer as argument; unfortunately in our case it was allocated by libX_blah_create() from version N, but then used by the symbol libX_doY() from version N+1 which expects a different data layout -> BOOM

Does a plugin wants Gtk or Qt? And why?

Yes absolutely! And, this is the biggest problem. This is so big that it is unrealistic. Right now a plugin needs to go through a huge pain in order to show a GUI. The reason why it can't use gtk or qt, is because those don't behave well when mixed together. And imagine multiple version being mixed together having symbols clash...

The consequence is that the plugin developper will use xcb, cairo, and basically write their own toolkit in order to display a knob. No one wants to write a GUI toolkit in order to write a plugin.

My proposal

1. Be able to run the plugins in their own process.
2. Plugin may not use an ABI but a protocol involving IPC.
3. There should be a registry of all the plugins available, to which you can communicate and create instances of a plugin.

How to realize it?

I will focus on the idea that DAWs an plugins will be distributed via flatpak and linked to its runtime. Obviously it does not make flatpak a requirement but instead forces the design to work across sandboxes and runtimes.

Plugins and DAWs may be linked statically or against a flatpak runtime, it does not matter as they will be isolated from each others.

Then, we need the plugin registry.

• it must be reachable from any sandboxes
• it must be able to index plugins available on the system and installed via flatpak
• it must be able to spawn plugin processes and establish IPC between the DAW and plugins

A proprietary plugin can:

• Link dynamically against (system|flatpak)'s runtime or link statically
• Use GTK or Qt (or whatever they like)
• Be distributed via flathub
• Be used by a DAW built against a different runtime

The plugin and daw can still implement vst2, vst3, lv2, ..., and it would be implicitely wrapped by a transparent host doing the daw <-> vst3 <-> bridge/plugin <--- IPC ---> bridge/host <-> vst3 <-> plugin.

We don't want to impose to the industry to support a new plugin api.

This will let a DAW sandboxed and started using runtime Y to use plugins from the system or even built for an other flatpak runtime.

In the end this work will turn Linux into a top level place for multimedia processing:

• Painless GUI using gtk/qt or whatever.
• Easy to develop
• Easy to debug
• Easy to distribute and get visibility in Flathub
• Easy to get a working system and stable system for the user
• Preserve user's freedom and control by linking dynamically to the flatpak runtime: the user has control over his runtime
• Preserve user's security by having different sanboxing parameters for each plugins and daw. Otherwise the DAW must have the maximum permissions requirements from each plugins, and it each plugins would inherit the daw's permissions which is not what we want, for example we may want to disable network access for most audio plugins.

Reducing IPC overhead

The overhead comes from context switching between processes, and the smaller the latency is (undestand audio interface buffer size) the higher the overhead becomes.

The solution is to bulk process.

Let's take a concrete exemple:

• DAW is using runtime X
• N plugins from the same vendor We could create a single plugin process for multiple plugins from the same vendor. That way, when the host host can group the processing requests toward nultiple plugins into one single bulk request, reducing the amount of context switch.

Bitwig is located here

/var/lib/flatpak/app/com.bitwig.BitwigStudio/x86_64/stable/6267badf04b6977108cfb068cedc00b8dae9d57e80c671c0db488088f54f831b/files/

Surge is located here

/var/lib/flatpak/runtime/org.freedesktop.LinuxAudio.Plugins.Surge/x86_64/20.08/7464c5cd5222d0d0228a8788ad6ff3bbf625429b43d7d67f5479dccce1e70cdb/files/

as far as I understand how plugins work they have to run within the Host... I am really not sure how this is supposed to work with Flatpak... the plugins would have to be inside the Host's container... as plugins do not execute

I would think there would need to be a universal runtime layer or container... one that acts as a go between the DAW of choice and every other plugin in the Flatpak ecosystem as each and every one of them will be in their own unique Silo and locked down System/Container

noticed that you starred this

will try this and get back... which runtime is Bitwig? does it matter?

Perhaps unorthodox idea, but why not build tailored Wine to the DAW? Like Steam does for games.

I personally use Windows versions even of the plugins for which there is a Linux version, just because it management wise easiest way to get shit done (when producing music). Other options is to have ~/.wine, ~/.vst, ~/.vst2 and what not.

All platforms considered it would be cool if plugin industry would give a shot doing something similar Sun tried to do years back with PWI (Public Windows Interface). It was a failure but in audio plugin it could work out. I.e. standardize a subset of Windows API that plugins would use. That could even mean ABI compatibility for plugin binaries using the same ISA

I.e. instead of macOS, Windows and Linux version of a plugin you'd just have ARM, x86 etc. versions

Plugins are in the just huge calculators with an UI, pretty simple stuff, when you think it that way. I/O wise they are most of the part as simple as Minesweeper game.

Plugin API's do not need any change. What is needed is a standard PRE (Plugin Runtime Environment) :-)

@tank-trax:

there is no way to navigate to this location as it is outside Flatpak Bitwig's container

From flatpaked Bitwig's perspective flatpaked plugins are in /app/extentions/Plugins/vst3 and /app/extentions/Plugins/lxvst. Need to add those paths in Bitwig's Settings->Locations. Not configured by default. Those paths are internal container paths.

@abique:

Can a DAW from the system use them?
Can a DAW using a different runtime use them?

You cannot use flatpaked plugins from non flatpak application. Each plugin extends linuxaudio base runtime which in turn extends freedesktop.org runtime and they all see those paths I mentioned earlier within the container's filesystem. I think you can use those plugins in any DAW that is built against the runtime which extends freedesktop.org runtime, for example gnome runtime, KDE runtime, or your own custom runtime as long as that custom runtime extends freedesktop.org runtime and in flatpak manifest it adds org.freedesktop.LinuxAudio.Plugins extention.

I do not remember exact technical reason why that linuxaudio extention was needed, though. Github user @hfiguiere is the creator of that extension and the most active maintainer, probably he could explain all this much better than me.

I think it is going to work well with flatpak with traditional plugin loading. Let us see.

@tank-trax

it connects audio using ALSA directly...

Btw, when using pipewire (default on latest Fedora) you can also use JACK as audio subsystem in Bitwig without needing to have actual JACK installed (real JACK will not work in flatpaked apps)

You cannot use flatpaked plugins from non flatpak application. Each plugin extends linuxaudio base runtime
[...]
I do not remember exact technical reason why that linuxaudio extention was needed, though. Github user @hfiguiere is the creator of that extension and the most active maintainer, probably he could explain all this much better than me

Not exactly. The thing with extension is that they need:

1. an extension point which determine what they are and where there are in the container - it is also important the filesystem mount point exists even to build.
2. a SDK to be build against

Typically an extension is specific to an app. Like a plugin for GIMP.
In that case, as we know, it's not, it just implement an API. Still it needs the two. So that "BaseExtension" app is only used to:

1. specify the extension point (like the application using it do)
2. specify a SDK, in that case the freendesktop-sdk, used to build it.

Note that the extension doesn't specify the SDK. The application does. Freedesktop-sdk is the baseline for both the GNOME and KDE runtime, so either can be used by the app.
There is no check performed.

The BaseExtension is just an empty shell.

Just to add, when I say no check is performed, the only check is that the version specified in the app add-extension object is matched against the branch for each extension.

version: '20.08' will get the extension from the //20.08 branch. But here nothing prevent an extension from being build against a different runtime, but it's just looking for trouble.
versions can specify multiple versions, but there is currently only a 20.08 branch.

Also the runtime matter much less if is no build (ie packaging from binaries)

Btw, when using pipewire (default on latest Fedora) you can also use JACK as audio subsystem in Bitwig without needing to have actual JACK installed (real JACK will not work in flatpaked apps)

The current Flatpak for Bitwig should work with Pipewire when using JACK, provided that Pipewire on the host also work (Fedora 34 for example). Any further trouble is linked to Pipewire, one notable issue being setting real-time priority.
It failed when the package was reviewed because of the bug in the runtime that has since been addressed.