Proprietary Audio Plugins for Linux, but the right way

README.md

Proprietary Audio Plugins for Linux, but the right way

Why the current approach to VST and VST3 does not work for proprietary plugins on Linux ?

When building plugins for Linux there are two options and one problem:

1. Link to the system libraries
2. Build all your dependencies and statically link to them
3. Can't use gtk or qt.

What's wrong with solution 1.?

• It forces to build using an old system in order to maintain a maximum level of compatibility. Which prevents from using modern compiler and library features.
• There is no guarentee that the plugin will load on the user system.

What's wrong with solution 2.?

• It requires work to maintain your development libraries, some might not even be working on the user system (paths with fontconfig?, configuration files format change, ...).
• More work to ensure maximum compatiblity toward libm, libc, ...
• The host will probably use dlopen(RTLD_LOCAL), because RTLD_DEEPBIND creates various issues, breaks most plugins and is not used in the end. RTLD_LOCAL means that the plugin will use the symbols from the host before its own, which might turn out to be bad for multiple reasons. While if everyone uses the symbols from the system's runtime there is no conflict at all. One problematic scenario is:
  • host is staticaly linked against libX version N
  • plugin is dynamicaly linked against libX version N+1
  • libX_blah_create() returns an opaque pointer libX_ptr, which has a different data layout in version N+1
  • libX_doY(libX_ptr) is a new symbol introduced in version N+1 which takes this opaque pointer as argument; unfortunately in our case it was allocated by libX_blah_create() from version N, but then used by the symbol libX_doY() from version N+1 which expects a different data layout -> BOOM

Does a plugin wants Gtk or Qt? And why?

Yes absolutely! And, this is the biggest problem. This is so big that it is unrealistic. Right now a plugin needs to go through a huge pain in order to show a GUI. The reason why it can't use gtk or qt, is because those don't behave well when mixed together. And imagine multiple version being mixed together having symbols clash...

The consequence is that the plugin developper will use xcb, cairo, and basically write their own toolkit in order to display a knob. No one wants to write a GUI toolkit in order to write a plugin.

My proposal

1. Be able to run the plugins in their own process.
2. Plugin may not use an ABI but a protocol involving IPC.
3. There should be a registry of all the plugins available, to which you can communicate and create instances of a plugin.

How to realize it?

I will focus on the idea that DAWs an plugins will be distributed via flatpak and linked to its runtime. Obviously it does not make flatpak a requirement but instead forces the design to work across sandboxes and runtimes.

Plugins and DAWs may be linked statically or against a flatpak runtime, it does not matter as they will be isolated from each others.

Then, we need the plugin registry.

• it must be reachable from any sandboxes
• it must be able to index plugins available on the system and installed via flatpak
• it must be able to spawn plugin processes and establish IPC between the DAW and plugins

A proprietary plugin can:

• Link dynamically against (system|flatpak)'s runtime or link statically
• Use GTK or Qt (or whatever they like)
• Be distributed via flathub
• Be used by a DAW built against a different runtime

The plugin and daw can still implement vst2, vst3, lv2, ..., and it would be implicitely wrapped by a transparent host doing the daw <-> vst3 <-> bridge/plugin <--- IPC ---> bridge/host <-> vst3 <-> plugin.

We don't want to impose to the industry to support a new plugin api.

This will let a DAW sandboxed and started using runtime Y to use plugins from the system or even built for an other flatpak runtime.

In the end this work will turn Linux into a top level place for multimedia processing:

• Painless GUI using gtk/qt or whatever.
• Easy to develop
• Easy to debug
• Easy to distribute and get visibility in Flathub
• Easy to get a working system and stable system for the user
• Preserve user's freedom and control by linking dynamically to the flatpak runtime: the user has control over his runtime
• Preserve user's security by having different sanboxing parameters for each plugins and daw. Otherwise the DAW must have the maximum permissions requirements from each plugins, and it each plugins would inherit the daw's permissions which is not what we want, for example we may want to disable network access for most audio plugins.

Reducing IPC overhead

The overhead comes from context switching between processes, and the smaller the latency is (undestand audio interface buffer size) the higher the overhead becomes.

The solution is to bulk process.

Let's take a concrete exemple:

• DAW is using runtime X
• N plugins from the same vendor We could create a single plugin process for multiple plugins from the same vendor. That way, when the host host can group the processing requests toward nultiple plugins into one single bulk request, reducing the amount of context switch.

If the DAW and plugin are built for different runtimes, they won't be able to see each other and there is no dlopen() which could work

Ah, I see. That clears things - thanks.

I built Bitwig Studio 3.3.1 flatpak for myself and tested with VST3 Surge as it is currently shipped via Flathub - works perfectly fine

There's a flathub distribution of surge out there?

There's a flathub distribution of surge out there?

Oh yes - org.freedesktop.LinuxAudio.Plugins.Surge. Works perfectly fine for me.

Oh yes - org.freedesktop.LinuxAudio.Plugins.Surge. Works perfectly fine for me.

@baconpaul here it is...

@baconpaul @takuvata I don't understand how it currently works.

flatpak search vst3
Name                      Description                                                        Application ID                                              Version          Branch       Remotes
DISTRHO-Ports             DISTRHO Plugin Ports LV2/VST/VST3                                  org.freedesktop.LinuxAudio.Plugins.DISTRHO-Ports            2021.03.15       20.08        flathub
sfizz                     sfizz is a sample-based musical synthesizer LV2/VST3 plugin        org.freedesktop.LinuxAudio.Plugins.sfizz                    0.5.1            20.08        flathub
sfizz                     sfizz is a sample-based musical synthesizer LV2/VST3 plugin        org.freedesktop.LinuxAudio.Plugins.sfizz                    0.4.0            19.08        flathub
WhiteElephantAudio        White Elephant Audio VST3 plugins                                  org.freedesktop.LinuxAudio.Plugins.WhiteElephantAudio       2021.03.05       20.08        flathub
Surge Synthesizer         Surge Synthesizer LV2/VST3                                         org.freedesktop.LinuxAudio.Plugins.Surge                    1.8.1            20.08        flathub
Surge Synthesizer         Surge Synthesizer LV2/VST3                                         org.freedesktop.LinuxAudio.Plugins.Surge                    1.7.1            19.08        flathub
Dexed                     Dexed FM Synthesizer VST/VST3 plugin                               org.freedesktop.LinuxAudio.Plugins.Dexed                    0.9.6            20.08        flathub
Dexed                     Dexed FM Synthesizer VST/VST3 plugin                               org.freedesktop.LinuxAudio.Plugins.Dexed                    0.9.5            19.08        flathub

Can a DAW from the system use them?
Can a DAW using a different runtime use them?

https://github.com/flathub/org.freedesktop.LinuxAudio.BaseExtension

flathub/org.freedesktop.LinuxAudio.BaseExtension#5

@abique - No idea!
@tank-trax - can you help out?

just installed Bitwig Flatpak (on a non-Audio dedicated workstation) and will see if I can get it to recognize plugins

so far it recognizes the two synths that are located in ~/.vst3

am installing Surge via flatpak install flathub org.freedesktop.LinuxAudio.Plugins.Surge//20.08

this is cool. it connects audio using ALSA directly...

the problem as I suspected is the plugins have their own container and Bitwig can only navigate in two locations

User and its System... which is not the System's System...

Flatpak Surge is installed in

/var/lib/flatpak/runtime/org.freedesktop.LinuxAudio.Plugins.Surge/x86_64/20.08/active/files/

there is no way to navigate to this location as it is outside Flatpak Bitwig's container

in my opinion to get this to work... all plugins would have to be in the same container.... which defeats the purpose of this experiment as it would require all plugins within this container to use the same runtime...

possibly symbolic links could work but this is an ugly hack and not really a solution (in my opinion)

this Flatpak limitation would also render all plugins in /user/lib/vst3 and /usr/local/lib/vst3 outside of Flatpak Bitwig's reach

tried a symbolic link and no dice... I would conjecture that Flatpak Bitwig will only be able to detect Surge if and only if it is a local install and not Flatpak

not optimistic about Flatpak DAWS and Flatpak plugins being able to work together...

for example with WINE... all the plugins are either in the WINE folder or accessible to locations outside the WINE container... with Flatpak each plugin is in its own container... which is not in the DAW container... this is problematic

Is bitwig and surge sharing the same runtime?
Maybe you need to add a location for the plugin?

I'll look into it later.

Bitwig is located here

/var/lib/flatpak/app/com.bitwig.BitwigStudio/x86_64/stable/6267badf04b6977108cfb068cedc00b8dae9d57e80c671c0db488088f54f831b/files/

Surge is located here

/var/lib/flatpak/runtime/org.freedesktop.LinuxAudio.Plugins.Surge/x86_64/20.08/7464c5cd5222d0d0228a8788ad6ff3bbf625429b43d7d67f5479dccce1e70cdb/files/

as far as I understand how plugins work they have to run within the Host... I am really not sure how this is supposed to work with Flatpak... the plugins would have to be inside the Host's container... as plugins do not execute

I would think there would need to be a universal runtime layer or container... one that acts as a go between the DAW of choice and every other plugin in the Flatpak ecosystem as each and every one of them will be in their own unique Silo and locked down System/Container

noticed that you starred this

will try this and get back... which runtime is Bitwig? does it matter?

Perhaps unorthodox idea, but why not build tailored Wine to the DAW? Like Steam does for games.

I personally use Windows versions even of the plugins for which there is a Linux version, just because it management wise easiest way to get shit done (when producing music). Other options is to have ~/.wine, ~/.vst, ~/.vst2 and what not.

All platforms considered it would be cool if plugin industry would give a shot doing something similar Sun tried to do years back with PWI (Public Windows Interface). It was a failure but in audio plugin it could work out. I.e. standardize a subset of Windows API that plugins would use. That could even mean ABI compatibility for plugin binaries using the same ISA

I.e. instead of macOS, Windows and Linux version of a plugin you'd just have ARM, x86 etc. versions

Plugins are in the just huge calculators with an UI, pretty simple stuff, when you think it that way. I/O wise they are most of the part as simple as Minesweeper game.

Plugin API's do not need any change. What is needed is a standard PRE (Plugin Runtime Environment) :-)

@tank-trax:

there is no way to navigate to this location as it is outside Flatpak Bitwig's container

From flatpaked Bitwig's perspective flatpaked plugins are in /app/extentions/Plugins/vst3 and /app/extentions/Plugins/lxvst. Need to add those paths in Bitwig's Settings->Locations. Not configured by default. Those paths are internal container paths.

@abique:

Can a DAW from the system use them?
Can a DAW using a different runtime use them?

You cannot use flatpaked plugins from non flatpak application. Each plugin extends linuxaudio base runtime which in turn extends freedesktop.org runtime and they all see those paths I mentioned earlier within the container's filesystem. I think you can use those plugins in any DAW that is built against the runtime which extends freedesktop.org runtime, for example gnome runtime, KDE runtime, or your own custom runtime as long as that custom runtime extends freedesktop.org runtime and in flatpak manifest it adds org.freedesktop.LinuxAudio.Plugins extention.

I do not remember exact technical reason why that linuxaudio extention was needed, though. Github user @hfiguiere is the creator of that extension and the most active maintainer, probably he could explain all this much better than me.

I think it is going to work well with flatpak with traditional plugin loading. Let us see.

@tank-trax

it connects audio using ALSA directly...

Btw, when using pipewire (default on latest Fedora) you can also use JACK as audio subsystem in Bitwig without needing to have actual JACK installed (real JACK will not work in flatpaked apps)

You cannot use flatpaked plugins from non flatpak application. Each plugin extends linuxaudio base runtime
[...]
I do not remember exact technical reason why that linuxaudio extention was needed, though. Github user @hfiguiere is the creator of that extension and the most active maintainer, probably he could explain all this much better than me

Not exactly. The thing with extension is that they need:

1. an extension point which determine what they are and where there are in the container - it is also important the filesystem mount point exists even to build.
2. a SDK to be build against

Typically an extension is specific to an app. Like a plugin for GIMP.
In that case, as we know, it's not, it just implement an API. Still it needs the two. So that "BaseExtension" app is only used to:

1. specify the extension point (like the application using it do)
2. specify a SDK, in that case the freendesktop-sdk, used to build it.

Note that the extension doesn't specify the SDK. The application does. Freedesktop-sdk is the baseline for both the GNOME and KDE runtime, so either can be used by the app.
There is no check performed.

The BaseExtension is just an empty shell.

Just to add, when I say no check is performed, the only check is that the version specified in the app add-extension object is matched against the branch for each extension.

version: '20.08' will get the extension from the //20.08 branch. But here nothing prevent an extension from being build against a different runtime, but it's just looking for trouble.
versions can specify multiple versions, but there is currently only a 20.08 branch.

Also the runtime matter much less if is no build (ie packaging from binaries)

Btw, when using pipewire (default on latest Fedora) you can also use JACK as audio subsystem in Bitwig without needing to have actual JACK installed (real JACK will not work in flatpaked apps)

The current Flatpak for Bitwig should work with Pipewire when using JACK, provided that Pipewire on the host also work (Fedora 34 for example). Any further trouble is linked to Pipewire, one notable issue being setting real-time priority.
It failed when the package was reviewed because of the bug in the runtime that has since been addressed.