AAP GUI extension
Android imposes strong restriction on how audio plugin UIs should be architected:
- Only one application can show the UI. Only either the DAW or the plugin Activity will be shown to users.
- For an Android application (in this case host/DAW), it is impossible to execute some code from other applications (for this case, plugins).
Flipping application activities is quite annoying, especially when they involve audio glitches.
Although, on the other hand, most of existing portable plugin GUIs (e.g. written with juce_gui_basics), it is still better with flipping activities, than nothing.
We will provide both solutions.
in-process GUI and cross-process GUI
There will be two kinds of GUI extensions for AAP:
- in-process GUI
- cross-process GUI
Cross-process GUI is based on Web technology (HTML/CSS/SVG/JS/wasm). In-process GUI can be anything.
Android apps cannot load external native code dynamically. Therefore it is impossible to run any GUI code that depends on native code. For example, Flutter, React Native, and Xamarin depends on native runtime. Ionic and React.js should be good to go.
(AAP itself is not going to provide any hosting support for those runtimes. Applications are free to provide common GUI runtime to make it happen. But at this state we have no idea on how such extensibility can be usable. Those non-platform frameworks usually package the runtime within user application and there is usually no clean way to separate runtime provider to the public, other than dev. runtime.)
Code separation for logic/audio and UI
Like any GUI application framework, AAP audio processors and its UI part should not be mixed.
LV2 achieves this code separation in clean way: the audio processor module and UI module are different shared library. The interaction between these components is achieved by port I/O. We follow this manner for cross-process UI too (we even force different programming architecture for cross-process).
There is a new metadata element with
<plugin> element in
<ui web="Android_Intent_Name_Web" activity="Android_Intent_Name_Activity" />
Android_Intent_Name_Web, AAP host queries the service on the Android device (or emulator). For
Android_Intent_Name_Activity it would just launch the specified activity, which can be either within the same app or in another application.
The web content provider is then used to provide "GUI package" which is a set of HTML files packaged within a zip.
Both UI styles are optional; a plugin can support only Web, or only activity, or both (or none).
AAP GUI package contents
The GUI zip package contains
index.html which is loaded into an Android
WebView for the plugin. anything else is optional.
AAPDescriptor. It is an
Object that comes with these function members:
extensionURI : String: the extension URI
initialize(): The host must invoke this function when it initializes the UI.
processEvent(port: Number, data: UInt8Array, offset: Number, length: Number): The host invokes this function to notify the plugin about plugin/host events (e.g. parameter changes).
getExtension(uri: String): The host may call this function to get an extension object for the argument URI.
onShow(): The host must invoke this function whenever it shows the plugin UI beforehand (possibly from the hidden state). It is invoked after
onHide(): The host must invoke this function whenever it hides (but not destry) the plugin UI afterward. It is invoked before
cleanup(): The host must invoke this function when it disposes the UI.
The host will provide a global object called
pluginId: String: the plugin ID.
instanceId : String: the plugin instance ID.
pluginContext : String: the plugin context.
write(port: Number, data: UInt8Array, offset: Number, length: Number): The user can call this function to write data to the specified port.
getExtension(uri: String): The user can call this function to retrieve extensions
Both Activity-based UI in another application and Web-based UI use binder based messaging to dispatch port I/O, just like host and plugin do.
Porting GUI to AAP cross-process GUI
mod-ui is the closest solution to this idea.
I came up with similar solution called aria2web which imports SFZ ARIA extension to HTML5, but mod-ui could bring tighter integration with our LV2 backend.
Not a current choice ATM.
There is juce_emscripten effort, but it depends on
Atomics that is not supported on Android yet.
If Android WebView started that, aap-juce still needs some way to separate UI and non-UI part, and inject audio processing part.
GL based technology (libGL, GLFW)
They might be options. It's up to whether they can build using emscripten.
pugl is (as long as the README says) a GUI framework based on thin dependencies on platform-specific APIs, and it supports X11, so it is likely able to build pugl for wasm.