Skip to content

Instantly share code, notes, and snippets.

@atsushieno

atsushieno/aap-gui-extensibility.md Secret

Last active Jul 3, 2020
Embed
What would you like to do?

AAP GUI extension

Android imposes strong restriction on how audio plugin UIs should be architected:

  • Only one application can show the UI. Only either the DAW or the plugin Activity will be shown to users.
  • For an Android application (in this case host/DAW), it is impossible to execute some code from other applications (for this case, plugins).

Flipping application activities is quite annoying, especially when they involve audio glitches.

Although, on the other hand, most of existing portable plugin GUIs (e.g. written with juce_gui_basics), it is still better with flipping activities, than nothing.

We will provide both solutions.

in-process GUI and cross-process GUI

There will be two kinds of GUI extensions for AAP:

  • in-process GUI
  • cross-process GUI

Cross-process GUI is based on Web technology (HTML/CSS/SVG/JS/wasm). In-process GUI can be anything.

Android apps cannot load external native code dynamically. Therefore it is impossible to run any GUI code that depends on native code. For example, Flutter, React Native, and Xamarin depends on native runtime. Ionic and React.js should be good to go.

(AAP itself is not going to provide any hosting support for those runtimes. Applications are free to provide common GUI runtime to make it happen. But at this state we have no idea on how such extensibility can be usable. Those non-platform frameworks usually package the runtime within user application and there is usually no clean way to separate runtime provider to the public, other than dev. runtime.)

Code separation for logic/audio and UI

Like any GUI application framework, AAP audio processors and its UI part should not be mixed.

LV2 achieves this code separation in clean way: the audio processor module and UI module are different shared library. The interaction between these components is achieved by port I/O. We follow this manner for cross-process UI too (we even force different programming architecture for cross-process).

AAP metadata

There is a new metadata element with <plugin> element in aap_metadata.xml

<ui web="Android_Intent_Name_Web" activity="Android_Intent_Name_Activity" />

Where both Android_Intent_Name_* are app_package_name/content_provider_name. For Android_Intent_Name_Web, AAP host queries the service on the Android device (or emulator). For Android_Intent_Name_Activity it would just launch the specified activity, which can be either within the same app or in another application.

The web content provider is then used to provide "GUI package" which is a set of HTML files packaged within a zip.

Both UI styles are optional; a plugin can support only Web, or only activity, or both (or none).

AAP GUI package contents

The GUI zip package contains index.html which is loaded into an Android WebView for the plugin. anything else is optional.

There is an Javascript interface called AAP Plugin Access API that is hooked by the AAP host WebView. Some Polyfill for that interface would be provided for use in anywhere else.

JavaScript AAP Plugin Access API

The HTML must come with a global Javascript object called AAPDescriptor. It is an Object that comes with these function members:

  • extensionURI : String : the extension URI
  • initialize() : The host must invoke this function when it initializes the UI.
  • processEvent(port: Number, data: UInt8Array, offset: Number, length: Number) : The host invokes this function to notify the plugin about plugin/host events (e.g. parameter changes).
  • getExtension(uri: String) : The host may call this function to get an extension object for the argument URI.
  • onShow() : The host must invoke this function whenever it shows the plugin UI beforehand (possibly from the hidden state). It is invoked after initialize() too.
  • onHide() : The host must invoke this function whenever it hides (but not destry) the plugin UI afterward. It is invoked before cleanup() too.
  • cleanup() : The host must invoke this function when it disposes the UI.

The host will provide a global object called AAPHostContext which comes with these members, that users can call from the Javascript UI code:

  • pluginId: String : the plugin ID.
  • instanceId : String : the plugin instance ID.
  • pluginContext : String : the plugin context.
  • write(port: Number, data: UInt8Array, offset: Number, length: Number) : The user can call this function to write data to the specified port.
  • getExtension(uri: String) : The user can call this function to retrieve extensions

Binder messaging

Both Activity-based UI in another application and Web-based UI use binder based messaging to dispatch port I/O, just like host and plugin do.

Porting GUI to AAP cross-process GUI

mod-ui

mod-ui is the closest solution to this idea.

I came up with similar solution called aria2web which imports SFZ ARIA extension to HTML5, but mod-ui could bring tighter integration with our LV2 backend.

JUCE

Not a current choice ATM.

There is juce_emscripten effort, but it depends on Atomics that is not supported on Android yet.

If Android WebView started that, aap-juce still needs some way to separate UI and non-UI part, and inject audio processing part.

GL based technology (libGL, GLFW)

They might be options. It's up to whether they can build using emscripten.

pugl

pugl is (as long as the README says) a GUI framework based on thin dependencies on platform-specific APIs, and it supports X11, so it is likely able to build pugl for wasm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.