shirakaba/How NativeScript's JS->native bindings work.md

## How NativeScript's JS->native bindings work.md

      
    Raw
  

              How NativeScript's JS->native bindings work.md
            
          
    [Draft] Deep-dive: How NativeScript's JS->native bindings work

What is NativeScript?

NativeScript is a JavaScript runtime that allows you to write an iOS or Android app entirely in TypeScript or JavaScript with no compromises. That is to say, it provides you with the same level of native access that writing the app in Obj-C/Swift (for iOS) or Java/Kotlin (for Android) does. Not just JSON-serialisable data types like NSString, but all data types are supported. To illustrate what it's all about, here's how you would get the battery level using NativeScript (the below snippet is entirely JS code!):
import { isIOS, isAndroid, Application } from "@nativescript/core";

/**
 * @returns the battery level (at least for iOS), as a number between 0.0 and 1.0.
 * @platform iOS, or Android API level 21.
 */
function getBatteryLevel(){
    if(isIOS){
        UIDevice.currentDevice.batteryMonitoringEnabled = true;
        const batteryLevel = UIDevice.currentDevice.batteryLevel;
        UIDevice.currentDevice.batteryMonitoringEnabled = false;
        return batteryLevel;
    } else if(isAndroid){
        const bm = Application.android.context.getSystemService(android.content.Context.BATTERY_SERVICE);
        return bm.getIntProperty(android.os.BatteryManager.BATTERY_PROPERTY_CAPACITY);
    }
    throw "Platform not yet supported!";
}
... Isn't that magical? 🧙‍♂️
You'll see that these JS calls map one-to-one to the equivalent native APIs, and they're synchronous, just like the APIs themselves. See Apple's Obj-C docs for UIDevice.currentDevice.batteryMonitoringEnabled and UIDevice.currentDevice.batteryLevel, for example.
Now, these were pretty simple API calls, but NativeScript can do just about everything the Obj-C and Java runtimes can do; for a real showcase, see my code for building an entire SpriteKit game in JS, which does things like extending native classes, implementing delegates, and passing around functions. And that's not even to mention its support for multi-threading, its APIs for managing the UI and responding to the app life cycle, and more – really, the sky's the limit!
Native API access for all!

Being able to access native APIs without leaving the JS context is incredibly empowering as a developer. It means you only have to master one IDE, one build system, and can develop rapidly due to hot reloading. And for me personally, I much prefer writing TS/JS code because the tooling is simply better (e.g. the language service for checking TS code is far faster than SourceKit is for Obj-C/Swift).
It feels like the NativeScript experience is something we should strive to have in all app development frameworks. And indeed, we in the NativeScript Technical Steering Committee have been working on how we can join forces with other frameworks, with our setup for creating a Capacitor app and calling out to NativeScript being our first success. As React Native supports V8 and JSC runtimes just like NativeScript does, it's also been an area of interest which we've been dabbling in. I gauged interest in this kind of native access through my react-native-native-runtime project, and also built react-native-nativescript-runtime as a proof-of-concept of dispatching work from React Native to NativeScript, the main issue being the tricky installation process (Xcode build settings management is very inflexible). We've continued to work on this aspect, with react-ns-embed being the latest effort to imagine how it could be integrated simply by installing a Cocoapod.
Wishing to streamline the process even further, I felt I needed to get more familiar with how NativeScript works at its lowest level to see whether any build steps could be simplified and whether any boilerplate could be removed. So please join me on my deep-dive into how NativeScript's JS->native bindings work!
NativeScript's architecture

First up, let's review NativeScript's architecture. To me, it essentially consists of three parts:

The metadata generators: Tools for generating metadata that expresses all the native APIs available to be bound (one for iOS, and another for Android);
The "iOS/Android runtimes": Forks of the V8 and JSC JS engines that are capable of generating bindings from that metadata to create a JS runtime for the given platform (I'll be focusing on the V8 iOS runtime);
NativeScript Core: A library of high-level APIs based on those bindings, allowing you to easily e.g. show an alert, render a text field, or manage the app lifecycle, etc. using ergonomic APIs that are also cross-platform, allowing you to develop both an iOS and Android app using the same code. This is everything you need to build an app end-to-end, but may be redundant if you're just using NativeScript alongside another cross-platform framework.

For this deep-dive, I'll be focusing on the first two topics.
What exactly is a JS->native binding?

The JS language alone doesn't actually give you all that much (see the latest ECMAScript language specification), so JS runtimes typically supplement the ECMAScript APIs with additional APIs from the platform (native APIs). For example, the browser adds various Web APIs like DOM, and Node.js adds many backend APIs like process. Often, these APIs access platform functionality (e.g. both the browser and Node.js can access the file system), and thus, at some point in the implementation, the JS engine needs to call a native API. To call a native API from the JS context, a binding must be established. How this is done depends on the JS runtime/engine.
How does one make JS->native bindings in each different JS runtime/engine?

It's invaluable to compare approaches, so let's do so:

In Node.js, one can technically access arbitrary native APIs from userland via child process; though setting up a true binding (e.g. representing native data types in JS) is done via C++ addons. There are three approaches for implementing C++ addons, and one of them, being based on V8's API, is relevant to V8 as well.
In the browser, the only way to bind new native APIs is by forking the source code.
In JavaScriptCore, one can set up bindings from userland using a few runtime APIs, or by forking the source code.
In React Native, regardless of the JS engine used, one can set up bindings from userland using a JSON bridge, though are limited to exchanging JSON-serialisable data. The new JSI API (not yet documented) allows representation of native data types in JS, however.
In NativeScript, metadata for all of your public userland native APIs, and all SDK APIs, are automatically generated at build-time and bound at runtime (we'll detail how later). To create a convenience function in JS that calls native APIs under-the-hood, you can either write it all in JS (as with our battery level example earlier) or use the plugins API if you'd prefer to handle the native APIs using native code like Obj-C or Java (e.g. because it allows you to adapt existing work).

So NativeScript is unique in that it allows you to call any native API and represent any native value from JS without having to write native code. Crucially, this means that you can access arbitrary native functionality without even recompiling the app, simply hot reloading as Webpack watches for code changes.
How does the NativeScript metadata generator tool work?

For this section, I'll focus on iOS, as that's my specialty.
In short, the tool reads all the public headers that go into your app, generates a binary metadata file describing the contents of all those headers. That metadata is included in your NativeScript app, and is read in at startup. As the app reads in all the items in the metadata, it sets up the relevant JS->native bindings.
In typical usage:

It searches for headers using clang's HeaderSearch API, and traverses them using clang's RecursiveASTVisitor, skipping over any modules that the user specified to be blocklisted in the metadata filtering options. For each included clang declaration, it runs the MetaFactory::create function, which determines what kind of declaration it is (enum, interface, etc.) and thus how to create an appropriate Meta class instance from it.
Once it has a list of all these Meta instances, it does a bit of cleanup (e.g. removing of duplicate members).
Next, it serialises the Meta objects to binary using this algorithm.
It can also optionally generate TypeScript definitions, and dump module maps.

What does this metadata look like?

The metadata that actually goes into your app is a binary file (specifically formatted a section of an object file to be handed over to the linker, ld), so it's not human-readable; but if we tell the tool to output YAML format, then it generates a file per module, like this ARKit.yaml file seen in the expected test output for the iOS metadata generator. Do please visit that link, because it's fascinating and I can't tell the whole story just by making small excerpts of it.
The metadata details everything the header details, such as signatures of methods, instance methods of classes, enum values, etc., as well as other information like where to find the given header.
When in the build process is the NativeScript metadata generator tool run?

The build process for any NativeScript app historically consists of four steps:

pre-build: Run the nativescript-pre-build shell script, which sets up many environment variables.
post-build: Run the nativescript-post-build shell script, which simply calls another script to strip all non-valid architectures from dynamic libraries in the app's Frameworks directory. I've not noticed this being done in other app development frameworks, so I'm not too sure what it's there for. Maybe just to reduce some space on disk during development?
pre-link: I think this actually just a no-longer-populated placeholder script, as it only consists of set -e.
link: The LD and LDPLUSPLUS build settings in Xcode are both set as $SRCROOT/internal/nsld.sh. These are not documented Xcode build settings, but word on the street is that they configure the linker (the default LD setting would be /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang). nsld.sh is merely a wrapper around the default linker; it's a script that calls the metadata generator to generate the metadata binary file (e.g. metadata-x86_64.bin) before passing control over to the default linker. And due to the OTHER_LDFLAGS build setting having been set to include the flags -sectcreate __DATA __TNSMetadata $(CONFIGURATION_BUILD_DIR)/metadata-$(CURRENT_ARCH).bin, our metadata binary file gets linked into the app as an object file section (see the -sectcreate CLI flag).

If we could get rid of all these changes to build settings, it would greatly improve the prospects of adopting NativeScript into existing projects.
How does the NativeScript runtime set up bindings from the generated metadata?

Supposing we've built our app, and have all the metadata linked, let's see how it's employed at runtime to set up bindings.

First, the app needs to get a reference to the metadata (remember that it's presently linked into the app as an object file section, so we need to retrieve it from memory). I've seen this done by two approaches. One, inline assembly:
extern char startOfMetadataSection __asm("section$start$__DATA$__TNSMetadata");
void* metadataPtr = &startOfMetadataSection;
... and two, by mach-o section lookup:
#import <mach-o/getsect.h>

static const struct mach_header_64 *mhExecHeaderPtr = &_mh_execute_header;
extern void* runtimeMeta(){
    NSString *sectname = @"__TNSMetadata";
    NSString *segname = @"__DATA";
    unsigned long size;
    void* meta = getsectiondata(&_mh_execute_header, [segname cStringUsingEncoding: NSUTF8StringEncoding], [sectname cStringUsingEncoding: NSUTF8StringEncoding], &size);
    return meta;
}
Note how both approaches refer to the section name and segment header that were specified along with the -sectcreate flag.
My understanding is that, due to the metadata being linked into the app as an object file segment, there's no deserialisation step necessary; I think it's stored in memory as a list of Meta class instances. At least, I can't see where the deserialisation step happens otherwise. The journey from there, as far as I can follow it, is:

Once you have a pointer to the metadata, you set it into a Config and initialise a NativeScript instance with it (as per the standard AppDelegate boilerplate). Internally, the NativeScript instance then sets that metadata pointer into a RuntimeConfig object before calling Runtime::Initialize(), which under-the-hood calls MetaFile::setInstance(RuntimeConfig.MetadataPtr). This sets the singleton metaFileInstance declared inside Metadata.mm, allowing various consumers, such as ArgConverter and MetadataBuilder to access the MetaFile by calling MetaFile::instance().
Next, Runtime::Init is called, and a bunch of methods on various classes are invoked. I'm not quite able to follow it all, but I can see what looks like data type marshalling happening inside ArgConverter and class construction happening inside ClassBuilder. I was interested to see that ClassBuilder uses Obj-C runtime helpers like class_addMethod to construct classes from the metadata, and binds every aspect of the class to JS via V8 APIs (likely those C++ addons APIs mentioned earlier). There is also clearly some important stuff going on in MetadataBuilder. Beyond Obj-C runtime helpers, I also saw some usage of dlsym, a dynamic linker API. There is really quite a lot to the NativeScript/runtime folder, though – more than I can hope to summarise!


... And this is as far as I've got! I want to continue digging until I have a real understanding of the flow. Once I've sorted it all out, I hope to explain it more smoothly.
Conclusion

NativeScript has traditionally been used as an end-to-end framework, but there's no reason it couldn't be used as a handy JS->native runtime for an app made using a different framework like React Native (or even a fully native app!). To make this viable, we're working on improving its embeddability, which requires thorough understanding of how it works at a low level to see if any long-established design patterns can be improved upon. To that end, we've done a deep-dive into how it works at the low-level.
In our deep-dive focusing on iOS and V8, we learned how the metadata is generated as a binary file (in the format of a mach-o object file section) at link-time using clang APIs, allowing the app to save time over reading a metadata file and deserialising it. The generation of bindings happens at runtime, using Obj-C runtime helpers and some dynamic linker (dlsym) APIs to reproduce the data mentioned in the metadata and then make bindings into V8 using V8 APIs.
As mentioned, I'll overhaul the section on bindings once I've got a more coherent story to tell, at which point I look forward to releasing the final draft of this article!