Extern functions.md

Binding an external function definition to a dynamic/shared library

How it works now:

User defines an external function:

noreturn ExitProcess(u32 uExitCode);
void main() {
    ExitProcess(0);
}

Then program is compiled like: vox main.vx C:\Windows\System32\kernel32.dll

The compiler then reads main.vx and C:\Windows\System32\kernel32.dll files.

Compiler parses the kernel32.dll file and reads all exported functions into the hashmap of external symbols.

When compiling main.vx module, compiler sees ExitProcess external function definition and looks into external symbol hashmap and resolves to the symbols that was registered from kernel32.dll.

When executable is generated compiler creates import table and inserts an entry for kernel32.dll library with single imported symbol ExitProcess.

Bads:

1. All external symbols are dumped into the same namespace, which has potential for collisions.
2. Compiler needs to read potentially big file into memory, which takes time.
3. User needs to provide library path on CLI.
4. When cross-compiling you need to have access to kernel32.dll file.

Goods:

1. You can provide any library or host module that exports ExitProcess and will resolve by name.
2. No attributes needed

Now we can see all the interesting requirements:

1. Namespaced name lookups (each external module has its own namespace and external symbol specifies namespace for lookup)
2. Rebindability (when module name specified in the source code may differ from file name)
3. Cross-platformity (Ideal: no .def files need to be distributed with the source; semi-ideal: .def files)
4. Performance (Ideal: no file reads needed; semi-ideal: read file that only contains exported symbol names (.def))
5. Reusing the same mechanism to bind not only to dynamic/shared library, but to any external module
6. Minimizing CLI switches (by default file name should be assumed the same as external module name, since it is the most common scenario with system libraries)

Why is rebindability useful? .dll/.so file name can be different between the projects using the module that defines external function. Also it gives possibility to reuse the Vox module in embedded context.

Proposed solution:

@extern(module, "kernel32")
noreturn ExitProcess(u32 uExitCode);

• Use an attribute to mark external function
• That attribute must specify the name of external module. External module shall be a string, because file name may contain chars that are invalid in the identifiers
• Default: name of external module is used as a name of .so/.dll library when compiling as an executable. .dll/.so` suffix will be added as (and if) required by the target plaform.
• CLI switch is introduced to change the default name of external module to a custom one --externModule=<external_mod>:<file_name>
• If non-namespaced lookups are forbidden, then all external functions must have the attribute attached
• External module name must not contain .so/.dll suffix, as the same definition may be reused on different platforms and in embedded environment where it is bound to host defined module.
• No dll is passed on the CLI
• In case user decides to name function diffently that it is exported in the external module another attribute can be specified. @mangle("<name_to_lookup>")

@mangle("ExitProcess")
@extern(module, "kernel32")
noreturn exit(u32 uExitCode);

Now the example can look like:

@extern(module, "kernel32")
noreturn ExitProcess(u32 uExitCode);
void main() {
    ExitProcess(0);
}

And compiled with vox main.vx.

Compiler will create import entry for ExitProcess in kernel32.dll library.

To rebind the kernel32 external module user can do vox main.vx --externModule=kernel32:custom.dll

Edit: After thinking more about the chosen solution, I found 1 case where passing .dll file to the compiler is benefitial.

When compiler has a list of symbols from the dll it can verify that all references from @extern(module, "mod_name") map to an existing symbol within the dll. Of course this means that exactly the same dll file must be loaded at load-time.

It's a small benefit, but it can be useful when the bindings are initially created to verify that everything is correct during build-time. Adding such an option on top of extern(module) shouldn't be too hard.