Skip to content

Instantly share code, notes, and snippets.

@mkolod
Created December 17, 2019 17:31
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mkolod/7fd39a9eef11ba4b6187a29e546e65f2 to your computer and use it in GitHub Desktop.
Save mkolod/7fd39a9eef11ba4b6187a29e546e65f2 to your computer and use it in GitHub Desktop.
Preloading shared lib (CUDA driver) before main runs
#include <iostream>
#include <iomanip>
#include <dlfcn.h>
using std::cout;
using std::cerr;
using std::endl;
void* h = []() {
void* handle = static_cast<void*>(dlopen("libcuda.so.1", RTLD_NOW | RTLD_GLOBAL));
if (handle) {
cout << "Opened libcuda.so.1 (real driver)" << endl;
} else {
cout << "CUDA driver does not exist. Loading stub." << endl;
handle = static_cast<void*>(dlopen("libcuda.so", RTLD_NOW | RTLD_GLOBAL));
if (handle) {
cout << "Opened stub." << endl;
} else {
cerr << "Stub could not be found." << endl;
}
}
char* error = dlerror();
if (error != nullptr) {
cerr << "Error in static init: " << error << endl;
} else {
cout << "Static load successful." << endl;
}
return handle;
}();
int main(int argc, char **argv) {
cout << "main()" << endl;
cout << "Is h a nullptr? " << std::boolalpha << (h == nullptr) << endl;
void* test = static_cast<void*>(dlopen("libcuda.so.1", RTLD_NOLOAD | RTLD_NOW | RTLD_GLOBAL));
if (test != nullptr) {
cout << "Library previously loaded" << endl;
} else {
cerr << "Failure: library not previously loaded" << endl;
}
}
@mkolod
Copy link
Author

mkolod commented Dec 17, 2019

Preloading shared lib (CUDA driver) before main() runs.

Experiments:

  1. Running with driver present (libcuda.so.1, which symlinks the actual driver so, e.g. /usr/lib/x86_64-linux-gnu/libcuda.so.430.64).

$g++ dlopen_driver.cpp -ldl -std=c++17 -o dlopen_driver && ./dlopen_driver
Opened libcuda.so.1 (real driver)
Static load successful.
main()
Is h a nullptr? false
Library previously loaded

  1. Running without the driver present, but with the driver stub present (libcuda.so at /usr/local/cuda/lib64/stubs or elsewhere on LD_LIBRARY_PATH):

$g++ dlopen_driver.cpp -ldl -std=c++17 -o dlopen_driver && ./dlopen_driver
CUDA driver does not exist. Loading stub.
Opened stub.
Static load successful.
main()
Is h a nullptr? false
Library previously loaded

Note: the stub succeeded and was seen as adequate for not having to load libcuda.so.1 again, because the SONAME of both the stub and of the real library is the same (libcuda.so.1):

Real driver:

$ readelf -d /usr/lib/x86_64-linux-gnu/libcuda.so.1 | grep SONAME
0x000000000000000e (SONAME) Library soname: [libcuda.so.1]

Stub:

readelf -d /usr/local/cuda/lib64/stubs/libcuda.so | grep SONAME
0x000000000000000e (SONAME) Library soname: [libcuda.so.1]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment