Skip to content

Instantly share code, notes, and snippets.

What would you like to do?

Issues with binaries for SmartOS users

Description of the problem

Node v4.x.y binaries have been built with compilers at versions greater than the ones used to build packages in the LTS pkgsrc repositories. They've been built with gcc 4.9 and gcc 4.8, when the compiler used to build packages from the 2014Q4 pkgsrc repository is gcc 4.7.3.

As a consequence, these node v4.x.y binaries downloaded from have requirements on versions of the C++ runtime that are greater than the C++ runtime provided by, for instance, the platform C++ runtime and the C++ runtime used by most (all?) 2014Q4 pkgsrc packages.,

Thus, one would think that, to run these node binaries, installing the runtime corresponding to these more recent compilers would be needed.

However, as described in this document, it turns out that these binaries run fine on a system with only the platform C++ runtime and the C++ runtime packaged with gcc 4.7's libraries.

It seems that this behavior might be the sign of a bug in the runtime linker that could become a bigger problem if these binaries stop working accidentally at some point.

Detailed description of the problem for node v4.2.2

A node v4.2.2 binary downloaded from runs fine on a SmartOS VM using a 2014Q4 pkgsrc repository (the current LTS version) and a 14.4.2 image (also the current LTS version):

[root@dev ~/node-1]# uname -a
SunOS dev 5.11 joyent_20150219T102159Z i86pc i386 i86pc Solaris
[root@dev ~/node-1]# cat /etc/motd 
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 14.4.2)

[root@dev ~/node-1]# cat /etc/pkgsrc_version 
release: 2014Q4
architecture: x86_64
[root@dev ~/node-1]# ~/node-v4.2.2-sunos-x86/bin/node -e 'console.log(process.versions)'
{ http_parser: '2.5.0',
  node: '4.2.2',
  v8: '',
  uv: '1.7.5',
  zlib: '1.2.8',
  ares: '1.10.1-DEV',
  icu: '56.1',
  modules: '46',
  openssl: '1.0.2d' }
[root@dev ~/node-1]#

However, ldd indicates that the node v4.2.2 binary cannot find a C++ runtime that matches its requirements:

[root@dev ~/node-1]# ldd -v ~/node-v4.2.2-sunos-x86/bin/node | grep CXX (GLIBCXX_3.4.15) =>       (version not found) (CXXABI_1.3) =>   (version not found)
[root@dev ~/node-1]#

Running that binary with LD_DEBUG=all:

55472:;  needed by /root/node-v4.2.2-sunos-x86/bin/node
55472: find; searching
55472:  trying path=/opt/local/gcc48/i486-sun-solaris2.11/lib/./
55472:  trying path=/opt/local/gcc48/lib/./
55472:  trying path=/lib/
55472:  trying path=/usr/lib/
55472: file=/usr/lib/  [ ELF ]; generating link map
55472:                 GLIBCXX_3.4.15
55472:                 CXXABI_1.3

we find that it uses the C++ runtime at /usr/lib/

[root@dev ~/node-1]# ls -l /usr/lib/
lrwxrwxrwx 1 root root 19 Feb 19  2015 /usr/lib/ ->
[root@dev ~/node-1]#

That runtime is at version 6.0.13, which corresponds, according to GCC's ABI Policy and Guidelines, to GCC 4.4.2 and ABI versions GLIBCXX_3.4.13 and CXXABI_1.3.3.

This runtime is clearly older than the runtime version required by node v4.2.2 (GLIBCXX_3.4.15), which corresponds according to that same document to GCC 4.6.0. It's not clear why, in these conditions, a v4.2.2 binary would run on such a platform.

According to preliminary investigations by @rmustacc, it might be due to a bug in the linker/loader. Running node v4.2.2 should actually fail with an error message indicating that the requirement on version GLIBCXX_3.4.15 of the C++ runtime is not met.

Detailed description of the problem for node v4.0.0 to node v4.2.1

The situation is different for node versions between v4.0.0 and v4.2.1. These versions have been built with g++ 4.9, whereas node v4.2.2 was built with g++ 4.8. What does it change in practice?

[root@dev ~/node-1]# ldd -v ~/node-v4.2.1-sunos-x86/bin/node | grep CXX (GLIBCXX_3.4.20) =>       (version not found) (CXXABI_1.3) =>   (version not found)
[root@dev ~/node-1]#

These node binaries depends on even newer versions of the C++ runtime: GLIBCXX_3.4.20 instead of GLIBCXX_3.4.15`.

The reason for that is g++ 4.9 uses __throw_out_of_range_fmt when throwing exceptions instead of __throw_out_of_range, and that adds a dependency on version 3.4.20 of the C++ runtime. Here's the output of nm on Linux that demonstrates that:

jgilli@ubuntu:~/dev/node$ nm ~/node-4.x-g++-4.9 | grep 3.4.20 | c++filt 
00000000013a4420 D TLSv1_enc_data
00000000013b4f20 D v8::internal::FLAG_turbo_osr
00000000013b4e20 D v8::internal::FLAG_log_instruction_period
                 U std::__throw_out_of_range_fmt(char const*, ...)@@GLIBCXX_3.4.20
00000000013c4120 b no_eventfd.9277
00000000013c4220 b process_title
00000000013a4220 d scsv.19260
jgilli@ubuntu:~/dev/node$ nm ~/node-4.x-g++-4.8 | c++filt | grep throw_out_of_range
                 U std::__throw_out_of_range(char const*)@@GLIBCXX_3.4

So node SmartOS binaries at versions between v4.0.0 and v4.2.1 should also not be able to run on a SmartOS 14.4.2 image and using a 2014Q4 pkgsrc repository, as indicated by ldd's output (note the version not found messages):

[root@dev ~/node-1]# ldd -v ~/node-v4.2.1-sunos-x86/bin/node | grep CXX (GLIBCXX_3.4.20) =>       (version not found) (CXXABI_1.3) =>   (version not found)
[root@dev ~/node-1]#

However, they do run, probably for the same reason that node v4.2.2 runs: by accident and due to a bug in the linker/loaded:

[root@dev ~/node-1]# ~/node-v4.2.1-sunos-x86/bin/node -p 'process.versions'
{ http_parser: '2.5.0',
  node: '4.2.1',
  v8: '',
  uv: '1.7.5',
  zlib: '1.2.8',
  ares: '1.10.1-DEV',
  icu: '56.1',
  modules: '46',
  openssl: '1.0.2d' }
[root@dev ~/node-1]#

Next steps

This section describes some of the potential next steps we could take to determine exactly what's going on, and to eventually fix the problem.

Investigating the runtime linker issue further

It's not clear yet what the linker/loader bug to which this document alludes is. It would be great if we could investigate it further to come up with a definitive answer. Who would have some time to do that, or help me investigate that issue in the next couple weeks?

Fixing the problem

If it turns out to be an actual bug in the runtime linker, fixing that bug could cause more harm than good, as it would probably make node v4.x binaries that do run today on some SmartOS platforms not run anymore.

Not fixing that bug might trigger some subtle bugs because node binaries are built with the assumption that they run with a more recent C++ runtime.

In any case, it seems that we may want to let users for node v4.x.y run node with the proper C++ runtime on their SmartOS platforms. Following are a few potential solutions for doing that.

Not providing SmartOS binaries on, and asking users to only use pkgsrc packages

Versions of node packaged in SmartOS pkgsrc repositories do not have that C++ runtime dependency problem as they depend on the appropriate specific C++ runtime versions. So for instance, the nodejs-v4.2.1 package includes a node binary built by g++ 4.8 and it depends on a package provinding g++ 4.8's runtime itself. As a result, all dependencies and requirement are met:

[root@dev ~/node-1]# pkgin install nodejs-4.2.1
calculating dependencies... done.

nothing to upgrade.
3 packages to be installed (0B to download, 94M to install):

gcc48-libs-4.8.3nb3 nodejs-4.2.1

proceed ? [Y/n] 
downloading packages...
installing packages...
installing gcc48-libs-4.8.3nb3...
installing nodejs-4.2.1...
pkg_install warnings: 0, errors: 0
reading local summary...
processing local summary...
updating database: 100%
marking nodejs-4.2.1 as non auto-removable
[root@dev ~/node-1]# node --version
[root@dev ~/node-1]# which node
[root@dev ~/node-1]# ldd `which node` =>     /opt/local/lib/ =>         /lib/64/ =>  /lib/64/ =>        /lib/64/ =>   /lib/64/ =>    /lib/64/ =>      /lib/64/ =>        /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/ =>     /lib/64/ =>         /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/ =>       /lib/64/ =>     /lib/64/ =>         /opt/local/gcc47/x86_64-sun-solaris2.11/lib/amd64/ =>    /lib/64/ =>    /lib/64/
[root@dev ~/node-1]#

As we can see from ldd's output, the C++ runtime used by that node binary is at /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/, which is not the default C++ runtime that comes with the platform or the dependency of packages found in the 2014Q4 (LTS) pkgsrc repository.

Removing SmartOS standalone binaries from would have the benefit of requiring SmartOS users to always use packages from pkgsrc to get their binaries, and guarantee that the dependencies on the C++ runtime are met.

Building node with g++ 4.7

Another potential solution could be to build node v4.x.y with g++ 4.7, because g++ 4.7 is the default compiler version for the current LTS pkgsrc repository. However, it doesn't seem to be an avenue that the Node.js project is likely to consider since:

  1. V8 explicitly states that it requires g++ 4.8 and later to build.
  2. Some previous experiments to build recent versions of node with g++ 4.7 failed.

So it seems that the only possible approach to solve that problem is to provide a C++ runtime that is compatible with g++ 4.8' runtime on SmartOS machines.

Making sure that the proper C++ runtime is available on most SmartOS machines

Currently, the platform C++ runtime is at version 6.0.13 (which corresponds to GCC 4.4.2), and it seems most packages have a dependency on gcc 4.7.3's runtime, so that means any binary built with gcc 4.7.3 would also find a proper runtime.

I'm wondering if it would make sense to provide more recent runtimes either in images, or in the platform's /usr/lib.

Evaluate a way to get things like Red Hat Developer's Toolset

The way that the Node.js project currently solves the problem of providing binaries built with recent compilers that run on older Linux systems is by using the RedHat Developer Toolset. It's a toolchain that bundles newer features of the C/C++ runtime in the binaries themselves so that these binaries can run on systems with older runtimes. See from slide 26 for further details.

It seems that getting something similar to that to work would be fairly complex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment