Last active
December 17, 2022 02:17
-
-
Save lemire/19186019db155401406f61ab7ded5ada to your computer and use it in GitHub Desktop.
How one might include simdutf in node js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AVX-512 is a collection of instruction sets. The AVX-512 kernel in simdutf relies on VBMI2, one of the AVX-512 | |
instruction sets. The simdutf library determines at runtime whether the processor supports VBMI2. If not, other (slower) | |
kernels are used, depending on the processor. | |
Not all compilers support the AVX-512 intrinsics that the simdutf library requires. Thus at compile-time, the library | |
checks for Visual Studio 2019 or better, or for the presence of the header 'avx512vbmi2intrin.h'. | |
At compile-time, it is thus assumed that if the compiler has access to 'avx512vbmi2intrin.h', then it can compile | |
the functions that are part of it and produce a VBMI2 code kernel. This should be a safe assumption. | |
We have that AVX-512 VBMI2 requires binutils 2.30 when compiling with GCC. Ubuntu 18.04 and up have binutils 2.30. | |
I believe that all Linux distributions that default on GCC 8 or better have at least binutils 2.30, so it may take a | |
non-standard system to mix GCC 8 with an older than 2.30 binutils. Furthermore, distributions with older binutils | |
are themselves old: Ubuntu 16.04 is no longer supported as of 2021. | |
GCC 8 was released in 2018, so after binutils 2.30 was released. | |
Similarly, nasm only supports AVX-512 VBMI2 starting with version 2.14, and LLVM requires 6.0 or better. | |
This conditions are more restrictive than the 'AVX-512' requirements that node seems to present at | |
https://github.com/nodejs/node/blob/cf748ad2b7f760e2df1bc8a811f887161b892534/BUILDING.md#supported-toolchains | |
I believe that this is for basic AVX-512 support. | |
So a solution might be to detect version of binutils that are older than 2.30 and to then disable AVX-512 VBMI2 support. | |
One could add in node/configure.py, something along these lines... | |
if (('gas_version' in variables and StrictVersion(variables['gas_version']) < StrictVersion('2.30')) or \ | |
('xcode_version' in variables and StrictVersion(variables['xcode_version']) < StrictVersion('10.0')) or \ | |
('llvm_version' in variables and StrictVersion(variables['llvm_version']) < StrictVersion('6.0')) or \ | |
('nasm_version' in variables and StrictVersion(variables['nasm_version']) < StrictVersion('2.14')) | |
o['defines'] += ['SIMDUTF_IMPLEMENTATION_ICELAKE=0'] | |
Note that there is probably no point in building AVX-512 with xcode since only one Apple macbook ever supported AVX-512 and | |
it is quite uncommon (few people have it). | |
Hopefully, "o['defines'] += ['SIMDUTF_IMPLEMENTATION_ICELAKE=0']" will ensure that simdutf.cpp is compiled with | |
the SIMDUTF_IMPLEMENTATION_ICELAKE=0 macro set. | |
If it happens, then chances are good that it will solve the issue. | |
You can it manually. | |
Grab https://github.com/lemire/docker_programming_station. You might clone it into ~/CVS/github/docker_programming_station | |
Make sure that docker is installed on your system and that it is an x64 docker (ARM wouldn't help). | |
Grab simdutf from https://github.com/simdutf/simdutf, put it | |
in ~/CVS/github/simdutf or some other convenient location, then go in simdutf/singleheader | |
and run "python amalgamate.py" or "python3 amalgamate.py". This will generate simdutf.h and simdutf.cpp. | |
While in simdutf/singleheader, run docker_programming_station/ubuntu16/run-docker-station bash. | |
This will start a bash shell starting in simdutf/singleheader. It is a relatively slow process since docker | |
needs to build the image (the first time). | |
E.g., if you put docker_programming_station at ~/CVS/github/docker_programming_station, | |
then move to ~/CVS/github/simdutf/singleheader and type | |
../../docker_programming_station/ubuntu16/run-docker-station bash | |
You will then be able to verify the issue... | |
~/CVS/github/simdutf/singleheader$ g++-8 amalgamation_demo.cpp | |
/tmp/ccJPftky.s: Assembler messages: | |
/tmp/ccJPftky.s:207774: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k1}' | |
/tmp/ccJPftky.s:208112: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k7}' | |
/tmp/ccJPftky.s:208595: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:208624: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k2}' | |
/tmp/ccJPftky.s:209174: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k5}' | |
/tmp/ccJPftky.s:209535: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:210015: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:210044: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:210606: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:210957: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:211453: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:211482: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:212084: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:212458: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:212951: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:212980: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k2}' | |
/tmp/ccJPftky.s:213535: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k1}' | |
/tmp/ccJPftky.s:213873: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k7}' | |
/tmp/ccJPftky.s:214356: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:214385: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k2}' | |
/tmp/ccJPftky.s:214935: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k5}' | |
/tmp/ccJPftky.s:215296: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:215776: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:215805: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:216406: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:216757: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:217253: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:217282: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:217884: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:218258: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:218751: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:218780: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k2}' | |
/tmp/ccJPftky.s:219774: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:220350: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:220542: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:221181: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:221757: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:221949: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:222697: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:223298: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k7}' | |
/tmp/ccJPftky.s:223515: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:224033: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:224897: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:225500: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:225719: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:226385: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k7}' | |
/tmp/ccJPftky.s:226988: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k2}' | |
/tmp/ccJPftky.s:227207: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:228004: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k3}' | |
/tmp/ccJPftky.s:228632: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:228876: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k4}' | |
/tmp/ccJPftky.s:229421: Error: no such instruction: `vpcompressw %zmm0,%zmm1{%k7}' | |
/tmp/ccJPftky.s:241789: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:242393: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k5}' | |
/tmp/ccJPftky.s:242405: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:242739: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k7}' | |
/tmp/ccJPftky.s:243343: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k6}' | |
/tmp/ccJPftky.s:243355: Error: no such instruction: `vpcompressb %zmm0,%zmm1{%k7}' | |
You can also verify that disabling icelake works... | |
~/CVS/github/simdutf/singleheader$ g++-8 amalgamation_demo.cpp -DSIMDUTF_IMPLEMENTATION_ICELAKE=0 | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment