Script to build gcc with OpenMP offloading to Nvidia devices (via nvptx)

build-gcc-offload-nvptx.sh

#!/bin/bash

#
# Build GCC with support for offloading to NVIDIA GPUs.
#

set -o nounset -o errexit

# Location of the installed CUDA toolkit
cuda=/usr/local/cuda


# directory of this script
MYDIR="$( cd -P "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

work_dir=$MYDIR/gcc-offload
install_dir=$work_dir/install

rm -rf $work_dir

# Build assembler and linking tools
mkdir -p $work_dir
cd $work_dir
git clone https://github.com/MentorEmbedded/nvptx-tools
cd nvptx-tools
./configure \
	--with-cuda-driver-include=$cuda/include \
	--with-cuda-driver-lib=$cuda/lib64 \
	--prefix=$install_dir
make
make install
cd ..


# Set up the GCC source tree
git clone https://github.com/MentorEmbedded/nvptx-newlib
wget -c http://gnu.mirror.globo.tech/gcc/gcc-7.3.0/gcc-7.3.0.tar.gz
tar xf gcc-7.3.0.tar.gz
cd gcc-7.3.0
contrib/download_prerequisites
ln -s ../nvptx-newlib/newlib newlib
target=$(./config.guess)
cd ..


# Build nvptx GCC
mkdir build-nvptx-gcc
cd build-nvptx-gcc
../gcc-7.3.0/configure \
	--target=nvptx-none \
	--with-build-time-tools=$install_dir/nvptx-none/bin \
	--enable-as-accelerator-for=$target \
	--disable-sjlj-exceptions \
	--enable-newlib-io-long-long \
	--enable-languages="c,c++,fortran,lto" \
	--prefix=$install_dir
make -j4
make install
cd ..


# Build host GCC
mkdir build-host-gcc
cd  build-host-gcc
../gcc-7.3.0/configure \
	--enable-offload-targets=nvptx-none \
	--with-cuda-driver-include=$cuda/include \
	--with-cuda-driver-lib=$cuda/lib64 \
	--disable-bootstrap \
	--disable-multilib \
	--enable-languages="c,c++,fortran,lto" \
	--prefix=$install_dir
make -j4
make install
cd .. 

should I really need to upgrade my OS ? or something else I am missing like linking something ?

CentOS v6 is 10 years old.... Any modern OS (released in the previous 12 months or so) will likely have a package for offloading so that you don't need to build the compiler yourself (Ubuntu has for example https://packages.ubuntu.com/search?keywords=gcc-7-offload-nvptx). If you really want to stick with CentOS 6, try at least upgrading binutils (maybe with spack).

Can you please try the same code on your machine and verify with nvprof to check it is really running on GPU.

Sorry, I can't. I haven't used this in a long time. The code is correct, so the issue is somewhere else.

Hello Matthiasdiener,

I have used the script to build gcc-offload on another machine with CentOS Linux release 7.6.1810 (Core), now I can run on GPU.
It confirms the problem was due to old OS.

Thanks

Hi Matthais,

Thank you very much for this. I was able to compile binaries that offload correctly for nvidia gpus. I was wondering if you have had any success building a shared library capable of offloading? As soon as I introduce the -shared flag, although it will compile and run, it no longer offloads. Do you have any suggestions in that regard?

Thanks again

Hello Matthiasdiener,

Can the gcc built using your build-gcc-offload-nvptx.sh script compile a shared library / Python extension written in C++ (rather than a stand-alone application) where we make use of OMP targeting GPU?

Thanks for any help / comment / example !

@dhidas and @ochubar: As far as I know, the offloaded code must be statically linked into the main binary to be actually running on a device. This seems to be a limitation across all compilers I have tried (not only gcc) and is unfortunately not well-publicized (I have not seen this limitation in any compiler documentation whatsoever, and it is not mentioned in the OpenMP standard).

Note that your binary can link to shared libraries, just the offloaded code needs to be statically linked to the binary itself.

Also note that putting the offloaded code into a static library (i.e., .a), and then linking that library statically to your binary seems to work fine.

Hi @matthiasdiener, I got the following error while building Host GCC with your script

configure: error: CUDA driver package required for nvptx support
make[1]: *** [configure-target-libgomp] Error 1
make[1]: Leaving directory `/scratch/oscarm/gcc-offload/build-host-gcc'

I am using CUDA 10.1, and I was unlucky in my search for solutions (w google). Do you have any idea about was going on? any help is appreciated!

Hi @matthiasdiener, I got the following error while building Host GCC with your script

configure: error: CUDA driver package required for nvptx support
make[1]: *** [configure-target-libgomp] Error 1
make[1]: Leaving directory `/scratch/oscarm/gcc-offload/build-host-gcc'

I am using CUDA 10.1, and I was unlucky in my search for solutions (w google). Do you have any idea about was going on? any help is appreciated!

Have you adjusted the path to your cuda installation (line 10)?

thx for your quick reply. Yes, I set the path as cuda=/usr/local/cuda-10.1

thx for your quick reply. Yes, I set the path as cuda=/usr/local/cuda-10.1

Hmm, then I'm not sure. Have you checked that /usr/local/cuda-10.1/include and /usr/local/cuda-10.1/lib64 contain the CUDA files?

After using the ls command I realized that there is a paste named stubs inside /usr/local/cuda-10.1/lib64 where is located libcuda.so, so I change the lines --with-cuda-driver-lib=$cuda/lib64 for --with-cuda-driver-lib=$cuda/lib64/stubs and the installation was successful. The question is now to know if it works as expected. Thanks.

After using the ls command I realized that there is a paste named stubs inside /usr/local/cuda-10.1/lib64 where is located libcuda.so, so I change the lines --with-cuda-driver-lib=$cuda/lib64 for --with-cuda-driver-lib=$cuda/lib64/stubs and the installation was successful. The question is now to know if it works as expected. Thanks.

That's great. You can try running any of the examples on this gist to see if it works (maybe check with nvprof that code is actually offloaded).

After using the ls command I realized that there is a paste named stubs inside /usr/local/cuda-10.1/lib64 where is located libcuda.so, so I change the lines --with-cuda-driver-lib=$cuda/lib64 for --with-cuda-driver-lib=$cuda/lib64/stubs and the installation was successful. The question is now to know if it works as expected. Thanks.

That's great. You can try running any of the examples on this gist to see if it works (maybe check with nvprof that code is actually offloaded).

It works properly! Thanks for sharing the script and the example codes :D

Hello @matthiasdiener, I am following your script to use OpenMP. I am new to programming so there is something I am not sure about...
Currently, I am using WSL2 Ubuntu. And I have already installed gcc(version 9.3.0). I was wondering if I should remove this GCC first and then follow your script.

Hello @matthiasdiener, I am following your script to use OpenMP. I am new to programming so there is something I am not sure about...
Currently, I am using WSL2 Ubuntu. And I have already installed gcc(version 9.3.0). I was wondering if I should remove this GCC first and then follow your script.

I don't have experience with WSL, so I'm not sure if this is going to work. I would probably recommend not removing the other gcc.

I'm kind off really new to Linux and building compilers...
Is the script supposed to end with
make: *** [Makefile:906: all] Error 2 ?

I'm kind off really new to Linux and building compilers...
Is the script supposed to end with
make: *** [Makefile:906: all] Error 2 ?

No, this means compilation failed. You need to check for other error messages before that

I'm kind off really new to Linux and building compilers...
Is the script supposed to end with
make: *** [Makefile:906: all] Error 2 ?

No, this means compilation failed. You need to check for other error messages before that

Well it was something along the lines "couldn't change directory"...
Anyways Ubuntu is not exactly my cup of tea, I've wiped it and I'm going to install another distro and try it in there.

i want to install gcc10 for offload of nvptx-none target. How to do it ?? please help me

i want to install gcc10 for offload of nvptx-none target. How to do it ?? please help me

Just follow the script above, replacing the version numbers with your desired version. Or use a package provided by your distro.

i am getting following error while using your script

/bin/sh: line 3: cd: x86_64-pc-linux-gnu/libstdc++-v3: No such file or directory
make[1]: *** [install-target-libstdc++-v3] Error 1

whats d problem ??

i am getting following error while using your script

/bin/sh: line 3: cd: x86_64-pc-linux-gnu/libstdc++-v3: No such file or directory
make[1]: *** [install-target-libstdc++-v3] Error 1

whats d problem ??

Sorry, no idea.

is gpu offloading possible with mpi+openmp 4.5 + ??

is gpu offloading possible with mpi+openmp 4.5 + ??

I think so.

Is it possible to modify this script for Windows?

Is it possible to modify this script for Windows?

I’m not familiar with doing this on windows, unfortunately.

hello
i installed gcc+nvptx using your script.
gcc compiler was installed normally, and I tried to compile a simple openACC FORTRAN program with gfortran.
but i get following errors:

gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c global_parameters_mod.F90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c argument_mod.F90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c kind_params_mod.f90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c gocean_mod.F90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c region_mod.f90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c grid_mod.f90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c kernel_mod.F90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c halo_mod.f90
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -c field_mod.f90
ar rv gocean_api.a argument_mod.o kind_params_mod.o global_parameters_mod.o gocean_mod.o grid_mod.o kernel_mod.o halo_mod.o field_mod.o region_mod.o
ar: creating gocean_api.a
a - argument_mod.o
a - kind_params_mod.o
a - global_parameters_mod.o
a - gocean_mod.o
a - grid_mod.o
a - kernel_mod.o
a - halo_mod.o
a - field_mod.o
a - region_mod.o
make[2]: Leaving directory '/data1/home/synim/test/nemolite2d_edata/api_v1.0'
gfortran -O3 -fopenacc -ffree-line-length-none -foffload=nvptx-none -I../../api_v1.0 -I/src -c nemolite2d.f90
f951: Warning: Nonexistent include directory ‘/src’ [-Wmissing-include-dirs]
gfortran -o nemolite2d.exe nemolite2d.o ../../api_v1.0/gocean_api.a  -lgfortran -lasan -latomic -lgomp
unresolved symbol sin
collect2: error: ld returned 1 exit status
mkoffload: fatal error: x86_64-pc-linux-gnu-accel-nvptx-none-gcc returned 1 exit status
compilation terminated.
lto-wrapper: fatal error: /data1/home/synim/offload/install/libexec/gcc/x86_64-pc-linux-gnu/11.3.1//accel/nvptx-none/mkoffload returned 1 exit status
compilation terminated.
/opt/rh/devtoolset-7/root/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:40: nemolite2d.exe] Error 1
make[1]: Leaving directory '/data1/home/synim/test/nemolite2d_edata/nemolite2d_serial/nemolite2d_orig'
make: *** [Makefile:24: nemolite2d] Error 2

Anyone who knows about this issue, please help.
thanks.

Anyone who knows about this issue, please help. thanks.

This script is for OpenMP offloading support, I haven't tested it with OpenAcc at all.

Hello,

I have tried to launch your script and everything passes except for the make of the Build Host GCC.
I have the following error:

make[3]: *** [Makefile:539 : sanitizer_platform_limits_posix.lo] Erreur 1 make[3] : on quitte le répertoire « /DISK4/offload/build-host-gcc/x86_64-pc-linux-gnu/libsanitizer/sanitizer_common » make[2]: *** [Makefile:472 : install-recursive] Erreur 1 make[2] : on quitte le répertoire « /DISK4/offload/build-host-gcc/x86_64-pc-linux-gnu/libsanitizer » make[1]: *** [Makefile:12708 : install-target-libsanitizer] Erreur 2 make[1] : on quitte le répertoire « /DISK4/offload/build-host-gcc » make: *** [Makefile:2301 : install] Erreur 2
It's written in French but you can see the libraries that cause the errors.

Thanks,

Hello，
Is there any GPU version requirement for this, I mean for arithmetic or architectural requirements, etc.?
Thanks,

Hello， Is there any GPU version requirement for this, I mean for arithmetic or architectural requirements, etc.? Thanks,

I don't know - it must depend on gcc version etc.