Skip to content

Instantly share code, notes, and snippets.

@prehensilecode

prehensilecode/ncbi_ngs_build_rhel8.md

Last active November 2, 2023 00:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save prehensilecode/91cf502d2f5fa8ea1290b13ca0b37970 to your computer and use it in GitHub Desktop.
Save prehensilecode/91cf502d2f5fa8ea1290b13ca0b37970 to your computer and use it in GitHub Desktop.
Download ZIP
HOWTO Build NCBI NGS on RHEL 8
Raw
ncbi_ngs_build_rhel8.md

NGS

Modules:

  • gcc/9.2.0
  • python/gcc/3.9.1

Configure

 ./configure --prefix=/ifs/opt/ncbi-ngs/2.10.9 --without-debug --relative-build-out-dir

Output:

CONFIGURING ngs-sdk
Configuring NGS-SDK package
checking system type... Linux
checking OS distributor... RedHatEnterprise
checking machine architecture... x86_64
checking for supported architecture... x86_64 (64 bits) is supported
checking for supported OS... Linux (linux) is supported
checking for supported tool chain... gcc tool chain is supported
checking for g++... yes
checking for Python 3... /ifs/opt/python/gcc/3.9.1/bin/python3
checking whether gcc accepts -Wno-array-bounds... yes
checking whether g++ accepts -static-libstdc++... yes

configure: creating 'ld.linux.exe_cmd.sh'
configure: creating 'Makefile.config.linux.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /ifs/opt/src/ncbi/OUTDIR/ngs-sdk
build output path: /ifs/opt/src/ncbi/OUTDIR/ngs-sdk/linux/gcc/x86_64/rel
includedir: /ifs/opt/ncbi-ngs/2.10.9/include
bindir: /ifs/opt/ncbi-ngs/2.10.9/bin
libdir: /ifs/opt/ncbi-ngs/2.10.9/lib
sharedir: /ifs/opt/ncbi-ngs/2.10.9/share
CC = gcc -c
CPP = g++
configured with: "'--prefix=/ifs/opt/ncbi-ngs/2.10.9' '--without-debug' '--relative-build-out-dir'"

CONFIGURING ngs-java
Configuring NGS-JAVA package
checking system type... Linux
checking OS distributor... RedHatEnterprise
checking machine architecture... x86_64
checking for supported architecture... x86_64 (64 bits) is supported
checking for supported OS... Linux (linux) is supported
checking for supported tool chain... jdk tool chain is supported
checking for javac... yes
checking for Python 3... /ifs/opt/python/gcc/3.9.1/bin/python3

configure: creating 'Makefile.config.linux.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /ifs/opt/src/ncbi/OUTDIR/ngs-java
build output path: /ifs/opt/src/ncbi/OUTDIR/ngs-java
includedir: /ifs/opt/ncbi-ngs/2.10.9/include
bindir: /ifs/opt/ncbi-ngs/2.10.9/bin
libdir: /ifs/opt/ncbi-ngs/2.10.9/lib
sharedir: /ifs/opt/ncbi-ngs/2.10.9/share
pythondir: /ifs/opt/ncbi-ngs/2.10.9
CC = gcc -c
CPP = g++
configured with: "'--prefix=/ifs/opt/ncbi-ngs/2.10.9' '--without-debug' '--relative-build-out-dir'"

Make

  • make -C ngs-sdk
    • successful
  • make -C ngs-java
    • successful (several Java warnings)
  • make -C ngs-python
    • successful

ncbi-vdb

This comes partway through the NCBI NGS build, per documentation

Configure

 ./configure --prefix=/ifs/opt/ncbi-ngs/2.10.9 --with-ngs-sdk-prefix=/ifs/opt/ncbi-ngs/2.10.9 --without-debug --relative-build-out-dir

Failed:

Configuring NCBI-VDB package
checking system type... Linux
checking OS distributor... RedHatEnterprise
checking machine architecture... x86_64
checking for supported architecture... x86_64 (64 bits) is supported
checking for supported OS... Linux (linux) is supported
checking for supported tool chain... gcc tool chain is supported
checking for g++... yes
checking for Python 3... /ifs/opt/python/gcc/3.9.1/bin/python3
checking whether gcc accepts -Wno-array-bounds... yes
checking whether g++ accepts -static-libstdc++... yes
checking for fuse library... yes
checking for hdf5 library... no
checking for magic library... yes
checking for xml2 library... yes
checking for ngs-sdk package...
        includes... no
        libraries... no
        libraries... /ifs/opt/src/ncbi/ncbi-vdb-2.10.9/setup/../../OUTDIR/ngs-sdk/linux/gcc/x86_64/rel/lib
configure: error: required ngs-sdk package not found.

Back to NGS

Make ngs-sdk install

  • make -C ngs-sdk install
    • successful
make: Entering directory '/ifs/opt/src/ncbi/ngs-2.10.9/ngs-sdk'
Checking make status of ngs-sdk/language...
Checking make status of ngs-sdk/dispatch...
Checking make status of ngs-sdk/adapter...
Checking make status of object libraries...
Installing libraries to /ifs/opt/ncbi-ngs/2.10.9/lib64
installing 'libngs-sdk.so.2.10.9'... success
installing 'libngs-c++.a.2.10.9'... success
installing 'libngs-adapt-c++.a.2.10.9'... success
Installing includes to /ifs/opt/ncbi-ngs/2.10.9/include
Installing examples to /ifs/opt/ncbi-ngs/2.10.9/share
Please add /ifs/opt/ncbi-ngs/2.10.9/lib64 to your LD_LIBRARY_PATH, e.g.:
      export LD_LIBRARY_PATH=/ifs/opt/ncbi-ngs/2.10.9/lib64:$LD_LIBRARY_PATH
Use /ifs/opt/ncbi-ngs/2.10.9/lib64 in your link commands, e.g.:
      export NGS_LIBDIR=/ifs/opt/ncbi-ngs/2.10.9/lib64
      ld -L$NGS_LIBDIR -lngs-sdk ...
make: Leaving directory '/ifs/opt/src/ncbi/ngs-2.10.9/ngs-sdk'

Check environment:

$ echo $LD_LIBRARY_PATH
/ifs/opt/ncbi-ngs/2.10.9/lib64:/ifs/opt/hdf5/gcc/1.12.0-serial/lib:/ifs/opt/perl-threaded/5.32.1/lib:/ifs/opt/python/gcc/3.9.1/lib:/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64:/cm/local/apps/gcc/9.2.0/lib:/cm/local/apps/gcc/9.2.0/lib64
host01::/ifs/opt/src/ncbi/ngs-2.10.9$ echo $NGS_LIBDIR
/ifs/opt/ncbi-ngs/2.10.9lib64

Make ngs-java install

  • make -C ngs-java install
    • successful
make: Entering directory '/ifs/opt/src/ncbi/ngs-2.10.9/ngs-java'
Checking make status of jars...
installing 'ngs-java.jar.2.10.9'...
Copying html docs to /ifs/opt/ncbi-ngs/2.10.9/share/doc/...
Installing examples to /ifs/opt/ncbi-ngs/2.10.9/share/examples-java...
Please add /ifs/opt/ncbi-ngs/2.10.9/jar/ngs-java.jar to your CLASSPATH, i.e.:
      export CLASSPATH=/ifs/opt/ncbi-ngs/2.10.9/jar/ngs-java.jar:$CLASSPATH
make: Leaving directory '/ifs/opt/src/ncbi/ngs-2.10.9/ngs-java'

Check environment:

$ echo $CLASSPATH
/ifs/opt/ncbi-ngs/2.10.9/ngs-java/ngs-java.jar

Back to ncbi-vdb

Configure

 ./configure --prefix=/ifs/opt/ncbi-ngs/2.10.9 --with-ngs-sdk-prefix=/ifs/opt/ncbi-ngs/2.10.9 --without-debug --relative-build-out-dir
  • successful
Configuring NCBI-VDB package
checking system type... Linux
checking OS distributor... RedHatEnterprise
checking machine architecture... x86_64
checking for supported architecture... x86_64 (64 bits) is supported
checking for supported OS... Linux (linux) is supported
checking for supported tool chain... gcc tool chain is supported
checking for g++... yes
checking for Python 3... /ifs/opt/python/gcc/3.9.1/bin/python3
checking whether gcc accepts -Wno-array-bounds... yes
checking whether g++ accepts -static-libstdc++... yes
checking for fuse library... yes
checking for hdf5 library... no
checking for magic library... yes
checking for xml2 library... yes
checking for ngs-sdk package...
        includes... /ifs/opt/ncbi-ngs/2.10.9
        libraries... /ifs/opt/ncbi-ngs/2.10.9/lib64
includes: /ifs/opt/ncbi-ngs/2.10.9/include
libraries: /ifs/opt/ncbi-ngs/2.10.9/lib64
checking for ngs-java package...
        jar... /ifs/opt/src/ncbi/ncbi-vdb-2.10.9/setup/../../OUTDIR/ngs-java/jar/ngs-java.jar
jar: /ifs/opt/src/ncbi/OUTDIR/ngs-java/jar/ngs-java.jar

configure: creating 'build/ld.linux.exe_cmd.sh'
configure: creating 'build/Makefile.config.linux.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /ifs/opt/src/ncbi/OUTDIR/ncbi-vdb
build output path: /ifs/opt/src/ncbi/OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel
includedir: /ifs/opt/ncbi-ngs/2.10.9/include
bindir: /ifs/opt/ncbi-ngs/2.10.9/bin
libdir: /ifs/opt/ncbi-ngs/2.10.9/lib
CC = gcc -c
CPP = g++
configured with: "'--prefix=/ifs/opt/ncbi-ngs/2.10.9' '--with-ngs-sdk-prefix=/ifs/opt/ncbi-ngs/2.10.9' '--without-debug' '--relative-build-out-dir'"
  • make
    • Warnings & errors:
NOTE - library libkdf5 cannot be built: It requires 'libhdf5' and its development headers.
...
/local/scratch/ccO5dB5I.s: Assembler messages:
/local/scratch/ccO5dB5I.s:1092: Error: `pinsrq' is not supported on `generic64.aes'
/local/scratch/ccO5dB5I.s:1116: Error: `pinsrq' is not supported on `generic64.aes'
/local/scratch/ccO5dB5I.s:1121: Error: `pinsrq' is not supported on `generic64.aes'
make[    ** [/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto/Makefile:194: cipher-vec.aes-ni.pic.o](3]:) Error 1
make[Leaving directory '/ifs/opt/src/ncbi/OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/krypto'
reattempting without AES-NI instructions

Make install

make install
  • Warnings & errors:
Checking make status of object libraries...
gcc -c  -std=c11  -DNDEBUG -m64   -DLINUX -DUNIX -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=linux64 -D_ARCH_BITS=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces/override -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto/linux -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto/unix -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces/cc/gcc/x86_64 -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces/cc/gcc -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces/os/linux -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces/os/unix -I/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces/ext -I/ifs/opt/ncbi-ngs/2.10.9/include -I. -MD -o cipher-vec.aes-ni.pic.o -fPIC -O3 -Wno-variadic-macros -fno-strict-aliasing -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -Wa,--noexecstack -Wall -D_LIBRARY -DUSEVEC -DUSEVECREG -DUSEAESNI -funsafe-math-optimizations -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -maes -Wa,-march=generic64+aes -Wa,-ahlms=/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto/cipher-vec.aes-ni.pic.o.list /ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto/cipher-vec.c
/local/scratch/ccXSWXIC.s: Assembler messages:
/local/scratch/ccXSWXIC.s:1092: Error: `pinsrq' is not supported on `generic64.aes'
/local/scratch/ccXSWXIC.s:1116: Error: `pinsrq' is not supported on `generic64.aes'
/local/scratch/ccXSWXIC.s:1121: Error: `pinsrq' is not supported on `generic64.aes'
make[4](3]:):     ** [cipher-vec.aes-ni.pic.o](/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/libs/krypto/Makefile:194:) Error 1
reattempting without AES-NI instructions
NOTE - library libkdf5 cannot be built: It requires 'libhdf5' and its development headers.
Installing libraries to /ifs/opt/ncbi-ngs/2.10.9/lib64
installing 'libncbi-ngs-c++.a.2.10.9'... success
installing 'libncbi-vdb.a.2.10.9'... success
installing 'libncbi-vdb.so.2.10.9'... success
installing 'libncbi-wvdb.a.2.10.9'... success
installing 'libncbi-wvdb.so.2.10.9'... success
Installing includes to /ifs/opt/ncbi-ngs/2.10.9/include
Installing configuration files to /ifs/opt/ncbi-ngs/2.10.9/lib64/ncbi
  • Check
$ ls -l /ifs/opt/ncbi-ngs/2.10.9/lib64
total 32568
lrwxrwxrwx 1 myname somegrp      19 Feb 27 20:06 libncbi-ngs-c++.a -> libncbi-ngs-c++.a.2
lrwxrwxrwx 1 myname somegrp      24 Feb 27 20:06 libncbi-ngs-c++.a.2 -> libncbi-ngs-c++.a.2.10.9
-rw-r--r-- 1 myname somegrp    6884 Feb 27 20:06 libncbi-ngs-c++.a.2.10.9
lrwxrwxrwx 1 myname somegrp      17 Feb 27 20:06 libncbi-ngs-c++-static.a -> libncbi-ngs-c++.a
lrwxrwxrwx 1 myname somegrp      15 Feb 27 20:06 libncbi-vdb.a -> libncbi-vdb.a.2
lrwxrwxrwx 1 myname somegrp      20 Feb 27 20:06 libncbi-vdb.a.2 -> libncbi-vdb.a.2.10.9
-rw-r--r-- 1 myname somegrp 7945612 Feb 27 20:06 libncbi-vdb.a.2.10.9
lrwxrwxrwx 1 myname somegrp      16 Feb 27 20:06 libncbi-vdb.so -> libncbi-vdb.so.2*
lrwxrwxrwx 1 myname somegrp      21 Feb 27 20:06 libncbi-vdb.so.2 -> libncbi-vdb.so.2.10.9*
-rwxr-xr-x 1 myname somegrp 5273944 Feb 27 20:06 libncbi-vdb.so.2.10.9*
lrwxrwxrwx 1 myname somegrp      13 Feb 27 20:06 libncbi-vdb-static.a -> libncbi-vdb.a
lrwxrwxrwx 1 myname somegrp      16 Feb 27 20:06 libncbi-wvdb.a -> libncbi-wvdb.a.2
lrwxrwxrwx 1 myname somegrp      21 Feb 27 20:06 libncbi-wvdb.a.2 -> libncbi-wvdb.a.2.10.9
-rw-r--r-- 1 myname somegrp 7358082 Feb 27 20:06 libncbi-wvdb.a.2.10.9
lrwxrwxrwx 1 myname somegrp      17 Feb 27 20:06 libncbi-wvdb.so -> libncbi-wvdb.so.2*
lrwxrwxrwx 1 myname somegrp      22 Feb 27 20:06 libncbi-wvdb.so.2 -> libncbi-wvdb.so.2.10.9*
-rwxr-xr-x 1 myname somegrp 4909760 Feb 27 20:06 libncbi-wvdb.so.2.10.9*
lrwxrwxrwx 1 myname somegrp      14 Feb 27 20:06 libncbi-wvdb-static.a -> libncbi-wvdb.a
lrwxrwxrwx 1 myname somegrp      20 Feb 27 19:42 libngs-adapt-c++.a -> libngs-adapt-c++.a.2
lrwxrwxrwx 1 myname somegrp      25 Feb 27 19:42 libngs-adapt-c++.a.2 -> libngs-adapt-c++.a.2.10.9
-rw-r--r-- 1 myname somegrp  183950 Feb 27 19:42 libngs-adapt-c++.a.2.10.9
lrwxrwxrwx 1 myname somegrp      18 Feb 27 19:42 libngs-adapt-c++-static.a -> libngs-adapt-c++.a
lrwxrwxrwx 1 myname somegrp      14 Feb 27 19:42 libngs-c++.a -> libngs-c++.a.2
lrwxrwxrwx 1 myname somegrp      19 Feb 27 19:42 libngs-c++.a.2 -> libngs-c++.a.2.10.9
-rw-r--r-- 1 myname somegrp  294722 Feb 27 19:42 libngs-c++.a.2.10.9
lrwxrwxrwx 1 myname somegrp      12 Feb 27 19:42 libngs-c++-static.a -> libngs-c++.a
lrwxrwxrwx 1 myname somegrp      15 Feb 27 19:42 libngs-sdk.so -> libngs-sdk.so.2*
lrwxrwxrwx 1 myname somegrp      20 Feb 27 19:42 libngs-sdk.so.2 -> libngs-sdk.so.2.10.9*
-rwxr-xr-x 1 myname somegrp  331992 Feb 27 19:42 libngs-sdk.so.2.10.9*
drwxrwsr-x 2 myname somegrp      56 Feb 27 20:06 ncbi/

Back to NGS to build ngs-python

Make ngs-python install

 make -C ngs-python install
  • success BUT
    • picks up the wrong python (see above): expecting it to be /ifs/opt/python/gcc/3.9.1/bin/python3
    • missing a "/" in the lib dir path
make: Entering directory '/ifs/opt/src/ncbi/ngs-2.10.9/ngs-python'
Installing examples to /ifs/opt/ncbi-ngs/2.10.9/share/examples-python
Installing ngs-python package to the current user...
$ python -m site --user-base
/home/myname/.local
$ python -m site --user-site
/home/myname/.local/lib/python3.6/site-packages
/usr/lib64/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'include_package_data'
  warnings.warn(msg)
... installed
Please add /ifs/opt/ncbi-ngs/2.10.9lib64:/ifs/opt/src/ncbi/OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel/lib to your LD_LIBRARY_PATH, e.g.:
      export LD_LIBRARY_PATH=/ifs/opt/ncbi-ngs/2.10.9lib64:/ifs/opt/src/ncbi/OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel/lib:$LD_LIBRARY_PATH
make: Leaving directory '/ifs/opt/src/ncbi/ngs-2.10.9/ngs-python'

sra-tools

Documentation: https://github.com/ncbi/sra-tools/wiki/Building-and-Installing-from-Source

Configure

 ./configure --prefix=/ifs/opt/ncbi-ngs/2.10.9 --with-ngs-sdk-prefix=/ifs/opt/ncbi-ngs/2.10.9 --with-ncbi-vdb-sources=/ifs/opt/src/ncbi/ncbi-vdb-2.10.9 --without-debug --relative-build-out-dir

Output:

Configuring SRA-TOOLS package
checking system type... Linux
checking OS distributor... RedHatEnterprise
checking machine architecture... x86_64
checking for supported architecture... x86_64 (64 bits) is supported
checking for supported OS... Linux (linux) is supported
checking for supported tool chain... gcc tool chain is supported
checking for g++... yes
checking for Python 3... /ifs/opt/python/gcc/3.9.1/bin/python3
checking whether gcc accepts -Wno-array-bounds... yes
checking whether g++ accepts -static-libstdc++... yes
checking for QMake... no
checking for fuse library... yes
checking for hdf5 library... no
checking for magic library... yes
checking for xml2 library... yes
checking for ngs-sdk package...
        includes... /ifs/opt/ncbi-ngs/2.10.9
        libraries... /ifs/opt/ncbi-ngs/2.10.9/lib64
includes: /ifs/opt/ncbi-ngs/2.10.9/include
libraries: /ifs/opt/ncbi-ngs/2.10.9/lib64
checking for ncbi-vdb package source files and build results...
        includes... /ifs/opt/src/ncbi/ncbi-vdb-2.10.9
        src... /ifs/opt/src/ncbi/ncbi-vdb-2.10.9
        libraries... /ifs/opt/src/ncbi/sra-tools-2.10.9/setup/../../OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel/ilib
includes: /ifs/opt/src/ncbi/ncbi-vdb-2.10.9/interfaces
sources: /ifs/opt/src/ncbi/ncbi-vdb-2.10.9
libraries: /ifs/opt/src/ncbi/OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel/lib
ilibraries: /ifs/opt/src/ncbi/OUTDIR/ncbi-vdb/linux/gcc/x86_64/rel/ilib

configure: creating 'build/ld.linux.exe_cmd.sh'
configure: creating 'build/Makefile.config.linux.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /ifs/opt/src/ncbi/OUTDIR/sra-tools
build output path: /ifs/opt/src/ncbi/OUTDIR/sra-tools/linux/gcc/x86_64/rel
includedir: /ifs/opt/ncbi-ngs/2.10.9/include
bindir: /ifs/opt/ncbi-ngs/2.10.9/bin
libdir: /ifs/opt/ncbi-ngs/2.10.9/lib
CC = gcc -c
CPP = g++
configured with: "'--prefix=/ifs/opt/ncbi-ngs/2.10.9' '--with-ngs-sdk-prefix=/ifs/opt/ncbi-ngs/2.10.9' '--with-ncbi-vdb-sources=/ifs/opt/src/ncbi/ncbi-vdb-2.10.9' '--without-debug' '--relative-build-out-dir'"

Make

  • Success

Install

  • Success
  • Check:
$ which fastq-dump
/ifs/opt/ncbi-ngs/2.10.9/bin/fastq-dump
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment