I am going to write a comment for every damn IT issue/fix I get...

fixes-found.md

This is an experiment. Is a GitHub Gist comment a good way to log publically how I fixed an installation or similar issue?

See comments

PyMOL does not install on Python 3.11 or above. That I know of there is nothing in 3.11 that breaks ported 2.7-code more than say 3.10. So

conda install -c conda-forge -c schrodinger freetype glew h5py libgcc-ng libholoplaycore libnetcdf libpng libstdcxx-ng libxml2 mtz2ccp4_px numpy
conda install -c conda-forge -c schrodinger --no-deps pymol

Zombie threads kill my webserver which is under a bogstandard Apache reverse proxy, so this root crontab restarts its service if the /status (/api for a Jupyter) fails on the original port:

if [[ $(curl -s -o /dev/null -w "%{http_code}" http://localhost:👾👾👾/status) != "200" ]]; then systemctl restart 👾👾👾.service; fi

I have not got it working inline on a cron job, so I have it in a root owned (700) script in /opt.

No Zombies!

I got asked this, which has become second nature but is clearly an assumed thing.

Jupyter displays an rdkit Chem.Mol as <rdkit.Chem.rdchem.Mol object at 0x11c847a70>.

To fix import from rdkit.Chem.Draw import IPythonConsole

Installing PLIP & OpenBabel in some circumstances:

conda install -c conda-forge openbabel lxml
conda install -c conda-forge --no-deps plip

How to check for anything suspicious on my homeserver. This will show the countries of logins:
zgrep sshd /var/log/auth.log* | grep -F 'Accepted' | sed -re 's/.*from ([^ ]+).*/\1/' | sort -u | xargs -n 1 geoiplookup
This does not however mean the auth.log was not tampered with.

This is an answer to a Q, so I am not sure of the exact cause, but I think a badly timed control-X on a conda install.
If odd errors happen on install, clean the index! conda clean --index-cache -y

The cluster I use has Nvidia driver 510, which does not accept cuda toolkit 12. The subversion of 11 is confusing as its cuda-toolkit=11.8.0 pytorch-cuda=11.3

It took me a minute or two to find my normal conda environment variables —I had put some in https://www.blopig.com/blog/2023/03/the-ultimate-modulefile-for-conda/

export CONDA_YES=true
export CONDA_CHANNELS="conda-forge,nvidia" #comma not space!
export CONDA_SOLVER=libmamba

Everyone hates tensorflow, but alphafold2 uses it so we are stuck with it.
Max Python is still 3.9. I have tried higher —not worth the fight. Colab seems to make Py 3.10 + TF + Cuda11 (cf. ColabFold), however the conda frozen yaml does not replicate on my systsem.
Ironically some dependencies no longer support CentOS 7 (glibc2.17 issue).
The internet is full of obsolete info, these are current:

• Pip tensorflow>=2.12 includes tensorflow-gpu functionality, hence why the latter is stuck at older versions.
• cuDNN is pip installed with jax[cuda12_pip]
• conda/dnf cuda-compat fixes the drama of 11 vs 12 drivers, mostly. These needs to be set however as always:

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:/usr/local/cuda/compat:/.singularity.d/libs:/usr/lib64/nvidia:/usr/lib/nvidia

In Numpy >1.20 a few aliases were deprecated, but can be undone:

import numpy as np
np.object = object
np.bool = bool
np.int = int
np.typeDict = np.sctypeDict

I keep having issues with libmamba solver for conda. This time its actually not libmamba, but libmambapy.
libmambapy is the Pythonic reimplementation of the C++ library libmamba.

Error while loading conda entry point: conda-libmamba-solver (/data/xchem-fragalysis/mferla/waconda/envs/cuda118/lib/python3.11/site-packages/../../libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /data/xchem-fragalysis/mferla/waconda/lib/python3.11/site-packages/libmambapy/../../../libmamba.so.2))

Unsurprisingly, this was not directly installed by me, but by jax. Ie. a random config requirement that really ought to not be there —the award for that goes to tensorflow whose requirement are effectly a conda environment export.

To fix this I did:

ldd --version

2.34
and then

CONDA_OVERRIDE_GLIB=2.34 conda install --solver classic libmambapy --force-reinstall

Installation of MaSIF via conda. 🚧 Not finished.
GitHub: https://github.com/lpdi-epfl/masif has no conda, because of its dozen requirements, some old.

# ## Start with creation
export CONDA_YES=true
# I believe this ancient prepandemic python is due to pymesh:
conda create -n masif python=3.6
conda activate masif
# I don't want stuff in $HOME/.local and PIP_NO_USERBASE:
export PYTHONUSERBASE=$CONDA_PREFIX
conda env config vars set PYTHONUSERBASE=$PYTHONUSERBASE

# ## APBS
#conda install -c conda-forge apbs #No: this does not install multivalue
wget https://github.com/Electrostatics/apbs/releases/download/v3.4.1/APBS-3.4.1.Linux.zip -O /tmp/APBS-3.4.1.Linux.zip
unzip /tmp/APBS-3.4.1.Linux.zip -d /tmp/APBS-3.4.1.Linux
rsync -ar /tmp/APBS-3.4.1.Linux/ $CONDA_PREFIX
export APBS_BIN=$CONDA_PREFIX/bin/apbs
conda env config vars set APBS_BIN=$APBS_BIN
export MULTIVALUE_BIN=$CONDA_PREFIX/share/apbs/tools/bin/multivalue
conda env config vars set MULTIVALUE_BIN=$MULTIVALUE_BIN

# ## MSMS
conda install -c bioconda msms
export MSMS_BIN=$CONDA_PREFIX/bin/msms
export PDB2XYZRN=$CONDA_PREFIX/bin/pdb_to_xyzrn
conda env config vars set MSMS_BIN=$MSMS_BIN
conda env config vars set PDB2XYZRN=$PDB2XYZRN

# ## PDB2PQR
conda install -c schrodinger pdb2pqr
export PDB2PQR_BIN=$CONDA_PREFIX/bin/pdb2pqr
conda env config vars set PDB2PQR_BIN=$PDB2PQR_BIN

# ## Open3D
conda install -c open3d-admin open3d

# ## Reduce
conda install -c ostrokach-forge reduce
conda env config vars set PATH=$PATH:$CONDA_PREFIX/bin/reduce
# forgive me for this transgression:
export PATH=$PATH
# reduce dict needs adding:
curl https://github.com/rlabduke/reduce/blob/master/reduce_wwPDB_het_dict.txt -o $CONDA_PREFIX/var/reduce_wwPDB_het_dict.txt
export REDUCE_HET_DICT=$CONDA_PREFIX/var/reduce_wwPDB_het_dict.txt

# ## PyMesh
conda install -c conda-forge pymesh2
export PYMESH_PATH=$CONDA_PREFIX/lib/python3.6/site-packages/pymesh
conda env config vars set PYMESH_PATH=$PYMESH_PATH

# MaSIF itself
git clone https://github.com/lpdi-epfl/masif $CONDA_PREFIX/masif_repo
export PATH=$CONDA_PREFIX/masif_repo/data/masif:$PATH

NB. masif_repo/requirements.txt is not a valid Python pip requirements.txt it's just rubbish.

The repo is not installed but simply dumped. That would need to be fixed especially the shell scripts

I have not looked into why conda install -c conda-forge pymesh2 seems fine while this https://github.com/PyMesh/PyMesh is lengthy

Installing ColabFold with a 11.x driver not compatible with cuda-compat in December 2023 (in singularity without root conda-toolkit where /.singularity.d/libs do not match).

conda create -n colabfold python=3.10
conda activate colabfold
conda env config vars set PYTHONUSERBASE=$CONDA_PREFIX
conda install ipykernel
conda install git
$CONDA_PREFIX/bin/python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
conda install -c conda-forge python=3.10 cudnn==8.2.1.32 cudatoolkit==11.6.0 openmm==7.7.0 pdbfixer -y
$CONDA_PREFIX/bin/python -m pip install --upgrade pip
$CONDA_PREFIX/bin/python -m pip install --no-warn-conflicts "colabfold[alphafold-minus-jax] @ git+https://github.com/sokrypton/ColabFold" tensorflow==2.12.0
$CONDA_PREFIX/bin/python -m pip install https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.25+cuda11.cudnn82-cp310-cp310-manylinux2014_x86_64.whl
$CONDA_PREFIX/bin/python -m pip install jax==0.3.25 chex==0.1.6 biopython==1.79
$CONDA_PREFIX/bin/python -m pip install -y jax==0.3.25 chex==0.1.6
conda install -c "nvidia/label/cuda-11.6.0" cuda-toolkit
export LD_LIBRARY=$CONDA_PREFIX
rm -rf $CONDA_PREFIX/lib/python3.10/site-packages/nvidia

The $LD_LIBRARY is not assigned /.singularity.d/libs as those mismatch in DLS.
Normally the values /usr/local/cuda/compat, /usr/local/nvidia/lib, /usr/local/nvidia/lib64 and $CONDA_PREFIX/lib may be used. Some instructions add /proc/driver/nvidia, which is 100% comedically wrong. Specifying a path to specific packages might do something but effort (eg. $CONDA_PREFIX/lib/python3.11/site-packages/nvidia/cudnn/lib.

The removal of the package by rm is because CuDNN is being installed via multiple packages: use find $CONDA_PREFIX -name "libdevice.*" to find the conflict locations.

The problem child is tensorflow, to test it:

import os

#os.environ['LD_LIBRARY_PATH'] = os.environ['CONDA_PREFIX']
import tensorflow as tf
import sys, os

print(sys.version_info)
print(tf.config.list_physical_devices('GPU'))
print('CUDA build:', tf.test.is_built_with_cuda())
print("CUDA version:", tf.sysconfig.get_build_info()["cuda_version"])
print("cuDNN version:", tf.sysconfig.get_build_info()["cudnn_version"])
print("CUDA library paths:", tf.sysconfig.get_lib())
a = tf.constant([[1.0, 2.0], [3.0, 4.0]])
b = tf.constant([[1.0, 1.0], [0.0, 1.0]])
c = tf.matmul(a, b)
print(c)

Note, Pytorch is easier to please while tensorflow does seem to not respect $LD_LIBRARY_PATH. It is possible to have a env with a working pytorch but broken tensorflow. In which case loading torch then tf will work, so be aware of that.

Colab installation of Pymol

import sys
!wget -qnc https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
!bash Mambaforge-Linux-x86_64.sh -bfp /usr/local
sys.path.append('/usr/local/lib/python3.10/site-packages')
!clear
!/usr/local/condabin/mamba install -y conda-forge::pymol-open-source

Works with

import os
os.environ['LD_LIBRARY_PATH'] = '/usr/local/lib:/usr/local/lib64:/usr/local/bin/openssl:/usr/local/nvidia/lib:/usr/local/nvidia/lib64'

import pymol2

git clone https://github.com/👾👾👾/👾👾👾.git gave: 'file:///lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0'
Fixed with conda install anaconda::openldap

Pyrosetta cannot read mmCIF/PDBx files. The following seems to work: pose: pyrosetta.Pose = prc.import_pose.pose_from_file('crysalin_lattice.cif', read_fold_tree=True, type=prc.import_pose.FileType.CIF_file) but it does not.
However, atom indices are ignored so I will just convert into PDB with over >9_999 atoms. 🤷

Installation of the RDKit Postgres catridge

conda create -n py310 python==3.10 ipykernel -y
conda activate py310
conda env config vars set PYTHONUSERBASE=$CONDA_PREFIX
export PYTHONUSERBASE=$CONDA_PREFIX

# https://www.rdkit.org/docs/Cartridge.html
conda install -y -c conda-forge -c rdkit rdkit-postgresql postgresql
# lets be a traditionalist
conda env config vars set PGDATA=$CONDA_PREFIX/var/lib/pgsql/data
export PGDATA=$CONDA_PREFIX/var/lib/pgsql/data
# starting rdkit db
initdb
echo "shared_buffers = 2048MB
work_mem = 128MB
synchronous_commit = off
full_page_writes = off" >> "$PGDATA/postgresql.conf"
pg_ctl -l $HOME2/logs/postgres.log start
createdb test
# dropdb test
psql -c 'create extension rdkit'test
psql -c 'create table raw_data (id SERIAL, smiles text, emol_id integer, parent_id integer)' test
...

Getting the Python API of the CCDC CSD working is not too straightforward —the older documentation does not help.
The installation of CCDC is obviously needed so: ./CSDInstallerOffline-2023.3-linux --accept-licenses install uk.ac.cam.ccdc --root $CCDC_HOME -c

cleaner environment variables:

export CCDC_HOME=$HOME/CCDC
conda activate base
# conda install --channel=https://conda.ccdc.cam.ac.uk csd-python-api # does not work for me
cp -r $CCDC_HOME/ccdc-software/csd-python-api.minoconda $CONDA_PREFIX/envs/CDS
conda activate CSD
# following is an alternative to `$CCDC_HOME/ccdc-utilities/software-activation/bin/ccdc_activator -a -k 👾👾👾`
conda env config vars set CCDC_LICENSING_CONFIGURATION='la-code;👾👾👾'
conda env config vars set CCDC_TOOLKIT_ASER_DATABASE=$CCDC_HOME/ccdc-data/csd/as544be
conda env config vars set CCDC_TOOLKIT_SQLITE_DATABASE=$CCDC_HOME/ccdc-data/csd/as544be_ASER.sqlite
conda env config vars set CCDC_ISOSTAR_DATA_DIRECTORY=$CCDC_HOME/ccdc-data/isostar
conda env config vars set CCDC_MOGUL_DATA=$CCDC_HOME/ccdc-data/mogul
conda env config vars set CCDC_MOGUL_INITIALISATION_FILE=$CONDA_PREFIX/lib/python3.9/site-packages/ccdc/parameter_files/mogul.ini
conda env config vars set CCDC_CROSSMINER_DATABASE=$CCDC_HOME/ccdc-data/crossminer/csd_pdb_crossminer.feat
conda env config vars set CCDC_CROSSMINER_FEATURE_DEFINITIONS=$CCDC_HOME/ccdc-data/crossminer/feature_definitions
conda env config vars set GOLD_DIR=$CCDC_HOME/ccdc-software/gold/GOLD
# always good:
conda env config vars set PYTHONUSERBASE=$CONDA_PREFIX
conda deactivate
conda activate CSD
# this applied to my system only (Mac OS), was fine in Fedora
# First error: `Symbol not found: _png_create_info_struct` (Mac OS 14)
conda install anaconda::libpng
# also for my notebook
conda install ipykernel

Module file for CCDC CSD.

#%Module1.0
##
## csd23-module
## Module file for setting up environment for CSD23
## Written by Matteo Ferla — matteo.ferla@stats.ox.ac.uk

proc ModulesHelp { } {

    puts stderr "\tAdds CSD23 CCDC Utilities and Software to your environment variables"
}

module-whatis "Adds CSD23 CCDC Utilities and Software to your environment variables"
set CCDC_HOME /vols/opig/apps/CSD23
prepend-path PATH /vols/opig/apps/CSD23/ccdc-utilities
prepend-path PATH /vols/opig/apps/CSD23/ccdc-software
setenv CCDC_LICENSING_CONFIGURATION "la-code;👾👾👾"
setenv CCDC_HOME /vols/opig/apps/CSD23
setenv CCDC_TOOLKIT_ASER_DATABASE $CCDC_HOME/ccdc-data/csd/as544be
setenv CCDC_TOOLKIT_SQLITE_DATABASE $CCDC_HOME/ccdc-data/csd/as544be_ASER.sqlite
setenv CCDC_ISOSTAR_DATA_DIRECTORY $CCDC_HOME/ccdc-data/isostar
setenv CCDC_MOGUL_DATA $CCDC_HOME/ccdc-data/mogul
setenv CCDC_MOGUL_INITIALISATION_FILE $CCDC_HOME/lib/python3.9/site-packages/ccdc/parameter_files/mogul.ini
setenv CCDC_CROSSMINER_DATABASE $CCDC_HOME/ccdc-data/crossminer/csd_pdb_crossminer.feat
setenv CCDC_CROSSMINER_FEATURE_DEFINITIONS $CCDC_HOME/ccdc-data/crossminer/feature_definitions
setenv GOLD_DIR $CCDC_HOME/ccdc-software/gold/GOLD

# Configure CONDA_ENVS_PATH
if { [info exists env(CONDA_ENVS_PATH)] } {
    setenv CONDA_ENVS_PATH $env(CONDA_ENVS_PATH):/vols/opig/apps/conda310/envs/CSD
} else {set
    setenv CONDA_ENVS_PATH /vols/opig/apps/conda310/envs/CSD
""

(Test code: https://github.com/matteoferla/Fragment-hit-follow-up-chemistry/blob/main/fragment_elaboration_scripts/mogul_zscore.py)