Skip to content

Instantly share code, notes, and snippets.

@matteoferla
Last active June 1, 2024 09:13
Show Gist options
  • Save matteoferla/2001c5578c24651bb04ef910004c7d52 to your computer and use it in GitHub Desktop.
Save matteoferla/2001c5578c24651bb04ef910004c7d52 to your computer and use it in GitHub Desktop.
I am going to write a comment for every damn IT issue/fix I get...

This is an experiment. Is a GitHub Gist comment a good way to log publically how I fixed an installation or similar issue?

See comments

@matteoferla
Copy link
Author

PyMOL does not install on Python 3.11 or above. That I know of there is nothing in 3.11 that breaks ported 2.7-code more than say 3.10. So

conda install -c conda-forge -c schrodinger freetype glew h5py libgcc-ng libholoplaycore libnetcdf libpng libstdcxx-ng libxml2 mtz2ccp4_px numpy
conda install -c conda-forge -c schrodinger --no-deps pymol

@matteoferla
Copy link
Author

Zombie threads kill my webserver which is under a bogstandard Apache reverse proxy, so this root crontab restarts its service if the /status (/api for a Jupyter) fails on the original port:

if [[ $(curl -s -o /dev/null -w "%{http_code}" http://localhost:๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ/status) != "200" ]]; then systemctl restart ๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ.service; fi

I have not got it working inline on a cron job, so I have it in a root owned (700) script in /opt.

No Zombies!

@matteoferla
Copy link
Author

I got asked this, which has become second nature but is clearly an assumed thing.

Jupyter displays an rdkit Chem.Mol as <rdkit.Chem.rdchem.Mol object at 0x11c847a70>.

To fix import from rdkit.Chem.Draw import IPythonConsole

@matteoferla
Copy link
Author

Installing PLIP & OpenBabel in some circumstances:

conda install -c conda-forge openbabel lxml
conda install -c conda-forge --no-deps plip

@matteoferla
Copy link
Author

matteoferla commented Oct 23, 2023

How to check for anything suspicious on my homeserver. This will show the countries of logins:
zgrep sshd /var/log/auth.log* | grep -F 'Accepted' | sed -re 's/.*from ([^ ]+).*/\1/' | sort -u | xargs -n 1 geoiplookup
This does not however mean the auth.log was not tampered with.

@matteoferla
Copy link
Author

This is an answer to a Q, so I am not sure of the exact cause, but I think a badly timed control-X on a conda install.
If odd errors happen on install, clean the index! conda clean --index-cache -y

@matteoferla
Copy link
Author

The cluster I use has Nvidia driver 510, which does not accept cuda toolkit 12. The subversion of 11 is confusing as its cuda-toolkit=11.8.0 pytorch-cuda=11.3

@matteoferla
Copy link
Author

It took me a minute or two to find my normal conda environment variables โ€”I had put some in https://www.blopig.com/blog/2023/03/the-ultimate-modulefile-for-conda/

export CONDA_YES=true
export CONDA_CHANNELS="conda-forge,nvidia" #comma not space!
export CONDA_SOLVER=libmamba

@matteoferla
Copy link
Author

matteoferla commented Nov 27, 2023

Everyone hates tensorflow, but alphafold2 uses it so we are stuck with it.
Max Python is still 3.9. I have tried higher โ€”not worth the fight. Colab seems to make Py 3.10 + TF + Cuda11 (cf. ColabFold), however the conda frozen yaml does not replicate on my systsem.
Ironically some dependencies no longer support CentOS 7 (glibc2.17 issue).
The internet is full of obsolete info, these are current:

  • Pip tensorflow>=2.12 includes tensorflow-gpu functionality, hence why the latter is stuck at older versions.
  • cuDNN is pip installed with jax[cuda12_pip]
  • conda/dnf cuda-compat fixes the drama of 11 vs 12 drivers, mostly. These needs to be set however as always:

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:/usr/local/cuda/compat:/.singularity.d/libs:/usr/lib64/nvidia:/usr/lib/nvidia

In Numpy >1.20 a few aliases were deprecated, but can be undone:

import numpy as np
np.object = object
np.bool = bool
np.int = int
np.typeDict = np.sctypeDict

@matteoferla
Copy link
Author

I keep having issues with libmamba solver for conda. This time its actually not libmamba, but libmambapy.
libmambapy is the Pythonic reimplementation of the C++ library libmamba.

Error while loading conda entry point: conda-libmamba-solver (/data/xchem-fragalysis/mferla/waconda/envs/cuda118/lib/python3.11/site-packages/../../libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /data/xchem-fragalysis/mferla/waconda/lib/python3.11/site-packages/libmambapy/../../../libmamba.so.2))

Unsurprisingly, this was not directly installed by me, but by jax. Ie. a random config requirement that really ought to not be there โ€”the award for that goes to tensorflow whose requirement are effectly a conda environment export.

To fix this I did:

ldd --version

2.34
and then

CONDA_OVERRIDE_GLIB=2.34 conda install --solver classic libmambapy --force-reinstall

@matteoferla
Copy link
Author

matteoferla commented Dec 20, 2023

Installation of MaSIF via conda. ๐Ÿšง Not finished.
GitHub: https://github.com/lpdi-epfl/masif has no conda, because of its dozen requirements, some old.

# ## Start with creation
export CONDA_YES=true
# I believe this ancient prepandemic python is due to pymesh:
conda create -n masif python=3.6
conda activate masif
# I don't want stuff in $HOME/.local and PIP_NO_USERBASE:
export PYTHONUSERBASE=$CONDA_PREFIX
conda env config vars set PYTHONUSERBASE=$PYTHONUSERBASE

# ## APBS
#conda install -c conda-forge apbs #No: this does not install multivalue
wget https://github.com/Electrostatics/apbs/releases/download/v3.4.1/APBS-3.4.1.Linux.zip -O /tmp/APBS-3.4.1.Linux.zip
unzip /tmp/APBS-3.4.1.Linux.zip -d /tmp/APBS-3.4.1.Linux
rsync -ar /tmp/APBS-3.4.1.Linux/ $CONDA_PREFIX
export APBS_BIN=$CONDA_PREFIX/bin/apbs
conda env config vars set APBS_BIN=$APBS_BIN
export MULTIVALUE_BIN=$CONDA_PREFIX/share/apbs/tools/bin/multivalue
conda env config vars set MULTIVALUE_BIN=$MULTIVALUE_BIN

# ## MSMS
conda install -c bioconda msms
export MSMS_BIN=$CONDA_PREFIX/bin/msms
export PDB2XYZRN=$CONDA_PREFIX/bin/pdb_to_xyzrn
conda env config vars set MSMS_BIN=$MSMS_BIN
conda env config vars set PDB2XYZRN=$PDB2XYZRN

# ## PDB2PQR
conda install -c schrodinger pdb2pqr
export PDB2PQR_BIN=$CONDA_PREFIX/bin/pdb2pqr
conda env config vars set PDB2PQR_BIN=$PDB2PQR_BIN

# ## Open3D
conda install -c open3d-admin open3d

# ## Reduce
conda install -c ostrokach-forge reduce
conda env config vars set PATH=$PATH:$CONDA_PREFIX/bin/reduce
# forgive me for this transgression:
export PATH=$PATH
# reduce dict needs adding:
curl https://github.com/rlabduke/reduce/blob/master/reduce_wwPDB_het_dict.txt -o $CONDA_PREFIX/var/reduce_wwPDB_het_dict.txt
export REDUCE_HET_DICT=$CONDA_PREFIX/var/reduce_wwPDB_het_dict.txt

# ## PyMesh
conda install -c conda-forge pymesh2
export PYMESH_PATH=$CONDA_PREFIX/lib/python3.6/site-packages/pymesh
conda env config vars set PYMESH_PATH=$PYMESH_PATH

# MaSIF itself
git clone https://github.com/lpdi-epfl/masif $CONDA_PREFIX/masif_repo
export PATH=$CONDA_PREFIX/masif_repo/data/masif:$PATH

NB. masif_repo/requirements.txt is not a valid Python pip requirements.txt it's just rubbish.

The repo is not installed but simply dumped. That would need to be fixed especially the shell scripts

I have not looked into why conda install -c conda-forge pymesh2 seems fine while this https://github.com/PyMesh/PyMesh is lengthy

@matteoferla
Copy link
Author

matteoferla commented Dec 21, 2023

Installing ColabFold with a 11.x driver not compatible with cuda-compat in December 2023 (in singularity without root conda-toolkit where /.singularity.d/libs do not match).

conda create -n colabfold python=3.10
conda activate colabfold
conda env config vars set PYTHONUSERBASE=$CONDA_PREFIX
conda install ipykernel
conda install git
$CONDA_PREFIX/bin/python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
conda install -c conda-forge python=3.10 cudnn==8.2.1.32 cudatoolkit==11.6.0 openmm==7.7.0 pdbfixer -y
$CONDA_PREFIX/bin/python -m pip install --upgrade pip
$CONDA_PREFIX/bin/python -m pip install --no-warn-conflicts "colabfold[alphafold-minus-jax] @ git+https://github.com/sokrypton/ColabFold" tensorflow==2.12.0
$CONDA_PREFIX/bin/python -m pip install https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.25+cuda11.cudnn82-cp310-cp310-manylinux2014_x86_64.whl
$CONDA_PREFIX/bin/python -m pip install jax==0.3.25 chex==0.1.6 biopython==1.79
$CONDA_PREFIX/bin/python -m pip install -y jax==0.3.25 chex==0.1.6
conda install -c "nvidia/label/cuda-11.6.0" cuda-toolkit
export LD_LIBRARY=$CONDA_PREFIX
rm -rf $CONDA_PREFIX/lib/python3.10/site-packages/nvidia

The $LD_LIBRARY is not assigned /.singularity.d/libs as those mismatch in DLS.
Normally the values /usr/local/cuda/compat, /usr/local/nvidia/lib, /usr/local/nvidia/lib64 and $CONDA_PREFIX/lib may be used. Some instructions add /proc/driver/nvidia, which is 100% comedically wrong. Specifying a path to specific packages might do something but effort (eg. $CONDA_PREFIX/lib/python3.11/site-packages/nvidia/cudnn/lib.

The removal of the package by rm is because CuDNN is being installed via multiple packages: use find $CONDA_PREFIX -name "libdevice.*" to find the conflict locations.

The problem child is tensorflow, to test it:

import os

#os.environ['LD_LIBRARY_PATH'] = os.environ['CONDA_PREFIX']
import tensorflow as tf
import sys, os

print(sys.version_info)
print(tf.config.list_physical_devices('GPU'))
print('CUDA build:', tf.test.is_built_with_cuda())
print("CUDA version:", tf.sysconfig.get_build_info()["cuda_version"])
print("cuDNN version:", tf.sysconfig.get_build_info()["cudnn_version"])
print("CUDA library paths:", tf.sysconfig.get_lib())
a = tf.constant([[1.0, 2.0], [3.0, 4.0]])
b = tf.constant([[1.0, 1.0], [0.0, 1.0]])
c = tf.matmul(a, b)
print(c)

Note, Pytorch is easier to please while tensorflow does seem to not respect $LD_LIBRARY_PATH. It is possible to have a env with a working pytorch but broken tensorflow. In which case loading torch then tf will work, so be aware of that.

@matteoferla
Copy link
Author

Colab installation of Pymol

import sys
!wget -qnc https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
!bash Mambaforge-Linux-x86_64.sh -bfp /usr/local
sys.path.append('/usr/local/lib/python3.10/site-packages')
!clear
!/usr/local/condabin/mamba install -y conda-forge::pymol-open-source

Works with

import os
os.environ['LD_LIBRARY_PATH'] = '/usr/local/lib:/usr/local/lib64:/usr/local/bin/openssl:/usr/local/nvidia/lib:/usr/local/nvidia/lib64'

import pymol2

@matteoferla
Copy link
Author

git clone https://github.com/๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ/๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ.git gave: 'file:///lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0'
Fixed with conda install anaconda::openldap

@matteoferla
Copy link
Author

Pyrosetta cannot read mmCIF/PDBx files. The following seems to work: pose: pyrosetta.Pose = prc.import_pose.pose_from_file('crysalin_lattice.cif', read_fold_tree=True, type=prc.import_pose.FileType.CIF_file) but it does not.
However, atom indices are ignored so I will just convert into PDB with over >9_999 atoms. ๐Ÿคท

@matteoferla
Copy link
Author

Installation of the RDKit Postgres catridge

conda create -n py310 python==3.10 ipykernel -y
conda activate py310
conda env config vars set PYTHONUSERBASE=$CONDA_PREFIX
export PYTHONUSERBASE=$CONDA_PREFIX

# https://www.rdkit.org/docs/Cartridge.html
conda install -y -c conda-forge -c rdkit rdkit-postgresql postgresql
# lets be a traditionalist
conda env config vars set PGDATA=$CONDA_PREFIX/var/lib/pgsql/data
export PGDATA=$CONDA_PREFIX/var/lib/pgsql/data
# starting rdkit db
initdb
echo "shared_buffers = 2048MB
work_mem = 128MB
synchronous_commit = off
full_page_writes = off" >> "$PGDATA/postgresql.conf"
pg_ctl -l $HOME2/logs/postgres.log start
createdb test
# dropdb test
psql -c 'create extension rdkit'test
psql -c 'create table raw_data (id SERIAL, smiles text, emol_id integer, parent_id integer)' test
...

@matteoferla
Copy link
Author

matteoferla commented Feb 28, 2024

Getting the Python API of the CCDC CSD working is not too straightforward โ€”the older documentation does not help.
The installation of CCDC is obviously needed so: ./CSDInstallerOffline-2023.3-linux --accept-licenses install uk.ac.cam.ccdc --root $CCDC_HOME -c

cleaner environment variables:

export CCDC_HOME=$HOME/CCDC
conda activate base
# conda install --channel=https://conda.ccdc.cam.ac.uk csd-python-api # does not work for me
cp -r $CCDC_HOME/ccdc-software/csd-python-api.minoconda $CONDA_PREFIX/envs/CDS
conda activate CSD
# following is an alternative to `$CCDC_HOME/ccdc-utilities/software-activation/bin/ccdc_activator -a -k ๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ`
conda env config vars set CCDC_LICENSING_CONFIGURATION='la-code;๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ'
conda env config vars set CCDC_TOOLKIT_ASER_DATABASE=$CCDC_HOME/ccdc-data/csd/as544be
conda env config vars set CCDC_TOOLKIT_SQLITE_DATABASE=$CCDC_HOME/ccdc-data/csd/as544be_ASER.sqlite
conda env config vars set CCDC_ISOSTAR_DATA_DIRECTORY=$CCDC_HOME/ccdc-data/isostar
conda env config vars set CCDC_MOGUL_DATA=$CCDC_HOME/ccdc-data/mogul
conda env config vars set CCDC_MOGUL_INITIALISATION_FILE=$CONDA_PREFIX/lib/python3.9/site-packages/ccdc/parameter_files/mogul.ini
conda env config vars set CCDC_CROSSMINER_DATABASE=$CCDC_HOME/ccdc-data/crossminer/csd_pdb_crossminer.feat
conda env config vars set CCDC_CROSSMINER_FEATURE_DEFINITIONS=$CCDC_HOME/ccdc-data/crossminer/feature_definitions
conda env config vars set GOLD_DIR=$CCDC_HOME/ccdc-software/gold/GOLD
# always good:
conda env config vars set PYTHONUSERBASE=$CONDA_PREFIX
conda deactivate
conda activate CSD
# this applied to my system only (Mac OS), was fine in Fedora
# First error: `Symbol not found: _png_create_info_struct` (Mac OS 14)
conda install anaconda::libpng
# also for my notebook
conda install ipykernel

@matteoferla
Copy link
Author

Module file for CCDC CSD.

#%Module1.0
##
## csd23-module
## Module file for setting up environment for CSD23
## Written by Matteo Ferla โ€” matteo.ferla@stats.ox.ac.uk

proc ModulesHelp { } {

    puts stderr "\tAdds CSD23 CCDC Utilities and Software to your environment variables"
}

module-whatis "Adds CSD23 CCDC Utilities and Software to your environment variables"
set CCDC_HOME /vols/opig/apps/CSD23
prepend-path PATH /vols/opig/apps/CSD23/ccdc-utilities
prepend-path PATH /vols/opig/apps/CSD23/ccdc-software
setenv CCDC_LICENSING_CONFIGURATION "la-code;๐Ÿ‘พ๐Ÿ‘พ๐Ÿ‘พ"
setenv CCDC_HOME /vols/opig/apps/CSD23
setenv CCDC_TOOLKIT_ASER_DATABASE $CCDC_HOME/ccdc-data/csd/as544be
setenv CCDC_TOOLKIT_SQLITE_DATABASE $CCDC_HOME/ccdc-data/csd/as544be_ASER.sqlite
setenv CCDC_ISOSTAR_DATA_DIRECTORY $CCDC_HOME/ccdc-data/isostar
setenv CCDC_MOGUL_DATA $CCDC_HOME/ccdc-data/mogul
setenv CCDC_MOGUL_INITIALISATION_FILE $CCDC_HOME/lib/python3.9/site-packages/ccdc/parameter_files/mogul.ini
setenv CCDC_CROSSMINER_DATABASE $CCDC_HOME/ccdc-data/crossminer/csd_pdb_crossminer.feat
setenv CCDC_CROSSMINER_FEATURE_DEFINITIONS $CCDC_HOME/ccdc-data/crossminer/feature_definitions
setenv GOLD_DIR $CCDC_HOME/ccdc-software/gold/GOLD

# Configure CONDA_ENVS_PATH
if { [info exists env(CONDA_ENVS_PATH)] } {
    setenv CONDA_ENVS_PATH $env(CONDA_ENVS_PATH):/vols/opig/apps/conda310/envs/CSD
} else {set
    setenv CONDA_ENVS_PATH /vols/opig/apps/conda310/envs/CSD
""

(Test code: https://github.com/matteoferla/Fragment-hit-follow-up-chemistry/blob/main/fragment_elaboration_scripts/mogul_zscore.py)

@matteoferla
Copy link
Author

RDKit to OpenBabel
[10:11:18] Cannot convert ' 3.' to unsigned int on line 4
This is caused by no title line in the Mol_block.

smiles = 'CN(C)CCCN1c2ccccc2Sc3c1cccc3'
promazine = AllChem.AddHs(Chem.MolFromSmiles(smiles))
AllChem.EmbedMolecule(promazine)
# REQUIRED: promazine.SetProp('_Name', 'promazine')
conv = ob.OBConversion()
conv.SetInAndOutFormats('mol', 'mol')
obmol = ob.OBMol()
conv.ReadString(obmol, Chem.MolToMolBlock(promazine))
roundtrip = Chem.MolFromMolBlock(conv.WriteString(obmol).strip())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment