gistfile1.txt

#!/bin/bash
#
#SBATCH -J fmriprep
#SBATCH --time=48:00:00
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=8G
#SBATCH -p HPC  # Queue names you can submit to
# Outputs ----------------------------------
#SBATCH -o log/%A-%a.out
#SBATCH -e log/%A-%a.err
#SBATCH --mail-user=%u@domain.tld
#SBATCH --mail-type=ALL
# ------------------------------------------

# Specify directories and names, change these according to the version of fmriprep  
BIDS_DIR="$STUDY/rawdata"
DERIVS_DIR="$STUDY/derivatives/fmriprep-1.5.2"
IMAGE_NAME="poldracklab_fmriprep_1.5.2-2019-12-02-6afc1e0e96cd.simg"
IMAGE_DIR="$STUDY/code/singularity"

# Prepare some writeable bind-mount points, these are local to prevent network traffic issues
TEMPLATEFLOW_HOST_HOME=/tmp/templateflow-$(whoami)
FMRIPREP_HOST_CACHE=/tmp/fmriprep-$(whoami)
mkdir -p ${TEMPLATEFLOW_HOST_HOME}
mkdir -p ${FMRIPREP_HOST_CACHE}

# Make derivatives folder
mkdir -p ${DERIVS_DIR} 

# Make variable to represent freesurfer license (download from the freesurfer website)
export SINGULARITYENV_FS_LICENSE=$STUDY/code/singularity/license.txt

# Make variable for the templateflow bind-mount point inside the container
export SINGULARITYENV_TEMPLATEFLOW_HOME="/templateflow"

#Specify the singularity command which sets up the environment, binds the $STUDY path to the bind mount point inside the container, and specifies the singularity image
SINGULARITY_CMD="singularity run --cleanenv -B $STUDY:/projectFolder -B ${TEMPLATEFLOW_HOST_HOME}:${SINGULARITYENV_TEMPLATEFLOW_HOME} -B $L_SCRATCH:/work $IMAGE_DIR/$IMAGE_NAME"

# Parse the participants.tsv file and extract one subject ID from the line corresponding to this SLURM task.
subject=$( sed -n -E "$((${SLURM_ARRAY_TASK_ID} + 1))s/sub-(\S*)\>.*/\1/gp" ${BIDS_DIR}/participants.tsv )

# Compose the command line
cmd="${SINGULARITY_CMD} /projectFolder/rawdata /projectFolder/derivatives/fmriprep-1.5.2_tempflow-davidm participant --participant-label $subject  -w /work --output-spaces MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5 -vv --no-submm-recon --fs-license-file $SINGULARITYENV_FS_LICENSE --skip_bids_validation"

# Setup done, run the command
echo Running task ${SLURM_ARRAY_TASK_ID}
echo Commandline: $cmd
eval $cmd
exitcode=$?

# Output results to a table
echo "sub-$subject   ${SLURM_ARRAY_TASK_ID}    $exitcode" \
      >> ${SLURM_JOB_NAME}.${SLURM_ARRAY_JOB_ID}.tsv
echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode
exit $exitcode

re: templateflow issue, nothing immediately jumps out - its possible your cache is out of date, could you try rm -rf /tmp/templateflow-$(whoami) before rerunning?

Also, L32 seems problematic, as $STUDY is mounted as /projectFolder within the container. It should probably be changed to

export SINGULARITYENV_FS_LICENSE=/projectFolder/code/singularity/license.txt

@FidgetteSpinneur just pinging you here in case you didn't see my response

re: templateflow issue, nothing immediately jumps out - its possible your cache is out of date, could you try rm -rf /tmp/templateflow-$(whoami) before rerunning?

We'll try that, thanks!

Also, L32 seems problematic, as $STUDY is mounted as /projectFolder within the container. It should probably be changed to

export SINGULARITYENV_FS_LICENSE=/projectFolder/code/singularity/license.txt

Ah. Noted, even though we haven't had any issues with that particular part to my knowledge.

Thanks!

re: templateflow issue, nothing immediately jumps out - its possible your cache is out of date, could you try rm -rf /tmp/templateflow-$(whoami) before rerunning?

We changed the script so that an entirely new folder is used per run:

TEMPLATEFLOW_HOST_HOME=$(mktemp -t -d $(whoami).templateflow.XXXXXX)

Still same symptoms -- template files are created, but many of them (especially the pertinent ones) size 0. The trick with downloading the templates by invoking the TF API manually also failed in the same way, cf. nipreps/fmriprep#2030 (comment) .... any further clues?

Thanks!

@FidgetSpinneur this problem has been showing its face more frequently, so there have been some changes in developmental versions of templateflow to combat this.

First, install the developmental version of templateflow:

pip install --upgrade --pre templateflow

this should install version 0.6.0rc2 at the moment.

Next, set your templateflow home to the temporary folder, and ensure you are using the S3 bucket:

export TEMPLATEFLOW_HOME=/path/to/folder
export TEMPLATEFLOW_USE_DATALAD=false

Then, you can run the following command to force the skeleton to update:

python -c "import templateflow as tf; tf.update(local=True, overwrite=False)"

Finally, you can fetch the necessary templates:

python -c "import templateflow.api as tfapi; tfapi.get('MNI152NLin2009cAsym')"
# you can add more tfapi.get() calls if you are need other templates for fmriprep to run

Let me know if that helps.

First, install the developmental version of templateflow:
[...]
Let me know if that helps.

Yes! We're good! We'll keep an eye on the releases, I'm guessing once fMRIPrep includes TF >= 0.6.0rc2 we can skip this step?

Thanks a lot for your help!