Install Ensembl's VEP v102 with local cache for running offline

ensembl_vep_102_with_offline_cache.md

Ensembl's VEP (Variant Effect Predictor) is popular for how it picks a single effect per gene as detailed here, its CLIA-compliant HGVS variant format, and Sequence Ontology nomenclature for variant effects.

Instead of the official instructions, we will use conda to install VEP and its dependencies. If you don't already have conda, install it into $HOME/miniconda3 as follows:

curl -sL https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh -o /tmp/miniconda.sh
sh /tmp/miniconda.sh -bfp $HOME/miniconda3

Add the conda bin folder into your $PATH so that all installed tools are accessible via command-line. You can also add this to your ~/.bashrc or ~/.profile for this to persist across logins:

export PATH=$HOME/miniconda3/bin:$PATH

Download and install VEP, its dependencies, and also samtools/bcftools/liftOver:

conda install -qy -c conda-forge -c bioconda -c defaults ensembl-vep==102.0 htslib==1.10.2 bcftools==1.10.2 samtools==1.10 ucsc-liftover==377

Download VEP's offline cache for GRCh38, and the reference FASTA:

mkdir -p $HOME/.vep/homo_sapiens/102_GRCh38/
rsync -avr --progress rsync://ftp.ensembl.org/ensembl/pub/release-102/variation/indexed_vep_cache/homo_sapiens_vep_102_GRCh38.tar.gz $HOME/.vep/
tar -zxf $HOME/.vep/homo_sapiens_vep_102_GRCh38.tar.gz -C $HOME/.vep/
rsync -avr --progress rsync://ftp.ensembl.org/ensembl/pub/release-102/fasta/homo_sapiens/dna_index/ $HOME/.vep/homo_sapiens/102_GRCh38/

(Optional) Download VEP's offline cache for GRCh37, and the reference FASTA which we must bgzip instead of gzip:

mkdir -p $HOME/.vep/homo_sapiens/102_GRCh37/
rsync -avr --progress rsync://ftp.ensembl.org/ensembl/pub/release-102/variation/indexed_vep_cache/homo_sapiens_vep_102_GRCh37.tar.gz $HOME/.vep/
tar -zxf $HOME/.vep/homo_sapiens_vep_102_GRCh37.tar.gz -C $HOME/.vep/
rsync -avr --progress rsync://ftp.ensembl.org/ensembl/pub/grch37/release-102/fasta/homo_sapiens/dna/Homo_sapiens.GRCh37.dna.toplevel.fa.gz $HOME/.vep/homo_sapiens/102_GRCh37/
gzip -d $HOME/.vep/homo_sapiens/102_GRCh37/Homo_sapiens.GRCh37.dna.toplevel.fa.gz
bgzip -i $HOME/.vep/homo_sapiens/102_GRCh37/Homo_sapiens.GRCh37.dna.toplevel.fa
samtools faidx $HOME/.vep/homo_sapiens/102_GRCh37/Homo_sapiens.GRCh37.dna.toplevel.fa.gz

Test running VEP in offline mode on a GRCh38 VCF:

curl -sLO https://raw.githubusercontent.com/Ensembl/ensembl-vep/release/102/examples/homo_sapiens_GRCh38.vcf
vep --species homo_sapiens --assembly GRCh38 --offline --no_progress --no_stats --sift b --ccds --uniprot --hgvs --symbol --numbers --domains --gene_phenotype --canonical --protein --biotype --tsl --pubmed --variant_class --shift_hgvs 1 --check_existing --total_length --allele_number --no_escape --xref_refseq --failed 1 --vcf --minimal --flag_pick_allele --pick_order canonical,tsl,biotype,rank,ccds,length --dir $HOME/.vep --fasta $HOME/.vep/homo_sapiens/102_GRCh38/Homo_sapiens.GRCh38.dna.toplevel.fa.gz --input_file homo_sapiens_GRCh38.vcf --output_file homo_sapiens_GRCh38.vep.vcf --polyphen b --af --af_1kg --af_esp --regulatory

Using conda to install VEP and its dependencies is not working. Please help me understand and solve this error:

$ conda install -qy -c conda-forge -c bioconda -c defaults ensembl-vep==102.0

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                                                                                                                                            

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package openssl conflicts for:
ensembl-vep==102.0 -> htslib -> openssl[version='>=1.1.1a,<1.1.2a|>=1.1.1g,<1.1.2a|>=1.1.1h,<1.1.2a|>=1.1.1i,<1.1.2a|>=1.1.1j,<1.1.2a|>=1.1.1k,<1.1.2a']
conda-forge/linux-64::python==3.9.6=h49503c6_0_cpython -> openssl[version='>=1.1.1k,<1.1.2a']The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.17=0
  - feature:|@/linux-64::__glibc==2.17=0
  - conda-forge/linux-64::python==3.9.6=h49503c6_0_cpython -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.17

@zjiang-lji were you able to solve this? I'm running into the same error and cannot figure it out

@zjiang-lji and @gauri-nagavkar - your errors indicate you are using conda with Python 3.9 which needs a newer version of openssl than VEP supports. Note that my instructions recommend installing conda with Python 3.7.

@zjiang-lji and @gauri-nagavkar - your errors indicate you are using conda with Python 3.9 which needs a newer version of openssl than VEP supports. Note that my instructions recommend installing conda with Python 3.7.

Like @zjiang-lji and @gauri-nagavkar I was stuck on an error with conda install for a very long time, but eventually fixed it. Here's what I did: Environment: ubuntu with python 3.9, anaconda already installed. I created a conda environment with conda create —name annotationEnv python=3.7 . Then I activated the environment with conda activate annotationEnv. Then within the environment I followed the gist verbatim including downloading and installing miniconda exporting it to be my path, conda install etc. It seems like a hack solution, but lots of answers online suggested anacondas within anacondas as the way forward with these sorts of errors

Always create a new environment for tools with too many dependencies. @gauri-nagavkar

In 2022 (as of today) the links that work for Grch37 (at least for me) are:

rsync -avr --progress rsync://ftp.ebi.ac.uk/ensemblorg/pub/release-102/variation/indexed_vep_cache/homo_sapiens_vep_102_GRCh37.tar.gz $HOME/.vep/

rsync -avr --progress rsync://ftp.ebi.ac.uk/ensemblorg/pub/grch37/release-102/fasta/homo_sapiens/dna/Homo_sapiens.GRCh37.dna.toplevel.fa.gz $HOME/.vep/homo_sapiens/102_GRCh37/

Hope this helps.

What would be the right version of python for installation ensembl-vep==108.1
Using individual env with python 3.7 or 3.8 all return

Package zlib conflicts for:
python=3.7 -> zlib[version='>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0|>=1.2.13,<1.3.0a0']
ensembl-vep==108.0 -> htslib -> zlib[version='1.2.11.*|>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0']The following specifications were found to be incompatible with your system:

• feature:/linux-64::__glibc==2.31=0
• python=3.7 -> libgcc-ng[version='>=11.2.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31

Thanks for the guide.
For people who might need a newer version of ensembl-vep, I suggest an alternative installation procedure that uses mamba.

1. Create a new env for ensembl-bep
   mamba create -n vep
2. I'd try installing without specifying the version to let mamba figure out the compatible versions among all the tools (then fix version later)
   mamba install -c conda-forge -c bioconda -c defaults ensembl-vep htslib bcftools samtools ucsc-liftover
3. Look for data in interest at this FTP site then download to local, please refer to the original post

For test running VEP in offline mode on a GRCh37 VCF (updated - 2023):

wget https://raw.githubusercontent.com/Ensembl/ensembl-vep/release/109/examples/homo_sapiens_GRCh37.vcf

The examples are on this site, but it is necessary to copy the raw data from the file
https://github.com/Ensembl/ensembl-vep/blob/release/109/examples/homo_sapiens_GRCh38.vcf

To work with samtools, the version needs to be updated by conda, currently, it is at version 1.17.

Use the link to install the latest version:
https://anaconda.org/bioconda/samtools