Remove duplicate variants from VCF file

remove_VCF_duplicates.sh

#!/bin/bash

myvcf="$1"

# Print header
bcftools view -h -O v "$myvcf"

# Sort and remove duplicates per chromosome
for i in {1..22} MT X Y 
do
  bcftools view -H -r "$i" -O v "$myvcf" | sort -u -k2,2n -k4,4d -k5,5d
done

remove_VCF_duplicates.v2_non_bcftools.sh

#!/bin/bash

# PLEASE NOTE: Only use this version if you do not have bcftools available.
#              This method is much slower than the other method and only accepts
#              uncompressed VCF files. It works, but it's not recommended,
#              especially for large VCF files.

$myvcf="$1"

cat \
  <(grep '^#' "$myvcf") \
  <(grep -E '^[[:digit:]]' "$myvcf" | sort -u -k1,1n -k2,2n -k4,4d -k5,5d) \
  <(grep -E '^[[:alpha:]]' "$myvcf" | sort -u -k1,1d -k2,2n -k4,4d -k5,5d)

~README.md

Remove duplicate variants from a VCF file

A duplicate variant is when multiple records have the same CHROM, POS, REF, and ALT. This script will pick one of the duplicate variants and discard the rest. The variant that is picked is the one that comes first in sorting order.

Use remove_VCF_duplicates.sh if you have bcftools installed. If you don't have bcftools use remove_VCF_duplicates.non_bcftools.sh, but keep in mind that this method is much slower and only accepts uncompressed VCF files.

Input

vcf, vcf.gz, bcf, bcf.gz

Output

vcf

Example usage

./remove_VCF_duplicates.sh dups.vcf.gz > nodups.vcf

Requirements

bcftools 1.3+

does this work?

Sorry your scripts don't work.

Amazing!! Thanks bro ❤️ . In my old phone i need remade the VCF, because surely it have a windows format and dont detect, but now works nice.
I am not sure but i think that maybe could considerate use unix2dos input.vcf output.vcf for linux users