This file (isisdata_rclone_filter_from.lis) contains rclone filtering patterns that will limit the files that are downloaded from ISISDATA sources that are non-essential.

isisdata_rclone_filter_from.lis

#######################################################################
#  isisdata_rclone_filter_from.lis
#
# Global patterns that apply to all SPICE archives that are downloaded
# using rclone. In fact, be forewarned that any pattern here applies
# globally to all SPICE and USGS kernel archives.
#
# Should a pattern inadvertantly remove a required kernel, a specific
# include pattern can be added on a case-by-case basis.
#
# + /ck/former_versions/
#
# Be sure to specify any includes patterns before exclude patterns
# that would that match the included files.
#
# Note this initial version filters mission SPICE archives that
# contain the largest number/size of files currently unused by ISIS.
# It reduces the download size by about 1TB!
#
# It is highy recommended to create a new isisdata directory when
# using this filter for the first timel. I don't think rclone deletes
# existing files.
#
# Examples:
#
## Lists results of filter. Exclude the --filter-from arg for current
##  download comparisons.
# rclone ls --config rclone.conf  messenger_naifKernels:
# rclone ls --config rclone.conf  messenger_usgs:
#
# rclone ls --config rclone.conf  --filter-from isisdata_rclone_filter_from.lis  messenger_naifKernels:
# rclone ls --config rclone.conf  --filter-from isisdata_rclone_filter_from.lis messenger_usgs: | grep spk
# rclone ls --config rclone.conf  --filter-from isisdata_rclone_filter_from.lis messenger_naifKernels: | grep spk
#
##  With downloadIsisData, all data, current state.
# mkdir isisdatafull
# ./downloadIsisData all $PWD/isisdatafull --config=rclone.conf  -vv --log-file=isisdata_unfiltered.log
#
# Filtered with the contents of this file provided in --filter-from arg.
# mkdir isisdatafiltered
# ./downloadIsisData all $PWD/isisdatafiltered --config=rclone.conf  --filter-from=isisdata_rclone_filter_from.lis -vv --log-file=isisdata_filtered.log
#
# References:
#    https://rclone.org/filtering/
#    https://docs.amazonaws.cn/en_us/snowball/latest/developer-guide/using-adapter-cli.html
#
#_History 2022-11-22 Kris J. Becker, Original Version
#_History 2022-12-08 Kris J. Becker, Updated documentation
#_History 2023-01-03 Kris J. Becker, Added removal of unused Kaguya
#                       kernels from the USGS repo
#######################################################################

#######################################################################
# NOTE: Typically, all external SPICE sources have root=./kernels/.
#       USGS sources are (essentially) root=$ISISDATA/{mission"}/.
#       USGS patterns appear as "- /kernels/dsk/bennu_g_*_v020.*".
#######################################################################

#######################################################################
# WARNING: This filters out all directories at root=./kernels that
#          contain ./kernels/ek and any/all zzarchive subdirectories.
#          If you absolutely have to have one, add explicit! include
#          pattern prior to the exclude filters.
#######################################################################
- /ek/**
- zzarchive/**


#######################################################################
# Cassini SPICE kernel download file name include/exclude patterns
#######################################################################
- /ck/S09_trend.txt
- /ck/cas_caps_*.bc
- /ck/cas_cda_*.bc
- /ck/cas_fix_*.bc
- /ck/cas_lemms_*.bc
- /ck/cas_skr_*.bc
- /ck/day118_combined.*
- /ck/probe_deploy_*.*
- /ck/probe_track_*.*
- /ck/s38_scr_changes_*.*
- /ck/s38_underground_mimi_*.*
- /ck/*_*r*.pdf
- /ck/*_*r*.txt
- /ck/*.lbl
- /ck/*_ISS_JPGs.tar.gz

- /ik/release.??/**

- /fk/Archive/**
- /fk/release.??/**

- /pck/*_Nav.tpc
- /pck/cpck_rock_*.*

- /spk/*.bsp.lbl
- /spk/????????_PE_?????_?????.bsp
- /spk/????????_RE_?????_?????.bsp
- /spk/????????_SE_?????_?????.bsp
- /spk/????????_SK_?????_?????.bsp

#######################################################################
# Chandrayaan1 SPICE kernel download file name include/exclude patterns
#######################################################################
- /ck/isro_21_day_att_*.bc
- /ck/isro_21_day_eph_*.bc
- /ck/isro_full_mission_predict_*.bc

- /fk/former_versions/CH1_V??.TF

- /ik/former_versions/CH1_*.TI

- /spk/isro_21_day_att_*.bsp
- /spk/isro_21_day_eph_*.bsp
- /spk/isro_full_mission_predict_*.bsp
- /spk/former_versions/isro_full_mission_predict_00.bsp

#######################################################################
# Dawn SPICE kernel download file name include/exclude patterns
#######################################################################
- /ck/dawn_hybrid_testing.bc
- /ck/dawn_ql_*.bc
- /ck/dawn_sa_*.bc
- /ck/dawn_sc_pred_*.bc
- /ck/dawn_sc_fstb3*.bc
- /ck/dawn_vir_?????????_?.bc

- /dsk/old_versions/**
- /dsk/dawn_ceres_*.bds

- /sclk/stdout.tsc

- /spk/dawn_*.bsp.lbl
- /spk/dawn_*_hamo.bsp
- /spk/dawn_dsn_*.bsp
- /spk/dawn_grv_*.bsp
- /spk/dawn_m_*.bsp
- /spk/dawn_mga_*.bsp
- /spk/dawn_p_*.bsp
- /spk/dawn_ref_*.bsp
- /spk/dawn_seq_*.bsp
- /spk/dawn_ssd_*.bsp

#######################################################################
# Juno SPICE kernel download file name include/exclude patterns
#######################################################################
- /ck/juno_*.lbl
- /ck/juno_chua_ali_*.bc
- /ck/juno_chua_rec_*.bc
- /ck/juno_chub_ali_*.bc
- /ck/juno_chub_rec_*.bc
- /ck/juno_chuc_ali_*.bc
- /ck/juno_chuc_rec_*.bc
- /ck/juno_chud_ali_*.bc
- /ck/juno_chud_rec_*.bc
- /ck/juno_mobib_ali_*.bc
- /ck/juno_mobib_rec_*.bc
- /ck/juno_mobob_ali_*.bc
- /ck/juno_mobob_rec_*.bc
- /ck/juno_sc2fgm?b_zero_*.bc
- /ck/juno_uvs_rec_*.bc

#######################################################################
# Kaguya SPICE kernel download file name include/exclude patterns.
# These USGS kernels are not used at all (~583 MB)
#######################################################################
- /kernels/ik/original/**
- /kernels/sclk/original/**
- /kernels/spk/original/**

#######################################################################
# OSIRIS-REx SPICE kernel download file name include/exclude patterns
# See https://naif.jpl.nasa.gov/pub/naif/pds/pds4/orex/
#######################################################################
- collection_spice_kernels_inventory_*.csv
- collection_spice_kernels_*.xml

- /ck/orx_ola*.*
- /ck/orx_sa_rel_*.*
- /ck/orx_*.xml

# This selects the 44cm GSD global and the 5cm Nightingale TAG site
# models and eliminates the rest. v021 models are consistent with
# the bennu_v17.tpc.
+ /dsk/bennu_g_00400mm_alt_ptm_0000n00000_v021.bds
+ /dsk/bennu_l_00050mm_alt_dtm_5595n04217_v021.bds
- /dsk/bennu_g*.*
- /dsk/bennu_l*.*

#-> This removes all the v020 shape models from USGS as they are
# incompatible with bennu_v17.tpc.
- /kernels/dsk/bennu_g_*_v020.*
- /kernels/dsk/bennu_l_*_v020.*

- /fk/orx_*.xml

- /ik/orx_*.xml

- /mk/orx_*.*

- /pck/bennu_*.xml
- /pck/pck*.xml

- /sclk/orx_sclkscet_*.xml

- /spk/orx_*.xml

#######################################################################
# Smart1 SPICE kernel download file name include/exclude patterns
#######################################################################
- /ck/former_versions/**
- /fk/former_versions/**
- /ik/former_versions/**
- /lsk/former_versions/**
- /pck/former_versions/**
- /sclk/former_versions/**
- /spk/former_versions/**

- /orbnum/**

#######################################################################
# MESSENGER SPICE kernel download file name include/exclude patterns
#
# MESSENGER SPK kernels are cummulative - each successive kernel
# contains all of the previously released kernels. It was managed
# during the mission by creating symbolic links of earlier existing
# SPK kernels to the the most recent downloaded/installed kernel.
# Now that the mission is over, this is no longer necessary.
#
# In addition, AWS S3 servers does not directly support symbolic links
# instead making copies of the linked file. This creates gigabytes
# of a single file. All symbolic links on AWS ISIS data should be
# removed. Just doing that makes this whole filter unneccessary.
#######################################################################
- /ck/msgr_????.bc
- /ck/mdis_hdr_sc_??????_??????v?.bc

- /ck/mdis_gm_nl??????_??????v?.bc
- /ck/mdis_gm_??????_??????v?.bc

+ /ck/msgr_mdis_gm??????_15????v?.bc
- /ck/msgr_mdis_gm??????_??????v?.bc

- /ck/mdis_sc_??????_??????v?.bc
- /ck/msgr_mdis_sc_??????_??????v?.bc
- /ck/msgr_mdis_sc_??????_??????.bc

+ /ck/msgr_mdis_sc040812_150430v?.bc
- /ck/msgr_mdis_sc??????_??????v?.bc

- /spk/msgr_antenna_v???.*
- /spk/msgr_de???_de???s.*

- /spk/msgr_??????_??????_recon_gsfc_?.*
- /spk/msgr_??????_??????_??????_od???sc*.*
- /spk/msgr_??????_??????_??????.*

# These are USGS produced SPK kernels that are currently being used as
# the full mission SPK ephemeris. This addresses symbolically linked
# files created as actual files on AWS S3. These patterns eliminate
# 72 files.
#
#->  AWS S3 does not support symbolic links!
#->  They are duplicate copies of the linked file on AWS S3.
#
# Also include any forthcoming kernels. See the MESSENGER kernel DB
# file $ISISDATA/messenger/kernels/spk/kernels.0184.db.
- /kernels/spk/msgr_20040803_20150501_od430sc_0.bsp
+ /kernels/spk/msgr_????????_????????_od431sc_[2-9].bsp
+ /kernels/spk/msgr_????????_????????_od4[3-9]?sc_?.bsp
+ /kernels/spk/msgr_????????_????????_od[5-9]??sc_?.bsp

# Exclude all the rest
- /kernels/spk/msgr_????????_????????_od430sc_?.bsp
- /kernels/spk/msgr_????????_????????_od???sc_?.*

Isis map aws s3