Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Query TESS data at MAST
# This script queries MAST for TESS FFI data for a single sector/camera/chip
# combination and downloads the data from the AWS public dataset rather than
# from MAST servers.
# Working with http://astroquery.readthedocs.io/en/latest/mast/mast.html
# Make sure you're running the latest version of Astroquery:
# pip install https://github.com/astropy/astroquery/archive/master.zip
from astroquery.mast import Observations
import boto3
# Query for observations in sector 1 (s0001), camera 1, chip 1 (1-1)
obsTable = Observations.query_criteria(obs_id="tess-s0001-1-1")
# Get the products associated with these observations
products = Observations.get_product_list(obsTable)
# Return only the calibrated FFIs (.ffic.fits)
filtered = Observations.filter_products(products,
productSubGroupDescription="FFIC",
mrp_only=False)
len(filtered)
# > 1282
# Enable 'cloud mode' for module which will return S3-like URLs for FITs files
# e.g. s3://stpubdata/tess/.../tess2018206192942-s0001-1-1-0120-s_ffic.fits
Observations.enable_cloud_dataset()
# Grab the S3 URLs for each of the observations
s3_urls = Observations.get_cloud_uris(filtered)
s3 = boto3.resource('s3')
# Create an authenticated S3 session. Note, download within US-East is free
# e.g. to a node on EC2.
s3_client = boto3.client('s3',
aws_access_key_id='YOURAWSACCESSKEY',
aws_secret_access_key='YOURSECRETACCESSKEY')
bucket = s3.Bucket('stpubdata')
# Just download a few of the files (remove the [0:3] to download them all)
for url in s3_urls[0:3]:
# Extract the S3 key from the S3 URL
fits_s3_key = url.replace("s3://stpubdata/", "")
root = url.split('/')[-1]
bucket.download_file(fits_s3_key, root, ExtraArgs={"RequestPayer": "requester"})
@pllim

This comment has been minimized.

Copy link

commented Jul 26, 2019

Need this at the beginning:

import os
os.environ['AWS_ACCESS_KEY_ID'] = 'myaccesskeyid'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'mysecretaccesskey'

And change to this:

print(len(filtered))  # If you just want to print and not store the value

s3_client is unused, so it should be deleted.

@pllim

This comment has been minimized.

Copy link

commented Jul 26, 2019

Instead of doing a url.replace(), why not use the include_bucket=False option when calling get_cloud_uris?

https://github.com/astropy/astroquery/blob/7f7b09eb8e05f875b1ea329e76262f2c31b54544/astroquery/mast/core.py#L1603

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.