Skip to content

Instantly share code, notes, and snippets.

@migueldvb
Last active March 1, 2018 20:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save migueldvb/8aa5c1c192be0b6d29467b19724145e5 to your computer and use it in GitHub Desktop.
Save migueldvb/8aa5c1c192be0b6d29467b19724145e5 to your computer and use it in GitHub Desktop.
#!/usr/bin/python
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/cadcbin/ssos/ssosclf.pl?lang=en&object=7968&search=bynameCADC&epoch1=&epoch2=&eellipse=&eunits=arcseconds&extres=no&xyres=no'
html = requests.get(url)
soup = BeautifulSoup(html.content, 'html.parser')
table = soup.find_all('table')[0]
df = pd.read_html(str(table), header=0)[0]
# exclude WISE data
df = df[~(df['Telescope/Instrument'] == 'WISE')]
# convert to datetime objects
df['datetime'] = pd.to_datetime(df['Date/Time'], format="%Y-%m-%d %H:%M:%S")
# groupby UT day and aggregate
df = df.groupby([df['datetime'].dt.date]).agg({'Exptime': ['sum', 'count'],
'Telescope/Instrument': 'unique',
'Filter': 'unique',
'Image target': 'unique'
}).reset_index()
# flatten column hierarchy
df.columns = [' '.join(col).strip() for col in df.columns.values]
# cut on exposure time
df = df[(df['Exptime sum'] > 3600) & (df['Exptime count'] >= 5)]
pd.options.display.width = 500
print(df)
@migueldvb
Copy link
Author

Output

UT date     Exptime count       Instrument             Filter              target
2000-05-05     8102    11  [ESO-NTT/SUSI2]       [V, B, R, I]      [ELST-PIZARRO]
2007-07-23     4000    20   [ESO-NTT/EMMI]       [B, V, R, I]           [133P_EP]
2010-01-13     4800    16  [ESO-NTT/EFOSC]                [R]        [COMET-133P]
2010-01-14    10210    31  [ESO-NTT/EFOSC]  [R, B, GUNN_I, V]        [COMET-133P]
2010-01-15    10800    28  [ESO-NTT/EFOSC]  [R, B, GUNN_I, V]        [COMET-133P]
2011-03-05    14700    47  [ESO-NTT/EFOSC]  [R, B, GUNN_I, V]        [COMET-133P]
2011-03-06     9000    24  [ESO-NTT/EFOSC]  [R, B, GUNN_I, V]        [COMET-133P]
2011-03-07     8100    21  [ESO-NTT/EFOSC]  [R, B, GUNN_I, V]        [COMET-133P]
2012-05-31     6530    21  [ESO-NTT/EFOSC]  [R, V, B, GUNN_I]      [COMET-P133EP]
2013-07-10     3984    12       [HST/WFC3]           [F350LP] [133P-ELST-PIZARRO]
2013-08-08     3640    14  [ESO-VLT/FORS2]                [R]        [COMET_133P]
2013-09-04    10800    36  [ESO-NTT/EFOSC]                [R]           [133P_EP]
2014-01-02     5400     6        [INT/WFC]             [r, B]              [133P]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment