TSTool example to download Colorado diversion data

-owf-example-tstool-diversions-co.md

TSTool example to download diversion time series data

by Steve Malers, Open Water Foundation, last updated 2016-02-11

Index of Open Water Foundation gists

Summary

This example shows how to use the TSTool software to retrieve Colorado ditch diversion time series data from various sources, and save in various formats.

Prerequisites

1. The example requires installation of the TSTool software
2. A basic understanding of the TSTool software is helpful.

Contents

Files in this gist are as follows:

1. -owf-example-tstool-diversions-co.md - example summary (this file)
2. co-download-diversions-dwr-ws-tsids.TSTool - a TSTool command file to download diversion data from Colorado Division of Water Resources web services and save in useful formats
3. KML-style.xml - provides styles for the KML output file
4. .gitattributes and .gitignore - these are used by the repository to allow proper editing of files on a file system

Data

Diversion data in this example are retrieved from the State of Colorado's HydroBase web services. Diversion data are recorded at structures (headgates, pipes, etc.), each of which have a "water district identifier" (WDID). A WDID is a combination of the two-digit water district number and structure identifier number, each part padded with zeros.

Process to Download Diversion Data using TSIDs

TSTool uses time series identifiers (TSIDs) to uniquely identify time series. Selecting the ColoradoWaterHBGuest datastore in TSTool will allow reading data from the CO DWR web services. Processing time series with specific TSIDs allows for fine control over the data that are downloaded. To download and save the diversion data using TSIDs:

1. Download the co-dwr-diversions-dwr-ws-tsids.TSTool command file from this gist and save on the computer where TSTool is installed.
2. Start TSTool
3. Open the command file with File / Open / Command file...
4. Run the commands by pressing the Run All Commands button. This will save several files in the same folder as the command file. This command file performs the following steps

Multiple output files are created to allow use of the data in Excel, other modeling tools, Google Earth (kml) and other software.

.gitattributes

#common settings that generally should always be used with your language specific settings

# Auto detect text files and perform LF normalization
# http://davidlaing.com/2012/09/19/customise-your-gitattributes-to-become-a-git-ninja/
* text=auto

#
# The above will handle all files NOT found below
#

# Documents
*.doc	 diff=astextplain
*.DOC	 diff=astextplain
*.docx diff=astextplain
*.DOCX diff=astextplain
*.dot  diff=astextplain
*.DOT  diff=astextplain
*.pdf  diff=astextplain
*.PDF	 diff=astextplain
*.rtf	 diff=astextplain
*.RTF	 diff=astextplain
*.md text
*.adoc text
*.textile text
*.mustache text
*.csv text
*.tab text
*.tsv text
*.sql text

# Graphics
*.png binary
*.jpg binary
*.jpeg binary
*.gif binary
*.tif binary
*.tiff binary
*.ico binary
*.svg binary
*.eps binary

.gitignore

# Ignore files in repository
!.gitignore
*.kml
*.csv
*.dv
*.json
*.xlsx

co-diversions-dwr-ws-tsids.TSTool

# This example queries diversion time series from Colorado Division of Water Resources web services
# - A few structures from the Tenmile Creek area in District 36
# - Monthly and daily total through headgate
# - Monthly infrequent report total through headgage
# - Save in several formats to illustrate hand-off to other tools
#
# The following are monthly total through headgate diversions
# 3600566 - CLINTON CREEK DITCH
3600566.DWR.DivTotal.Month~ColoradoWaterHBGuest
# 3600830 - SUPPLY CANAL NO 1 CLINTON
3600830.DWR.DivTotal.Month~ColoradoWaterHBGuest
#-----------------------------
# The following are daily total through headgate diversions
# 3600566 - CLINTON CREEK DITCH
3600566.DWR.DivTotal.Day~ColoradoWaterHBGuest
# 3600830 - SUPPLY CANAL NO 1 CLINTON
3600830.DWR.DivTotal.Day~ColoradoWaterHBGuest
#-----------------------------
# The following are infrequent monthly time series
# 3601085 - COPPER MTN WEST LAKE PIPELINE 1
3601085.DWR.IDivTotal.Month~ColoradoWaterHBGuest
# 3601087 - COPPER MTN WHEELER GULCH PIPELINE
3601087.DWR.IDivTotal.Month~ColoradoWaterHBGuest
# 3601088 - COPPER MTN TENMILE CR PIPELINE
3601088.DWR.IDivTotal.Month~ColoradoWaterHBGuest
# 3601089 - COPPER MTN BEELER LAKE PIPELINE
3601089.DWR.IDivTotal.Month~ColoradoWaterHBGuest
# 3601090 - COPPER MTN MCKENZIE LAKE PIPELINE
3601090.DWR.IDivTotal.Month~ColoradoWaterHBGuest
# 3601091 - COPPER MTN STAFFORD LAKE PIPELINE
3601091.DWR.IDivTotal.Month~ColoradoWaterHBGuest
#-----------------------------
# Save in various formats that may be useful
# - KML, can be used with Google Earth
WriteTimeSeriesToKml(TSList=AllMatchingTSID,TSID="*.*.*.Month.*",OutputFile="TenmileCreek-Diversions.kml",Name="Diversions",Description="Diversion structures",LongitudeProperty="longdecdeg",LatitudeProperty="latdecdeg",PlacemarkName="%L",PlacemarkDescription="%D",StyleFile="KML-Style.xml",StyleUrl="#exampleStyleMap")
# GeoJSON - can be used with GIS, web mapping
WriteTimeSeriesToGeoJSON(TSList=AllMatchingTSID,TSID="*.*.*.Month.*",OutputFile="TenmileCreek-Diversions.json",LongitudeProperty="longdecdeg",LatitudeProperty="latdecdeg")
# CSV file - generic format that loses metadata
WriteDelimitedFile(TSList=AllMatchingTSID,TSID="*.*.*.Month.*",OutputFile="TenmileCreek-Diversions.csv",DateTimeColumn="Date",Precision=2)
# Excel file - generic format that loses metadata
NewExcelWorkbook(OutputFile="TenmileCreek-Diversions.xlsx",Worksheets="DivTotal-Month,DivTotal-Day,IDivTotal-Month",KeepOpen=True)
WriteTimeSeriesToExcel(TSList=AllMatchingTSID,TSID="*.*.DivTotal.Month.*",MissingValue=Blank,OutputFile="TenmileCreek-Diversions.xlsx",Worksheet="DivTotal-Month",ExcelAddress="A1",KeepOpen=True,DateTimeColumn="Date",TimeColumn="Date")
WriteTimeSeriesToExcel(TSList=AllMatchingTSID,TSID="*.*.DivTotal.Day.*",MissingValue=Blank,OutputFile="TenmileCreek-Diversions.xlsx",Worksheet="DivTotal-Day",ExcelAddress="A1",KeepOpen=True,DateTimeColumn="Date",TimeColumn="Date")
WriteTimeSeriesToExcel(TSList=AllMatchingTSID,TSID="*.*.IDivTotal.Month.*",MissingValue=Blank,OutputFile="TenmileCreek-Diversions.xlsx",Worksheet="IDivTotal-Month",ExcelAddress="A1",KeepOpen=True,DateTimeColumn="Date",TimeColumn="Date")
CloseExcelWorkbook(OutputFile="TenmileCreek-Diversions.xlsx")
# DateValue file - TSTool format that keeps metadata
WriteDateValue(TSList=AllMatchingTSID,TSID="*.*.*.Month.*",OutputFile="TenmileCreek-Diversions.dv",IncludeProperties="*")

KML-Style.xml

<Style id="highlightPlacemark">
  <IconStyle>
    <Icon>
      <href>http://maps.google.com/mapfiles/kml/paddle/red-stars.png</href>
    </Icon>
  </IconStyle>
</Style>
<Style id="normalPlacemark">
  <IconStyle>
    <Icon>
      <href>http://maps.google.com/mapfiles/kml/paddle/wht-blank.png</href>
    </Icon>
  </IconStyle>
</Style>
<StyleMap id="exampleStyleMap">
  <Pair>
    <key>normal</key>
    <styleUrl>#normalPlacemark</styleUrl>
  </Pair>
  <Pair>
    <key>highlight</key>
    <styleUrl>#highlightPlacemark</styleUrl>
  </Pair>
</StyleMap>
<Style id="polygonFillStyle">
  <PolyStyle>
    <color>ff0000ff</color>
    <outline>0</outline>
    <fill>1</fill>
  </PolyStyle>
</Style>