Skip to content

Instantly share code, notes, and snippets.

@monocongo
monocongo / xarray_groupby_example.py
Created June 7, 2016 15:25
Example usage of xarray's split-apply-combine, using stack/groupby/apply/unstack
from __future__ import division
import logging
import numpy as np
import sys
import xarray
# set up a basic, global logger
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
# for each file in the list we concatenate it to the base dataset
output_file_name = root_file_name + '.v7_plus_first_guess.nc'
with xarray.open_dataset(base_file_name) as output_dataset:
# loop over the first guess files, concatenating (appending?) each to the base (output) dataset
for first_guess_file in first_guess_files:
# open the first guess dataset, concatenate it with the base (output) dataset
with xarray.open_dataset(first_guess_file, decode_times=False) as first_guess_dataset:
output_dataset = xarray.concat([output_dataset.precip[:, :, :], first_guess_dataset.p[0]], dim='time, lat, lon')
@monocongo
monocongo / palmer_numba_error_example.py
Created July 4, 2018 03:19
The below code causes an error with Numba. The stack trace is as follows:
import numba
import numpy as np
@numba.jit
def _pdsi_from_zindex(Z):
## INITIALIZE PDSI AND PHDI CALCULATIONS
# V is the sum of the Uw (Ud) values for the current and previous months of an
# established dry (wet) spell and is used in calculating the Pe value for a month.
@monocongo
monocongo / numba_error_gist.py
Last active July 4, 2018 03:32
The below code causes an error with Numba. The issue appears to be around the variables PV and V, i.e. if we comment out line 70 the error does not occur. This code is a minimal complete example of the issue that is occuring when executing a larger Python file from which this code was extracted.
import numba
import numpy as np
@numba.jit
def _pdsi_from_zindex(Z):
## INITIALIZE PDSI AND PHDI CALCULATIONS
# V is the sum of the Uw (Ud) values for the current and previous months of an
# established dry (wet) spell and is used in calculating the Pe value for a month.
@monocongo
monocongo / colab_drive_files_create.py
Created July 7, 2018 21:22
Create files in Google Colaboratory local environment from files stored in Google Drive
"""## Pull data files from Google Drive
Install PyDrive which will be used to access Google Drive and kick off the process to authorize the notebook running in the Google Colaboratory environment to touch our Drive files. When this cell executes it'll provide a link to authenticate into a Google Drive account to instatiate a PyDrive client. The Drive account that is selected should be one which has access to our all variables dataset file that we'll use for training/testing our model.
"""
#!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
@monocongo
monocongo / transform_to_supervised.py
Created July 26, 2018 12:58
Transform multivariate time series forecasting problems into supervised learning problems (Pandas DataFrame transformation)
import pandas as pd
def transform_to_supervised(df,
previous_steps=1,
forecast_steps=1,
dropnan=True):
"""
Transforms a DataFrame containing time series data into a DataFrame
containing data suitable for use as a supervised learning problem.
@monocongo
monocongo / docker_commands.md
Created June 14, 2019 15:34
Commands for facilitating copy of files into a Docker container

Find the Docker container ID (from localhost bash):

$ export DOCKER_CONTAINER=`sudo docker ps -a --no-trunc -q`

If more than a single Docker container is running then get the PyTorch container ID by grepping pytorch from the docker ps -a output, then using the final value of the output (the container's name) we can get the container ID via docker inspect:

$ sudo docker ps -a | grep pytorch
@monocongo
monocongo / out.txt
Created July 2, 2019 15:56
Output from attempted build of a distributable executable using PyInstaller on Linux (Ubuntu 18.04)
27 INFO: PyInstaller: 3.4
27 INFO: Python: 3.6.8
28 INFO: Platform: Linux-4.18.0-22-generic-x86_64-with-Ubuntu-18.04-bionic
28 INFO: wrote /home/james/git/deep_monitor/deep_monitor.spec
28 DEBUG: Testing for UPX ...
29 INFO: UPX is not available.
30 DEBUG: script: /home/james/git/deep_monitor/deep_monitor.py
30 INFO: Extending PYTHONPATH with paths
['/home/james/git/deep_monitor', '/home/james/git/deep_monitor']
30 INFO: checking Analysis
@monocongo
monocongo / conda_environment_management.md
Created July 2, 2019 19:48
Build *conda environment, save requirements.txt and environment.yml

Create and activate a new conda environment:

$ conda create -n myenv python=3 --yes
$ conda activate myenv

Add the conda-forge channel into the conda configuration:

$ conda config --add channels conda-forge

Add packages into the environment using both conda and pip:

@monocongo
monocongo / venv_basics.md
Created July 3, 2019 14:11
Python virtual environment basics

Configure Python development environment

Install pip, virtualenv, and virtualenvwrapper
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python3 get-pip.py
$ rm get-pip.py
$ sudo pip install virtualenv virtualenvwrapper
Update .bashrc