Skip to content

Instantly share code, notes, and snippets.

View seumasmorrison's full-sized avatar

James Morrison seumasmorrison

View GitHub Profile
@seumasmorrison
seumasmorrison / pandas-example.py
Created August 17, 2012 10:21
Sample input data and python code for Pandas time-series problem
import pandas as pd
data_frame = pd.io.parsers.read_csv('time-series.csv',index_col = 0, names=['index','heave'])
_max = [[5, 259.0]]
indexes = [x[0] for x in _max ]
index = data_frame.index
timestamps = [index[z] for z in indexes]
#['2011-12-30 00:00:04']
print data_frame.ix[timestamps]
# heave
@seumasmorrison
seumasmorrison / SyntaxErrors
Created September 6, 2012 14:31
python3 matplotlib install - output
/usr/local/lib/python3.2/subprocess.py:389: RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads.
"program uses threads.", RuntimeWarning)
basedirlist is: ['/usr/local', '/usr']
============================================================================
BUILDING MATPLOTLIB
matplotlib: 1.2.x
python: 3.2.3 (default, Sep 6 2012, 14:51:03) [GCC 4.1.2
20080704 (Red Hat 4.1.2-46)]
platform: linux2
@seumasmorrison
seumasmorrison / log_to_dataframe.py
Created November 15, 2012 13:31
Cumulus log files to Pandas Dataframe
# Python modules for taking Cumulus ( sandaysoft.com ) logs into a Pandas
# ( pandas.pydata.org ) DataFrame, and using the date and time columns to
# create a DateTimeIndex for the DataFrame and sort upon it.
import pandas as pd
import os
import glob
from datetime import datetime
log_path = 'D:\\log_files_folder'
@seumasmorrison
seumasmorrison / historical_concat.py
Last active October 12, 2015 21:19
Script for producing date sorted Pandas DataFrames and writing Excel ( xlsx ) files from historical files which can be resampled ( 30 minutes specified ). Used with his/hiw files from Datawell Directional Waverider MK II/III produced by RFBuoy v2.1.27
import os
import glob
import pandas as pd
from datetime import datetime
# Example below assumes following folder hierarchy D:\Buoy_Data\Buoy_Name\Year\Month\
buoy_names = ['Buoy_Name']
buoy_path = 'D:\\Buoy_Data\\'
his_columns = ['date_time', 'Tp', 'dirp', 'sprp', 'Tz', 'Hm0', 'TI', 'T1', 'Tc',
{
"metadata": {
"name": "Basemap HebMarine"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
@seumasmorrison
seumasmorrison / arcgis_raster_to_xyz
Created January 16, 2013 14:33
Used for producing xyz bathymetry data from ArcGIS 10 ouputs, with the intention of importing into DHI Mike21.
# Initial data obtained from hdr file produced with flt file when running
# Raster to Float in ArcToolBox and numpy file ( npy ) from RasterToNumpyArray
# in ArcPy
import numpy as np
np.load('numpy_bathy.npy')
cell_size = 3
no_of_columns = 18325
no_of_rows = 13303
lower_left_x = 607512
lower_left_y = 6455598
@seumasmorrison
seumasmorrison / iter_loadtxt.py
Created January 18, 2013 10:35
Fast and memory efficient solution for loading well formed large csv files
# Code copied from Joe Kington's StackOverflow answer
# http://stackoverflow.com/a/8964779/1135883
# For large files with regular records this is a much faster and memory
# efficient solution
def iter_loadtxt(filename, delimiter=' ', skiprows=0, dtype=float):
def iter_func():
with open(filename, 'r') as infile:
for _ in range(skiprows):
next(infile)
@seumasmorrison
seumasmorrison / grid_bathy_xyz.py
Created January 18, 2013 11:53
Takes a numpy array of bathymetry points from a grid ( exportable from a raster in ArcGIS using ArcPy ) and writes a tab separated xyz file, for import into gis or in this case it was written for importing bathymetry as scatter data into DHI MIKE21. Requires a hdr file for width, height, origin and cell size information.
# Initial data obtained from hdr file produced with flt file when running
# Raster to Float in ArcToolBox and numpy file ( npy ) from RasterToNumpyArray
# in ArcPy
# Numpy uses top left as origin and geopgraphic coordinate systems treat the
# origin as lower left, you can either flip the incoming bathymetry maxtrix
# vertically or create a descending range of y_coordinates
import numpy as np
bathy = np.load('D:\\numpy_bathy.npy')
@seumasmorrison
seumasmorrison / plot_3d_surface_xyz.py
Last active December 14, 2015 07:08
Example code and input data for Stack Overflow question
# -*- coding: utf-8 -*-
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.mlab import griddata
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import matplotlib.pyplot as plt
import numpy as np
import sys
def xyz_ret(file):
@seumasmorrison
seumasmorrison / ipython_history_query.py
Last active December 15, 2015 15:49
Example of how to query IPython history, searching using a known string used in a session and then using the session_id from the first query to get the whole of that session.
#Example of searching for a string in IPython sqlite history
from IPython.core.history import HistoryAccessor
history_accessor = HistoryAccessor()
sqlite_cursor = history_accessor.search('*threshold*')
results = sqlite_cursor.fetchall()
for result in results:
print result
#From the first number in the printed results you can see the relevant session
#number, and then below retrieve the whole session using get_range
session_results = history_accessor.get_range(1321)