bw4sz

import geopandas as gp
import rasterio
import os

def shapefile_to_annotations(shapefile, rgb, savedir="."):
    """
    Convert a shapefile of annotations into annotations csv file for DeepForest training and evaluation
    Args:
        shapefile: Path to a shapefile on disk. If a label column is present, it will be used, else all labels are assumed to be "Tree"
        rgb: Path to the RGB image on disk

bw4sz

import threading

from keras.applications.inception_v3 import InceptionV3
from keras.optimizers import Adam
from keras.utils.io_utils import HDF5Matrix


class threadsafe_iter:
    """Takes an iterator/generator and makes it thread-safe by
    serializing call to the `next` method of given iterator/generator.

bw4sz

import numpy as np
import rasterio as rio
import dask
import dask.array as da

class RioDataset():
    def __init__(self, filepath, band=1):
        self.band = band
        self.dataset = rio.open(filepath)
        self.dtype = self.dataset.dtypes[band-1]

bw4sz

Aim

We often have very large datasets that need to be handled using our high performance computing system. We also want the comfort of working on our laptops, and not being tied to the command line. The goal of this wiki page is to outline how to use dask and juypterlab notebooks to manipulate spatial data on hipergator. This is an evolving document that will be updated as we find best practices.

Initial Setup (only need to do once)

• Log on to hipergator.
• Create a conda virtual environment. We want some control over our own python environment, and not be so dependent on the python modules provided by admin. Start in our home directory

bw4sz

x<-runif(1e4,0,24)
alpha=0.5
beta=0.5
beta2=0
k=0
timef<-function(x){
    y<-alpha + beta* cos((2*pi*x)/(24+k)) + beta2 * sin((2*pi*x)/24+k)^2
    return(data.frame(x,y))
}
y<-timef(x)

bw4sz

> m2$parameters.to.save
 [1] "detect"      "alpha"       "beta1"       "beta2"       "intercept"   "sigma_alpha"
 [7] "sigma_beta1" "sigma_beta2" "gamma1"      "gamma2"      "fit"         "fitnew"     
[13] "ynew"       
> newm<-update(m2,n.iter=runs,n.burnin=runs*.95,n.thin=5,parameters.to.save="dprior")
  |**************************************************| 100%
  |**************************************************| 100%
> newm$parameters.to.save
 [1] "detect"      "alpha"       "beta1"       "beta2"       "intercept"   "sigma_alpha"
 [7] "sigma_beta1" "sigma_beta2" "gamma1"      "gamma2"      "fit"         "fitnew"