This quick guide for getting a Jupyter Notebook up and running on Bridges, a supercomputer managed by the Pittsburgh Supercomputing Center. Bridges is a new machine designed to accommodate non-traditional uses of High Performance Computing (HPC) resources like data science and digital humanities. Bridges is available through XSEDE, which is the system that manages access to multiple supercomputing resources. Through XSEDE, Bridges is available researchers or educators at US academic or non-profit research institutions (see the XSEDE eligibility policies) Allocations are free, but there is a somewhat difficult to understand application process filled with jargon and acronyms that take time to understand. See the XSEDE getting started guide for more information about getting acc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
render_with_jobs <- function(){ | |
rstudioapi::verifyAvailable() | |
jobs_file <- tempfile(tmpdir = "/tmp", fileext = ".R") | |
rmd_to_render <- rstudioapi::selectFile(caption = "Choose an Rmd file...", | |
filter = "Rmd files (*.Rmd)") | |
if (is.null(rmd_to_render)){ | |
stop("You must choose an Rmd file to proceed!") | |
} | |
cat(paste0('rmarkdown::render("', rmd_to_render, '")'), file = jobs_file) | |
rstudioapi::jobRunScript(path = jobs_file, |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from sklearn.metrics import roc_auc_score | |
from math import sqrt | |
def roc_auc_ci(y_true, y_score, positive=1): | |
AUC = roc_auc_score(y_true, y_score) | |
N1 = sum(y_true == positive) | |
N2 = sum(y_true != positive) | |
Q1 = AUC / (2 - AUC) | |
Q2 = 2*AUC**2 / (1 + AUC) | |
SE_AUC = sqrt((AUC*(1 - AUC) + (N1 - 1)*(Q1 - AUC**2) + (N2 - 1)*(Q2 - AUC**2)) / (N1*N2)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# on local | |
pempath="$1" | |
ec2target="$2" | |
ssh -T -i "${pempath}" -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ubuntu@${ec2target} << 'EOBLOCK' | |
# on remote machine | |
sudo apt-get -y update | |
sudo apt-get -y upgrade |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### ------- Load Packages ---------- ### | |
library("purrr") | |
library("future") | |
library("dplyr") | |
library("randomForest") | |
library("rsample") | |
library("ggplot2") | |
library("viridis") | |
### ------- Helper Functions for map() ---------- ### | |
# breaks CV splits into train (analysis) and test (assessmnet) sets |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Set API Key | |
Sys.setenv(SIGOPT_API_TOKEN="HERE") | |
# Start a local H2O cluster for training models | |
library(h2o) | |
h2o.init(nthreads = -1) | |
# Load a dataset | |
data(iris) | |
y <- "Species" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#plotly box or lasso select linked to | |
# DT data table | |
# using Wage data | |
# the out group: is sex:Male, region:Middle Atlantic + | |
library(ggplot2) | |
library(plotly) | |
library(dplyr) | |
library(ISLR) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
library(foreach) | |
library(iterators) | |
library(doParallel) | |
library(tcltk) | |
# Choose number of iterations | |
n <- 1000 | |
cl <- makeCluster(8) |
NewerOlder