Skip to content

Instantly share code, notes, and snippets.

View jasonge27's full-sized avatar

Jason Ge jasonge27

79 followers · 30 following
View GitHub Profile
@jasonge27
jasonge27 / conda_conflict_mf_keras
Created January 28, 2021 22:50
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package python conflicts for:
keras -> h5py -> python[version='3.7.*|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|3.8.*']
pyarrow[version='>=0.17.1'] -> numpy[version='>=1.16,<2.0a0'] -> python[version='2.7.*|3.5.*|3.6.*|3.7.*|>=2.7,<2.8.0a0|3.9.*|3.8.*|>=3.5,<3.6.0a0|3.4.*']
pyarrow[version='>=0.17.1'] -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']
keras -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|3.4.*']
python==3.7
@jasonge27
jasonge27 / metaflow-cfn-template-economy.yml
Created July 16, 2020 22:16
A minimal & economy AWS setup for metaflow (spot instance for batch, no sagemaker, no multi-region RDS)
AWSTemplateFormatVersion: '2010-09-09'
Description: Stack for complete deployment of Metaflow
Parameters:
VPCCidr:
Type: String
Default: 10.20.0.0/16
Description: 'CIDR for the Metaflow VPC'
Subnet1Cidr:
Type: String
@jasonge27
jasonge27 / helloaws_conda.py
Created March 25, 2020 07:07
from metaflow import FlowSpec, step, batch, retry, conda
class HelloAWSFlow(FlowSpec):
"""
A flow where Metaflow prints 'Metaflow says Hi from AWS!'
Run this flow to validate your AWS configuration.
"""
@jasonge27
jasonge27 / snark-hyper-example.yaml
Created September 28, 2018 23:41
version: 1
experiments:
lr_search:
image: tensorflow/tensorflow:1.11.0
parameters:
lr: '0.2-0.5'
sampling: random # | grid | bayesian
command:
python train.py --lr {{lr}}
@jasonge27
jasonge27 / Snark AI Usage
Created June 26, 2018 06:42
# Basic Usage
Step 1. Install snark through pip3
```
$ pip3 install snark --user
```
Step 2. Go to lab.snark.ai to sign up. Sign in through the CLI
```
$ snark login
```
@jasonge27
jasonge27 / Princeton.md
Last active February 17, 2016 03:10 — forked from smsharma/Princeton.md
Running stuff on the Princeton clusters.

##Feynman: feynman.princeton.edu

Feynman-hepheno has somewhere between 12 and 16 nodes (unclear).

  • Three areas -- home (~), group (/group/hepheno/) and group storage (/mnt/hepheno/)
  • Best to work from your user directory in /group/hepheno since ~ has very little storage space
  • The contents of /group/hepheno/group-setup.sh are automatically sourced on cd into the group directory /group/hepheno/:
    • Activates a python virtual environment (ML: chmod others into the venv so they can pip install etc?)
    • Initiates MPI and sets a few environment variables