Skip to content

Instantly share code, notes, and snippets.

View aniquetahir's full-sized avatar
💤
( 👁 〰️ 👁 )

Anique aniquetahir

💤
( 👁 〰️ 👁 )
View GitHub Profile
set nocompatible " required
filetype off " required
" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
" alternatively, pass a path where Vundle should install plugins
"call vundle#begin('~/some/path/here')
@aniquetahir
aniquetahir / fairness_simple.ipynb
Created August 23, 2022 03:57
Simple MLP Fairness
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@aniquetahir
aniquetahir / jax_mlp.py
Created August 21, 2022 12:42
Simple Boilerplate for a Jax MLP with Dropout
NUM_LAYERS = 3
HIDDEN_DIM = 128
DROPOUT = 0.25
NUM_EPOCHS = 1000
key = jax.random.PRNGKey(6)
def loss_fn(params, key, x, y, is_training=True):
logits = mlp_apply(params, key, x, is_training)
loss = optax.softmax_cross_entropy_with_integer_labels(logits, y)
! 7/15/2020 https://colab.research.google.com
colab.research.google.com##.parameter-hints-widget.visible > style:style(right: 20px; min-width: 200px; left: unset !important;)
! 8/1/2020 https://www.reddit.com
www.reddit.com###tooltip708392
www.reddit.com##.email-verification-tooltip
! google sneaky tracking
google.com##+js(remove-attr.js, onmousedown, .g a[href])
function upvoteAll(){
let all_elems = document.querySelectorAll('a.vote-up');
let unvoted = [];
all_elems.forEach(e=>unvoted.push(e));
unvoted = unvoted.filter(e=>e.className.indexOf('upvoted')==-1);
if(unvoted.length>0){
unvoted[0].click();
}
}
setInterval(upvoteAll, 1000);
cd ~/
# Set up project
mkdir projects
cd projects
git clone https://github.com/aniquetahir/yellowtaxi.git
git clone https://github.com/aniquetahir/datapolygamyutils.git
export POLYUTILS=$(pwd)/datapolygamyutils
sudo apt install maven axel pypy
cd yellowtaxi
#!/anaconda/bin/python
import sys
def GetIntersection(source_data, check_data):
num_intersection = 0
for row in check_data:
if row in source_data:
num_intersection+=1
return num_intersection
/*---------------------------- Start an example Spatial Join Query using Cartesian Product algorithm ----------------------------*/
/* Please see http://www.public.asu.edu/~jiayu2/geospark/javadoc/latest/ for function documentation */
val objectRDD = new PointRDD(sc, "/home/SparkUser/Downloads/GeoSpark/src/test/resources/arealm.csv", 0, FileDataSplitter.CSV, false, 10); /* The O means spatial attribute starts at Column 0 and the 10 means 10 RDD partitions */
val rectangleRDD = new RectangleRDD(sc, "/home/SparkUser/Downloads/GeoSpark/src/test/resources/zcta510.csv", 0, FileDataSplitter.CSV, false); /* The O means spatial attribute starts at Column 0. You might need to "collect" all rectangles into a list and do the Carteian Product join. */
val resultSize = JoinQuery.SpatialJoinQuery(objectRDD, rectangleRDD, true).count();
/*---------------------------- End an example Spatial Join Query using Cartesian Product algorithm ----------------------------*/
CSE 512 Naive Spatial Join
1.Create a PointRDD objectRDD;
2.Create a RectangleRDD queryWindowRDD;
3.Collect rectangles from queryWindowRDD to one Java List L;
4. For each rectangle R in L
do RangeQuery.SpatialRangeQuery(objectRDD, queryEnvelope, 0, true);
End;
5.Collect all results; //"Collect" is a standard function under SparkContext.
6.Parallelize the results to generate a RDD in this format: JavaPairRDD<Envelope, HashSet<Point>>.;//"Parallelize" is a standard function under SparkContext.
7.Return the result RDD;
import sys
AUTHORS_FILE_NAME = ""
AFFILIATIONS_FILE_NAME = ""
AUTHOR_AFFILIATION_FILE_NAME = ""
authors_file = open(AUTHORS_FILE_NAME)
affiliations_file = open(AFFILIATIONS_FILE_NAME)
author_affiliation_file = open(AUTHOR_AFFILIATION_FILE_NAME)