farrajota

Missing values in data

Types of missing values

Missing completely at random (MCAR)

MCAR exists when missing values are randomly distributed across all observations. Missingness in given variable does not depend on any other variable, whether observed or unobserved. MCAR can be confirmed by dividing respondents into those with and without missing data, then using t-tests of mean differences on income, age, gender, and other key variables to establish that the two groups do not

farrajota

When imputation should not be used

• If data are MCAR, imputation may not be not needed.
• If missingness is due to unmeasured variables related to the dependent variable, data are MNAR and should not be imputed.
• Imputation assumes data are MAR and should not be used with sparse data. Sparse data occur when missingness is non-random, such as a shopping cart survey of items purchased (coded 1) or not purchased (coded 0), because the null response (0) is non-random, due to unmeasured factors possibly not even known to the shopper.
• Imputation should not be used to impute all the data for a subject
• Imputation should not be used for a missing value for a given observation if that observation is also missing values on predictively critical variables in the imputation model. While this is difficult to check for each value to be imputed, a table of missing value patterns will show how many cases missing on a given variable also have missing values on other variables. In some cases this may lead a researcher

farrajota

function mpii_convert_json(  )
% convert mpii annotations .mat file to .json


%% load annotation file
fprintf('Load annotations... ')
data = load('/media/HDD2/Datasets/Human_Pose/mpii/mpii_human_pose_v1_u12_2/mpii_human_pose_v1_u12_1.mat');
fprintf('Done.\n')
%% open file
fprintf('Open file mpii_human_pose_annotations.json\n')

farrajota

data science checklist

---

Step 1

---

Data loading# data science checklist

---

Step 1

farrajota

-- Available torch7 pre-trained models for download 

Alexnet (trained by me) 
cudnn: https://ln.sync.com/dl/3d9c28e80#cvjp4mjs-mc2y8jrt-9wq5yebx-yrdchcei
mean = {0.48037518790839, 0.45039056120456, 0.39922636057037}
std =  {0.27660147027775, 0.26883440068399, 0.28014687231841}
img size: 3x224x224

overfeat: https://github.com/jhjin/overfeat-torch
mean = {118.380948, 118.380948, 118.380948}

farrajota

Here I've put the most important data structures I could find with the most efficient implementations in Python I am aware of. Although this is a continuous process and many structures do not have any implementation reference available yet, if you happen to know a more efficient algorithm to implement one or more of this structures in Python (or any language) feel free to ping me :).

Basic Data Structures

Data Structure	Python
Arrays	numpy Tensorflow PyTorch

farrajota

"""
Using dask's multithreaded scheduler to speedup download of multiple files from
an s3 bucket
"""

import os
from functools import partial

import botocore
import boto3

farrajota

I've been working with Apache Kafka for over 7 years. I inevitably find myself doing the same set of activities while I'm developing or working with someone else's system. Here's a set of Kafka productivity hacks for doing a few things way faster than you're probably doing them now. 🔥

• Show me all my Kafka topics and their partitions, replicas, and consumers
• Show me the contents of a topic
• Create a Kafka topic
• Produce messages to a Kafka topic
• Validate the schema of messages before producing to a topic
• Do all of this at a distance

Get the tools

farrajota

Missing data

A Simple Example of a Missing Data Analysis Understanding the Reasons Leading to Missing Data Ignorable Missing Data Other Types of Missing Data Processes Examining the Patterns of Missing Data Diagnosing the Randomness of the Missing Data Process

Rules of Thumb

farrajota

model.modules[1].parameters = function() return nil end -- freezes the layer when using optim 
model.modules[1].accGradParameters = function() end -- overwrite this to reduce computations