kristoff-it

Sample and Group Privates

This document aims to explain my solution to the problem of precomputing/indexing private characteristics of a sequence. The example proposed will be a slightly simplified version of VCF cross-filtering to find private mutations, but the idea is more general and I will briefly explore some

kristoff-it

{
 "metadata": {
  "name": "Introduzione alle reti neurali artificiali"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {

kristoff-it

Keybase proof

I hereby claim:

• I am kristoff-it on github.
• I am kristoff (https://keybase.io/kristoff) on keybase.
• I have a public key ASDGE4EbKiDm97tzWMZp7ki56MuKQQj-SQXLSlH87OYPmAo

To claim this, I am signing this object:

kristoff-it

export fn HelloWorld_Command(ctx: ?*redis.RedisModuleCtx, argv: ?[*]?*redis.RedisModuleString, argc: c_int) c_int {
    _ = redis.RedisModule_ReplyWithSimpleString.?(ctx, c"Hello World!");
    return redis.REDISMODULE_OK;
}

kristoff-it

#include "redismodule.h"

int HelloWorld_Command(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    RedisModule_ReplyWithSimpleString(ctx, "Hello World!");
    return REDISMODULE_OK;
}

int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    if (RedisModule_Init(ctx,"testmodule",1,REDISMODULE_APIVER_1)
        == REDISMODULE_ERR){

kristoff-it

>>> full_df = spark.read.csv("pantheon.tsv", sep="\t", quote="", header=True, inferSchema=True)
>>> full_df.dtypes
[('en_curid', 'int'), ('name', 'string'), ('numlangs', 'int'), ('birthcity', 'string'), ('birthstate', 'string'), ('countryName', 'string'), ('countryCode', 'string'), ('countryCode3', 'string'), ('LAT', 'double'), ('LON', 'double'), ('continentName', 'string'), ('birthyear', 'string'), ('gender', 'string'), ('occupation', 'string'), ('industry', 'string'), ('domain', 'string'), ('TotalPageViews', 'int'), ('L_star', 'double'), ('StdDevPageViews', 'double'), ('PageViewsEnglish', 'int'), ('PageViewsNonEnglish', 'int'), ('AverageViews', 'double'), ('HPI', 'double')]

kristoff-it

>>> data = full_df.select("en_curid", "countryCode", "occupation")
>>> data.show(2)
+--------+-----------+-----------+
|en_curid|countryCode| occupation|
+--------+-----------+-----------+
|     307|         US| POLITICIAN|
|     308|         GR|PHILOSOPHER|
+--------+-----------+-----------+
only showing top 2 rows

kristoff-it

>>> data.write.format("org.apache.spark.sql.redis").option("table", "people").option("key.column", "en_curid").save()

kristoff-it

> HGETALL people:2113653
1) "countryCode"
2) "DE"
3) "occupation"
4) "SOCCER PLAYER"

kristoff-it

>>> df = spark.read.format("org.apache.spark.sql.redis").option("table", "people").option("key.column", "en_curid").load()
>>> df.show(2)
+--------+-----------+----------+
|en_curid|countryCode|occupation|
+--------+-----------+----------+
|  915950|         ZW|   SWIMMER|
|  726159|         UY|POLITICIAN|
+--------+-----------+----------+
only showing top 2 rows