Skip to content

Instantly share code, notes, and snippets.

View nelsonsequiera's full-sized avatar
🤷‍♂️
Experiment - Fail - Learn - Repeat

Nelson Sequiera nelsonsequiera

🤷‍♂️
Experiment - Fail - Learn - Repeat
2 followers · 4 following
View GitHub Profile
@nelsonsequiera
nelsonsequiera / 00_Redshift best distribution key test.txt
Last active December 31, 2020 06:04
Redshift best distribution key test
* get current sort and dist keys
* Materialize a single column to check distribution
* Identify the table OID
* get skewed data stats
@nelsonsequiera
nelsonsequiera / _0 redshift column encoding test.md
Created December 22, 2020 17:21
redshift testing column encoding selecting best column encoding
  • get the column count
  • create a new table with encodings you want to test
  • insert the same column to all the column in new table
  • Query the STV_BLOCKLIST system table
  • encoding with lowest storage block count is the best encoding.
@nelsonsequiera
nelsonsequiera / distribution style.md
Last active December 23, 2020 06:12
redshift best suggested recommended column encoding by data types and by encoding types. column encodings: raw AZ64 Byte-dictionary Delta LZO Mostly Runlength Text255 ZSTD Data types:BOOLEAN DOUBLE SMALLINT INTEGER BIGINT DECIMAL REAL DOUBLE CHAR VARCHAR DATE TIMESTAMP TIMESTAMPTZ TIME
style use case
key same value same slice, for joins
all dimension small table for frequent joins, data is copied to all nodes first slice
even unknown cases
  • Set DISTKEY to the column most used in a JOIN
  • Set SORTKEY to the column(s) most used in a WHERE
@nelsonsequiera
nelsonsequiera / Redshift usefull queries.txt
Last active December 31, 2020 06:04
Redshift usefull Queries
Redshift usefull queries
@nelsonsequiera
nelsonsequiera / insta_followers_data.py
Created October 30, 2020 18:25
import requests
import json
import time
import csv
import pandas as pd
# step 1: get all followers html content manually from insta page. in one line keep only one username (without quotes and comma.)
def get_usernames():
@nelsonsequiera
nelsonsequiera / Analysis Paralysis
Last active July 8, 2019 08:46
A pottery teacher split her class into two halves.
To the first half she said, “You will spend the semester studying pottery, planning, designing, and
creating your perfect pot. At the end
of the semester, there will be a competition to see whose pot is the best”.
To the other half she said, “You will spend your semester making lots of pots. Your grade will be
based on the number of completed pots you
finish. At the end of the semester, you’ll also have the opportunity to enter your best pot into a
competition.”
@nelsonsequiera
nelsonsequiera / Data Engineering Checklist.md
Last active October 26, 2023 16:07
@nelsonsequiera
nelsonsequiera / README.md
Last active July 6, 2019 13:57

sudo bash ./entermedia-docker.sh alchemy-dam 100

@nelsonsequiera
nelsonsequiera / object_leveller.js
Created May 14, 2019 07:05
js object leveller
module.exports.objectLeveller = (obj, outputObj, outputPath) => {
var outputAccObj = outputObj || {};
var outputAccPath = outputPath || [];
return Object.getOwnPropertyNames(obj).reduce(function (outputAccObj, key) {
outputAccPath.push(key);
if (typeof obj[key] === 'object' && !!obj[key]) {
objectLeveller(obj[key], outputAccObj, outputAccPath);
} else {
outputAccObj[outputAccPath.join('_')] = obj[key];
@nelsonsequiera
nelsonsequiera / AWS Glue Dev Endpoint.md
Last active May 2, 2019 14:38