Skip to content

Instantly share code, notes, and snippets.

View saj1th's full-sized avatar

Sajith Appukuttan saj1th

39 followers · 73 following
View GitHub Profile
@saj1th
saj1th / icepick-v1-manifest-from_avro.json
Last active April 6, 2021 00:55
{
"status": 1,
"snapshot_id": {
"long": 5289619966059993000
},
"data_file": {
"file_path": "s3://db-sa-datasets/ice-warehouse/data/icepick/data/00000-0-e1d6df32-ebbd-46af-b484-a0ba73eedccc-00001.parquet",
"file_format": "PARQUET",
"partition": {},
"record_count": 1,
@saj1th
saj1th / icepick-v1-manifest-list-from-avro.json
Last active April 6, 2021 00:56
{
"manifest_path": "s3://db-sa-datasets/ice-warehouse/data/icepick/metadata/8bbc2cc6-d5a6-4bec-9660-064ff387af88-m0.avro",
"manifest_length": 5654,
"partition_spec_id": 0,
"added_snapshot_id": {
"long": 5289619966059993000
},
"added_data_files_count": {
"int": 3
},
@saj1th
saj1th / icepick-v1.json
Last active April 6, 2021 00:45
{
"format-version": 1,
"table-uuid": "7b487059-ed1a-4462-bee8-fc2defee430d",
"location": "s3://db-sa-datasets/ice-warehouse/data/icepick",
"last-updated-ms": 1617666462797,
"last-column-id": 2,
"schema": {
"type": "struct",
"fields": [
{
@saj1th
saj1th / icepick-v0.json
Created April 6, 2021 00:32
{
"format-version": 1,
"table-uuid": "7b487059-ed1a-4462-bee8-fc2defee430d",
"location": "s3://db-sa-datasets/ice-warehouse/data/icepick",
"last-updated-ms": 1617666074935,
"last-column-id": 2,
"schema": {
"type": "struct",
"fields": [
{
@saj1th
saj1th / computeFileStats.scala
Last active April 6, 2021 00:24
def byteFormat(fileSize: Long): String = {
if (fileSize <= 0) return "0 B"
val units: Array[String] = Array("B", "kB", "MB", "GB", "TB", "PB", "EB")
val grp: Int = (Math.log10(fileSize) / Math.log10(1024)).toInt
f"${fileSize / Math.pow(1024, grp)}%3.3f ${units(grp)}"
}
def computeFileStats(path: String): (String, String) = {
val formatter = java.text.NumberFormat.getIntegerInstance
var (bytes, count) = (0L, 0L)
@saj1th
saj1th / z2ipynb.py
Last active February 20, 2019 20:37
Script to convert zeppelin notebooks to ipynb
# Based on https://github.com/rdblue/jupyter-zeppelin
import os, sys
import re
import csv
import json
import html
import nbformat
import codecs
from io import StringIO
@saj1th
saj1th / fun-with-functions.md
Created May 31, 2017 09:42 — forked from jdegoes/fun-with-functions.md
Fun with Functions! - Exercises Only

These exercises were part of a LambdaConf Outpost meetup. You may find the instructions here.

  1. Develop a model for boolean values (Bool), which may be either true or false.

    Think: Do you need an if-then-else construct? Why or why not?

    Bonus: Develop a model for eithers (Either), whih can be one thing ("left") or another ("right"), and model a boolean as a partitioning of a set into two disjoint sets.

  2. Develop a model for optional values (Maybe / Option), which are containers that are either empty ("nothing" / "none") or hold a single value ("just" / "some").

@saj1th
saj1th / Advanced-FP-with-Scala.md
Created May 31, 2017 09:40 — forked from jdegoes/Applied-FP-with-Scala.md

Advanced Functional Programming with Scala - Notes

Copyright © 2017 Fantasyland Institute of Learning. All rights reserved.

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x
@saj1th
saj1th / SlowBackend.go
Last active April 24, 2017 12:05
func (e Events) Track(w http.ResponseWriter, r *http.Request) {
start := time.Now()
var nuid, event string
// reads the cookie value from request - (function from go standard lib - no backend op here)
nuidCookie, err := r.Cookie(CookieName)
if err != nil {
//setting var
nuid = uuid.New()
} else {
@saj1th
saj1th / README.md
Created November 7, 2016 17:18 — forked from leonardofed/README.md
A curated list of AWS resources to prepare for the AWS Certifications


A curated list of AWS resources to prepare for the AWS Certifications

A curated list of awesome AWS resources you need to prepare for the all 5 AWS Certifications. This gist will include: open source repos, blogs & blogposts, ebooks, PDF, whitepapers, video courses, free lecture, slides, sample test and many other resources.

Index: