Skip to content

Instantly share code, notes, and snippets.

View saucam's full-sized avatar

Yash Datta saucam

  • Columbia University
  • Singapore
View GitHub Profile
@saucam
saucam / README.md
Created January 16, 2022 06:49 — forked from maelvls/README.md
Ubuntu, libsecret, git-credential-helper

Dealing with secrets

GNOME comes with libsecret. You can use libsecret to store your git credentials:

sudo apt install libsecret-1-0 libsecret-1-dev libglib2.0-dev
sudo make --directory=/usr/share/doc/git/contrib/credential/libsecret
git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret
@saucam
saucam / GSoC_2020_report.md
Last active August 28, 2020 12:23
Google Summer of Code 2020: Project Report

Manipulation of massive astronomical data using graphs

Background

The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), will generate data of the order of TBs and will send out a stream of around 10 million alerts per night. LSST has committed to making a part of the useful data it collects public in the form of alerts generated in real-time. AstroLab Software has developed Fink, an Apache Spark based broker infrastructure, which is able to analyze large streams of alerts data from the telescopes like LSST, and then redistribute it to subscribers, enabling a wide range of applications and services to consume this data. This processed data needs to be stored for visualizing and post-processing. The efficient manipulation and visualization of patterns in this extremely large dataset is a real challenge for which we created grafink as

Applied Functional Programming with Scala - Notes

Copyright © 2016-2018 Fantasyland Institute of Learning. All rights reserved.

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x
@saucam
saucam / fpmax.scala
Created August 4, 2018 14:08 — forked from jdegoes/fpmax.scala
FP to the Max — Code Examples
package fpmax
import scala.util.Try
import scala.io.StdIn.readLine
object App0 {
def main: Unit = {
println("What is your name?")
val name = readLine()

Serialization of Scala closures that contain local defs

Several Apache Spark APIs rely on the ability to serialize Scala closures. Closures may reference non-Serializable objects, preventing them from being serialized. In some cases (SI-1419 and others), however, these references are unnecessary and can be nulled out, allowing otherwise-unserializable closures to be serialized (in Spark, this nulling is performed by the ClosureCleaner).

Scala 2.12's use of Java 8 lambdas for implementing closures appears to have broken our ability to serialize closures which contain local defs. If we cannot resolve this problem, Spark will be unable to support Scala 2.12 and will be stuck on 2.10 and 2.11 forever.

As an example which illustrates this problem, the following closure has a nested localDef and is defined inside of a non-serializable class:

``