yashk

package underscore
import org.apache.hadoop.fs.{Path, PathFilter}
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat

import org.apache.spark.sql.SparkSession


 class TmpFileFilter  extends PathFilter {
  override def accept(path: Path): Boolean = {
    println(path)

yashk

hdfs dfs -du "$1" | awk '{print $1,$2,$3}' | sort -nr | xargs -n3 sh -c 'printf "%s  %s  %s\n" $(numfmt --to=iec $0) $(numfmt --to=iec $1) $2'

yashk

✔️Tech-Powered Sales by Justin Michael
✔️The Incredible History of India's Geography by Sanjiv Sanyal
✔️The Art of Saying No by Damon Zahariades
✔️No Limits by Mukesh B.
✔️Measure What Matters by John Doerr
✔️How I Almost Blew It by Sidharth Rao
✔️Meetings That Get Results by Brian Tracey
✔️Linked Inbound by Sam Rathling
✔️The Million Pound Linkedin Message by Daniel Disney
✔️The Psychology of Money by Morgan Housel

yashk

import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class TestCsv {
    private static final String CSV_FILE_NAME = "hive_out.csv";

yashk

• http://www.mahatmagandhischoolpune.org/donate
• https://milaap.org/fundraisers/support-mgschool

yashk

~/work/st/notes/gist.txt

yashk

Primary editor(s) :

• Intellij Idea ,
• VS Code
• VIM

Command line :

• Iterm2 with solazied dark theme
• Shell - fish (give it a try it is better than zsh or bash)
• command prompt starship
• autojump (no need for cd after it learns your dirs)

yashk

https://forums.databricks.com/questions/277/how-do-i-avoid-the-no-space-left-on-device-error.html

yashk

;;
;; NS CHEATSHEET
;;
;; * :require makes functions available with a namespace prefix
;;  and optionally can refer functions to the current ns.
;;
;; * :import refers Java classes to the current namespace.
;;
;; * :refer-clojure affects availability of built-in (clojure.core)
;;   functions.

yashk

1. Auto Scaling Systems With Elastic Spark Streaming: Spark Summit East talk by PhuDuc Nguyen

• https://www.youtube.com/watch?v=N10k7tRsTUA

2. Tool for launching spark cluster in aws

• https://github.com/nchammas/flintrock

3. Spark dynamic allocation

• https://www.slideshare.net/databricks/dynamic-allocation-in-spark
• https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-dynamic-allocation.html