Yordan Georgiev YordanGeorgiev
YordanGeorgiev
/ how-to-create-a-symlink-on-linux
Created
February 17, 2018 09:48
[create a symlink] how-to create a symlink on linux #linux #bash
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # START === create symlink | |
| export link_path=/opt/ysg | |
| export target_path=/vagrant/opt/ysg | |
| mkdir -p `dirname $link_path` | |
| unlink $link_path | |
| ln -s "$target_path" "$link_path" | |
| ls -la $link_path; | |
| # STOP === create symlink |
YordanGeorgiev
/ pkk-auth.sh
Created
February 17, 2018 09:52
public private key auth for ssh #linux #ssh #pkk
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # START === how-to implement public private key ( pkk ) authentication | |
| # create pub priv keys on server | |
| # START copy | |
| ssh-keygen -t rsa | |
| # Hit enter twice | |
| # copy the rsa pub key to the ssh server | |
| scp ~/.ssh/id_rsa.pub $ssh_user@$ssh_server:/home/$ssh_user/ | |
| # STOP copy | |
| # now go on the server | |
| ssh $ssh_user@$ssh_server |
YordanGeorgiev
/ sbt-clean-up.sh
Last active
April 11, 2019 08:13
[sbt-clean-update-test] how-to clean a scala project, update and test it without idea #sbt #scala #test #idea
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #file: src/bash/sbt-clean-update-test.sh | |
| #aim: compile and test scala in the shell without idea ... | |
| alias find=find # probably obsolete in your env | |
| find ~/.sbt ~/.ivy2 -name '*.lock' -print -delete | |
| alias find="find -L" | |
| rm -fvr ~/.sbt/0.13/plugins/target | |
| rm -fvr ~/.sbt/0.13/plugins/project/target | |
| rm -fvr ~/.sbt/1.0/plugins/target |
YordanGeorgiev
/ scala-foreach-loop.scala
Created
February 17, 2018 10:06
[foreach loop with var] how-to assign a var into scala foreach loop #scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // how-to assign var in foreach loop | |
| val objFileHandler = new FileHandler () | |
| objFileHandler.getFileTree( new File ( dataCsvDir ) ) | |
| .filter(_.getName.endsWith(".csv")) | |
| .foreach{ | |
| x => { | |
| var f = x; | |
| println ( f.toString() ) | |
| /* some operation */ | |
| } |
YordanGeorgiev
/ scala-spark-dataframe-fold-left.scala #scala #spark #fold-left
Created
February 17, 2018 10:08
[fold left usage in scala spark] how-to use fold left in scala on a dataframe obj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // START foldLeft usage | |
| val outDf: DataFrame = lstColumnsToIterate | |
| .foldLeft(inDf)((tmpDf, iterableColToAdd) => { | |
| tmpDf.withColumn(iterableColToAdd,expr(funcToApply).as(iterableColToAdd)) | |
| }) | |
| .groupBy(lstGroupByCols.distinct.head, lstGroupByCols.distinct.tail: _*) | |
| .agg(lstAggregationCols.distinct.head, lstAggregationCols.distinct.tail: _*) | |
| // STOP foldLeft usage |
YordanGeorgiev
/ spark-dataframe-fullouter-join-on-nullable-columns.scala
Last active
February 17, 2018 10:42
[full outer join on nullable columns for spark dataframe] how-to apply a full outer join on a spark dataframe #scala #spark #dataframe #joins
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val lstKeyCols = List("col1" , "col2" , "col3" ) | |
| dfLeft | |
| .join( | |
| dfRight, | |
| dfLeft("col1") <=> dfRight("col1_") | |
| && dfLeft("col2") <=> dfRight("col2_") | |
| && dfLeft("col3") <=> dfRight("col3_"), | |
| "fullouter" | |
| ) | |
| .drop(lstKeyCols.map(_ + "_"): _*) |
YordanGeorgiev
/ iterate-over-rdd-rows.scala
Last active
November 30, 2018 10:18
[iterate over rdd rows] how-to iterate over RDD rows and get DataFrame in scala spark #scala #spark
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // note if you could implement withColumn + udf it has been usually over 10x faster ... | |
| val rddRows: RDD[Row] = | |
| inDf.rdd.map(row => { | |
| val lstRow = row.toSeq.toList | |
| var lstRowNew = lstRow | |
| // do stuff on the new lstRow here | |
| Row.fromSeq(lstRowNew) |
YordanGeorgiev
/ create-dataframe-with-schema
Last active
February 17, 2018 10:29
[create dataframe with schema] how-to create a dataframe obj with schema in scala spark #scala #spark #dataframe
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val spark = SparkSession.builder().getOrCreate() | |
| import spark.implicits._ | |
| val df = spark | |
| .createDataFrame( | |
| spark.sparkContext.parallelize( | |
| Seq( | |
| Row( | |
| Map(("key1","val1") -> 1) |
YordanGeorgiev
/ scala-singleton.scala
Created
February 17, 2018 10:29
[object singleton] how-to create object singleton in scala #scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| object SingleTon { | |
| def apply(): SingleTon = { | |
| new SingleTon() | |
| } | |
| } | |
| class SingleTon { |
YordanGeorgiev
/ scala-spark-dataframe-pipeline.scala
Created
February 17, 2018 10:40
[dataframe pipeline for spark] how-to build a dataframe processing pipeline in scala spark #scala #spark #dataframe #control-flow
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| private def runPipeLine(cnf: Configuration): DataFrame = { | |
| val dfOut: DataFrame = | |
| new Phase1(cnf).process() | |
| .transform(new Phase2(cnf).process) | |
| return dfOut | |
| } | |
| class Phase1 extends DataFrameStage { | |
OlderNewer