umbertogriffo

import org.apache.commons.lang.SystemUtils
import org.apache.spark.mllib.random.RandomRDDs._
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}

import scala.math.sqrt

/**
  * Created by Umberto on 08/02/2017.
  */

umbertogriffo

import java.util.*;
import java.util.Map.Entry;
import java.util.stream.Collectors;

/**
 * Created by Umberto on 16/05/2017.
 */

public class HashMapUtils {

umbertogriffo

/*
This is a collections of examples about Apache Spark's RDD Api. These examples aim to help me test the RDD functionality.

References:

http://spark.apache.org/docs/latest/programming-guide.html
http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html
*/
object RddAPI {

umbertogriffo

umbertogriffo

package test.idlike.spark.datastructure;

import org.apache.commons.lang3.SystemUtils;
import org.apache.spark.api.java.JavaDoubleRDD;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.VoidFunction;
import scala.Tuple2;

umbertogriffo

Falsehoods programmers believe about time

This is a compiled list of falsehoods programmers tend to believe about working with time.

Don't re-invent a date time library yourself. If you think you understand everything about time, you're probably doing it wrong.

Falsehoods

• There are always 24 hours in a day.
• February is always 28 days long.
• Any 24-hour period will always begin and end in the same day (or week, or month).

umbertogriffo

import org.apache.spark.sql.functions._

val mediumDf = Seq((0, "zero"), (4, "one")).toDF("id", "value")
val largeDf = Seq((0, "zero"), (2, "two"), (3, "three"), (4, "four"), (5, "five")).toDF("id", "value")

mediumDf.show()
largeDf.show()

/*
+---+-----+

umbertogriffo

#!/usr/bin/env bash

# This installs Python under Rosetta and assign it to pyenv.
# This way of installing Python avoids: ModuleNotFoundError: No module named '_ctypes'
# pyenv has to be installed from Github https://laict.medium.com/install-python-on-macos-11-m1-apple-silicon-using-pyenv-12e0729427a9

version=$1

if [ "$#" -ne 1 ]; then
    echo "Illegal number of parameters. Usage:"

umbertogriffo

MacOS 12 M1 (Apple Silicon) - Build LightGBM from GitHub

Install CMake (3.16 or higher):

brew install cmake
# On MacOS 11 M1 - Apple Silicon
ibrew install cmake

Install OpenMP:

umbertogriffo

This installs Tensorflow under M1 (Apple Silicon)

Note

pyenv has to be installed from Github https://laict.medium.com/install-python-on-macos-11-m1-apple-silicon-using-pyenv-12e0729427a9

Table of contents

• Introduction
• Solution
• Pipenv and Conda