-
Kerberized Cluster
-
Enable hive interactive server in hive
-
Get following details from hive for spark or try this HWC Quick Test Script
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// define the name of the Azure Databricks notebook to run | |
val notebookToRun = ??? | |
// define some way to generate a sequence of workloads to run | |
val jobArguments = ??? | |
// define the number of workers per job | |
val workersPerJob = ??? | |
import java.util.concurrent.Executors |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# This function is used to reduce memory of a pandas dataframe | |
# The idea is cast the numeric type to another more memory-effective type | |
# For ex: Features "age" should only need type='np.int8' | |
# Source: https://www.kaggle.com/gemartin/load-data-reduce-memory-usage | |
def reduce_mem_usage(df): | |
""" iterate through all the columns of a dataframe and modify the data type | |
to reduce memory usage. | |
""" | |
start_mem = df.memory_usage().sum() / 1024**2 | |
print('Memory usage of dataframe is {:.2f} MB'.format(start_mem)) |
Copyright © 2016-2018 Fantasyland Institute of Learning. All rights reserved.
A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.
val square : Int => Int = x => x * x
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
All exercises are attempted on https://coderpad.io |
- General
- Mastering Observable - http://docs.couchbase.com/developer/java-2.0/observables.html
- Android
- Grokking RxJava, Part 1-4 - http://blog.danlew.net/2014/09/15/grokking-rxjava-part-1/
- Loading data from multiple sources with RxJava - http://blog.danlew.net/2015/06/22/loading-data-from-multiple-sources-with-rxjava/
- Don't break the chain: use RxJava's compose() operator - http://blog.danlew.net/2015/03/02/dont-break-the-chain/
- Pro RxJava
- Hot and Cold Observable - http://davesexton.com/blog/post/Hot-and-Cold-Observables.aspx
- To use subject or not to use subject - http://davesexton.com/blog/post/To-Use-Subject-Or-Not-To-Use-Subject.aspx
- RxJava Threading Examples - http://www.grahamlea.com/2014/07/rxjava-threading-examples/