This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// returns a CompletableFuture which completes only when all the completable futures completes | |
CompletableFuture.allOf(completableFuture1, completableFuture2, ...); | |
// returns a CompletableFuture which completes when any one of the completable future completes | |
CompletableFuture.anyOf(completableFuture1, completableFuture2, ...); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// callback is executed after the completion of | |
// completableFuture1, completableFuture2 | |
completableFuture1.thenCombine(completableFuture2, biFunction); | |
completableFuture1.thenAcceptBoth(completableFuture2, biConsumer); | |
completableFuture1.runAfterBoth(completableFuture2, runnable); | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CompletableFuture.supplyAsync(supplier) | |
.thenApply(function) | |
.thenAccept(consumer) | |
.thenRun(runnableTask); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ExecutorService executorService = Executors.newSingleThreadExecutor(); | |
CompletableFuture.runAsync(task1, executorService) | |
.thenRunAsync(task2, executorService) | |
.join(); | |
executorService.shutdown(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Supplier<List<Integer>> supplier = () -> { | |
try { | |
Thread.sleep(1000); | |
} catch (InterruptedException e) { | |
e.printStackTrace(); | |
} | |
return Arrays.asList(1, 2, 3); | |
}; | |
Consumer<List<Integer>> consumer = input -> { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ExecutorService executorService = Executors.newSingleThreadExecutor(); | |
Runnable runnableTask = () -> System.out.println("This is a Runnable task"); | |
Future<?> future1 = executorService.submit(runnableTask); | |
future1.wait(); | |
Callable<String> callableTask = () -> "This is a Callable task"; | |
Future<String> future2 = executorService.submit(callableTask); | |
String result = future2.get(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
export SPARK_HOME="/path/to/spark-home" | |
export PATH="$SPARK_HOME/bin:$PATH" | |
export PYSPARK_SUBMIT_ARGS="pyspark-shell" | |
export PYSPARK_DRIVER_PYTHON=ipython | |
export PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from pyspark.sql import SparkSession | |
from pyspark.sql.functions import col, lower | |
import sys | |
def main(argv): | |
spark = SparkSession.builder\ | |
.appName('spark-movies-app')\ | |
.getOrCreate() |