Created
September 23, 2016 07:13
-
-
Save dhwajraj/dd33c82341b083ad1fcaea3ca60dc541 to your computer and use it in GitHub Desktop.
Launching PySpark application which uses tensorflow serving based prediction on kinesis streaming data
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| spark-submit --packages org.apache.spark:spark-streaming-kinesis-asl-assembly_2.10:1.6.1 --master yarn-cluster \ | |
| --deploy-mode cluster --executor-memory 16g --num-executors 2 --driver-memory 6g --executor-cores 4 \ | |
| --conf spark.yarn.executor.memoryOverhead=1000 --py-files /mnt/app.egg /mnt/KinesisReceiver.py |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment