This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
class LoadService(metaclass=abc.ABCMeta): | |
""" | |
Abstract Service class defines all the api which Pinion | |
framework will expose to the end user. Concrete subclasses | |
has to provided their own implementation for these apis. | |
In case if a concrete subclass doesn't provide an | |
implementation then a NoImplementationFoundError error | |
will be raised by the LoadService. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
class LoadService(metaclass=abc.ABCMeta): | |
""" | |
Abstract Service class defines all the api which Pinion | |
framework will expose to the end user. Concrete subclasses | |
has to provided their own implementation for these apis. | |
In case if a concrete subclass doesn't provide an | |
implementation then a NoImplementationFoundError error | |
will be raised by the LoadService. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<property> | |
<name>spark.master</name> | |
<value>yarn</value> | |
</property> | |
<property> | |
<name>spark.eventLog.enabled</name> | |
<value>true</value> | |
</property> | |
<property> | |
<name>spark.eventLog.dir</name> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
export SPARK_HOME=/usr/lib/spark | |
hadoop fs -mkdir /spark-jars | |
hadoop fs -put ${SPARK_HOME}/jars/*.jar /spark-jars/ | |
hadoop fs -rm /spark-jars/*hive*1.2.1*.jar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
export SPARK_HOME=/usr/lib/spark | |
export HIVE_HOME=/usr/lib/hive | |
cp ${SPARK_HOME}/jars/scala-library-2.11.12.jar ${HIVE_HOME}/lib/ | |
cp ${SPARK_HOME}/jars/spark-core_2.11-2.4.4.jar ${HIVE_HOME}/lib/ | |
cp ${SPARK_HOME}/jars/spark-network-common_2.11-2.4.4.jar ${HIVE_HOME}/lib/ | |
cp ${SPARK_HOME}/jars/spark-unsafe_2.11-2.4.4.jar ${HIVE_HOME}/lib/ |