Ye Yin hustcat

## hadoop_tips.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              0 stars
            
          
                hustcat
                / hadoop_tips.md
            
            
              Created
              March 31, 2017 08:52
            
              
                Hadoop tips
              
          
    hive


Impala与Hive的比较


## minikube_tips.md

      
              1 file
            
          
              2 forks
            
          
              0 comments
            
          
              1 star
            
          
                hustcat
                / minikube_tips.md
            
            
              Last active
              September 8, 2018 23:12
            
              
                Minikube tips
              
          
    Install

Install minikube:
# curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.18.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Install kubectl:

  
## spark_internal_posts.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              0 stars
            
          
                hustcat
                / spark_internal_posts.md
            
            
              Last active
              March 22, 2017 11:35
            
              
                spark实现相关文章
              
          
Apache Spark: core concepts, architecture and internals
https://spark-summit.org/2014/a-deeper-understanding-of-spark-internals/
Resource Allocation in Mesos: Dominant Resource Fairness

scheduler


Spark源码分析之-scheduler模块
深入理解Spark 2.1 Core （二）：DAG调度器的原理与源码分析


## spark_compile_debug_errors.md

      
              1 file
            
          
              0 forks
            
          
              1 comment
            
          
              7 stars
            
          
                hustcat
                / spark_compile_debug_errors.md
            
            
              Last active
              November 18, 2021 13:04
            
              
                spark编译调试错误
              
          
运行exmaple中示例错误1:

Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration

VM options启动参数增加:
-Dspark.master=local


## spark_internal.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              0 stars
            
          
                hustcat
                / spark_internal.md
            
            
              Last active
              March 27, 2017 06:47
            
              
                spark源码
              
          
    Spark driver

Spark driver的初始化主要围绕SparkContext的初始化，SparkContext初始化完之后，才能向Spark集群提交任务。

spark context初始化

    // Create and start the scheduler
    val (sched, ts) = SparkContext.createTaskScheduler(this, master, deployMode) ///创建任务调度器
    _schedulerBackend = sched


## virt-install-fedora
#!/bin/bash

if ! [ $# -eq 1 ]; then
    echo "Usage: $0 <node-name>"
    exit 1
fi

# Directory where everything happens
DIR=~/work/virt-install
# The image downloaded from the http://fedoraproject.org/en/get-fedora#clouds site
	#!/bin/bash

	if ! [ $# -eq 1 ]; then
	echo "Usage: $0 <node-name>"
	exit 1
	fi

	# Directory where everything happens
	DIR=~/work/virt-install
	# The image downloaded from the http://fedoraproject.org/en/get-fedora#clouds site