M2shad0w

Default Spark env config error in HDP2.4 installation Default Spark env config error in HDP2.4 installation:

#SPARK_DRIVER_MEMORY="512 Mb"

This line include a wrong config param "512 Mb", it should be "512M" without space. This will cause spark-shell yarn client start error:

Invalid initial heap size.

Suggest to update the template.

M2shad0w

用户信息 第一个要配置的是你个人的用户名称和电子邮件地址。这两条配置很重要，每次 Git 提交时都会引用这两条信息，说明是谁提交了更新，所以会随更新内容一起被永久纳入历史记录：

$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com

如果用了 --global 选项，那么更改的配置文件就是位于你用户主目录下的那个，以后你所有的项目都会默认使用这里配置的用户信息。如果要在某个特定的项目中使用其他名字或者电邮，只要去掉 --global 选项重新配置即可，新的设定保存在当前项目的 .git/config 文件里。

M2shad0w

# Define source, channel, sink
agent1.sources = r1
agent1.channels = ch2
agent1.sinks = avro-sink1

# Configure channel

agent1.channels.ch2.type = file
agent1.channels.ch2.capacity = 1000000
agent1.channels.ch2.transactionCapacity = 500000

M2shad0w

name := "DES and AES encryption in Scala"

version := "1.0"

scalaVersion := "2.9.1"

libraryDependencies += "commons-codec" % "commons-codec" % "1.6"

M2shad0w

#!/bin/bash
#Zeppelin installation on Spark EMR cluster

sudo yum update -y
sudo yum install -y git
# Installing maven: https://gist.github.com/sebsto/19b99f1fa1f32cae5d00
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

M2shad0w

Reading specs from /Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/build/build-cc-core-pass-2/./gcc/specs
COLLECT_GCC=/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/build/build-cc-core-pass-2/./gcc/xgcc
Target: xtensa-lx106-elf
Configured with: /Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/src/gcc-4.8.2/configure --build=x86_64-build_apple-darwin15.2.0 --host=x86_64-build_apple-darwin15.2.0 --target=xtensa-lx106-elf --prefix=/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/buildtools --with-local-prefix=/Volumes/esp-open-sdk/esp-open-sdk/xtensa-lx106-elf/xtensa-lx106-elf/sysroot --disable-libmudflap --with-sysroot=/Volumes/esp-open-sdk/esp-open-sdk/xtensa-lx106-elf/xtensa-lx106-elf/sysroot --with-newlib --enable-threads=no --disable-shared --with-pkgversion='crosstool-NG 1.20.0' --disable-__cxa_atexit --with-gmp=/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/buildtools --with-mpfr=/Volumes/esp-open-sdk/es

M2shad0w

Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
TungstenExchange hashpartitioning(additional#3,200), None
+- TungstenAggregate(key=[additional#3], functions=[(count(additional#3),mode=Partial,isDistinct=false)], output=[additional#3,count#81L])
   +- Project [additional#3]
      +- Filter (((created_at#0 >= 2016-01-25) && (created_at#0 < 2016-01-26)) && (action#2 = search))
         +- InMemoryColumnarTableScan [additional#3,created_at#0,action#2], [(created_at#0 >= 2016-01-25),(created_at#0 < 2016-01-26),(action#2 = search)], InMemoryRelation [created_at#0,eventable_type#1,action#2,additional#3,track_id#4,device#5,city_code#6], true, 10000, StorageLevel(true, true, false, true, 1), ConvertToUnsafe, Some(tracklogs)

	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)

M2shad0w

This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by GNU C Runtime Library configure 1.0, which was
generated by GNU Autoconf 2.64.  Invocation command line was

  $ /Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/src/gcc-4.8.2/libgcc/configure --cache-file=./config.cache --with-cross-host=x86_64-build_apple-darwin15.2.0 --prefix=/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/buildtools --with-local-prefix=/Volumes/esp-open-sdk/esp-open-sdk/xtensa-lx106-elf/xtensa-lx106-elf/sysroot --with-sysroot=/Volumes/esp-open-sdk/esp-open-sdk/xtensa-lx106-elf/xtensa-lx106-elf/sysroot --with-newlib --enable-threads=no --disable-shared --with-pkgversion=crosstool-NG 1.20.0 --disable-__cxa_atexit --with-isl=/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/buildtools --with-libelf=/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/xtensa-lx106-elf/buildtools --enable-lto --ena

M2shad0w

[INFO ]  Performing some trivial sanity checks
[DEBUG]  Testing '! ( -n  )'
[DEBUG]  Testing '! ( -n  )'
[DEBUG]  Testing '! ( -n  )'
[DEBUG]  Testing '! ( -n  )'
[DEBUG]  Testing '! ( -n  )'
[DEBUG]  Testing '! ( -n  )'
[DEBUG]  ==> Executing: 'mkdir' '-p' '/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build' 
[DEBUG]  ==> Executing: 'rm' '-f' '/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/backtrace' 
[DEBUG]  ==> Executing: 'touch' '/Volumes/esp-open-sdk/esp-open-sdk/crosstool-NG/.build/foo' 

M2shad0w