Skip to content

Instantly share code, notes, and snippets.

View AKB428's full-sized avatar
🥳
Out sick

AKB428 AKB428

🥳
Out sick
57 followers · 42 following
View GitHub Profile
@AKB428
AKB428 / .zshenv
Created May 26, 2015 03:12
zshの設定
#http://webtech-walker.com/archive/2008/12/15101251.html
#補間
autoload -U compinit
compinit
#文字コード
export LANG=ja_JP.UTF-8
#プロンプト
@AKB428
AKB428 / SparkSubmit_sample1.scala
Created May 24, 2015 18:14
spark-submitにjarを渡すためにsbt assemblyするためのbuild.sbt ref: http://qiita.com/AKB428/items/ac0416fda8e01973db0f
import org.apache.spark
import org.apache.spark.{SparkContext, SparkConf}
/**
* Created by Siori on 15/05/22.
*/
object SparkSubmit_sample1 {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("WordCount Application")
val sc = new SparkContext(conf)
@AKB428
AKB428 / CDH5.4LINK
Created May 24, 2015 14:43
CDH5.4LINK
@AKB428
AKB428 / Spark-submit success log
Last active August 29, 2015 14:21
Spark-submit success log
[cloudera@quickstart spark-word-count-sample]$ spark-submit target/scala-2.11/sparkWorkCount-assembly-1.0.jar hdfs://localhost/user/cloudera/20141204_000141.csv 君の嘘
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/avro/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/05/24 06:30:05 INFO SparkContext: Running Spark version 1.3.0
15/05/24 06:30:08 INFO SecurityManager: Changing view acls to: cloudera
@AKB428
AKB428 / SparkSubmmmitCDH起動失敗
Created May 24, 2015 12:57
SparkSubmmmitCDH 起動失敗
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/05/24 05:53:30 INFO SparkContext: Running Spark version 1.3.0
15/05/24 05:53:33 INFO SecurityManager: Changing view acls to: cloudera
15/05/24 05:53:33 INFO SecurityManager: Changing modify acls to: cloudera
15/05/24 05:53:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloudera); users with modify permissions: Set(cloudera)
15/05/24 05:53:35 INFO Slf4jLogger: Slf4jLogger started
15/05/24 05:53:36 INFO Remoting: Starting remoting
15/05/24 05:53:36 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@quickstart.cloudera:55466]
15/05/24 05:53:36 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@quickstart.cloudera:55466]
@AKB428
AKB428 / SparkAssembly
Last active August 29, 2015 14:21
SparkAssembly
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /home/cloudera/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.3.1.jar:com/google/common/base/Absent.class
[error] /home/cloudera/.ivy2/cache/com.google.guava/guava/bundles/guava-14.0.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /home/cloudera/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.3.1.jar:com/google/common/base/Function.class
[error] /home/cloudera/.ivy2/cache/com.google.guava/guava/bundles/guava-14.0.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /home/cloudera/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.3.1.jar:com/google/common/base/Optional$1$1.class
[error] /home/cloudera/.ivy2/cache/com.google.guava/guava/bundles/guava-14.0.1.jar:com/goog
@AKB428
AKB428 / gist:2ae6a05b04875aecae82
Last active August 29, 2015 14:21
CDH VM5.4
起動
SSHはcloudera / clouderaでROOTで入れる
これで　Windows Terminal Soft->CDHはできる。
あとは外からVPN経由で->Windows->VM CDHができるよう
ポート２２のポートフォワーディングを追加
Windowsのポート２２を空ける
@AKB428
AKB428 / file0.txt
Last active August 29, 2015 14:21
HighchartsでHTML特殊文字を文字として表示する方法 ref: http://qiita.com/AKB428/items/0135e43ef73059710b60
<g class="highcharts-legend-item" zIndex="1" transform="translate(8,48)"><text x="21" y="15" style="color:#333333;font-size:12px;font-weight:bold;cursor:pointer;fill:#333333;" text-anchor="start" zIndex="2">東京喰種トーキョーグール&amp;radic;A</text><rect x="0" y="4" width="16" height="12" zIndex="3" fill="#f15c80"></rect></g>
@AKB428
AKB428 / insConoha1
Last active August 29, 2015 14:21
新Conoha　インストールメモ
oracle jdk8インストール
yum install tmux
curl -O ついいたーデータ
最新版のsbtインストール
git のインストール
@AKB428
AKB428 / maki_csv_parser.rb
Created May 20, 2015 02:03
maki_csv_parser.rb
#ツイート文中の改行を削除
write_first_filename = "output1_" + ARGV[0]
write_second_filename = "output2_" + ARGV[0]
file_out = open(write_first_filename,"w")
file = open(ARGV[0],"r:UTF-8")
file.each_line do |line|
line2 = line.strip
#ツイート末尾の場合のみ改行付きで書き込み