Skip to content

Instantly share code, notes, and snippets.

View myui's full-sized avatar

Makoto YUI myui

136 followers · 4 following
View GitHub Profile
@myui
myui / tez-amconfiguration.md
Last active August 29, 2015 14:04

AMConfiguration should be remained public because it is called by Hive in org.apache.hadoop.hive.ql.exec.tez.TezSessionState.

The following exception caused when using Tez-trunk on Hive 0.13.1/Hadoop 2.4.1.

Exception in thread "main" java.lang.IllegalAccessError: tried to access class org.apache.tez.client.AMConfiguration fro
m class org.apache.hadoop.hive.ql.exec.tez.TezSessionState
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:184)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:123)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:355)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
@myui
myui / tfidf_test.tsv
Created September 10, 2014 16:39
We can make this file beautiful and searchable if this error is corrected: No tabs found in this TSV file in line 0.
1|Justice, in its broadest context, includes both the attainment of that which is just and the philosophical discussion of that which is just. The concept of justice is based on numerous fields, and many differing viewpoints and perspectives including the concepts of moral rightness based on ethics, rationality, law, religion, equity and fairness. Often, the general discussion of justice is divided into the realm of social justice as found in philosophy, theology and religion, and, procedural justice as found in the study and application of the law.
2|Wisdom (sophia) is the ability to think and act using knowledge, experience, understanding, common sense, and insight.[1] Wisdom has been regarded as one of four cardinal virtues; and as a virtue, it is a habit or disposition to perform the action with the highest degree of adequacy under any given circumstance. This implies a possession of knowledge or the seeking thereof in order to apply it to the given circumstance. This involves an understanding of people,
@myui
myui / gist:fc15755d1ab3ad1e3f2a
Created September 22, 2014 06:25
HDFS caching 
hdfs dfsadmin -report

# プールの作成
hdfs cacheadmin -addPool expr

#プールの削除
hdfs cacheadmin -removePool expr

# プールの確認
@myui
myui / kddcup12track2-mllib.md
Last active August 29, 2015 14:07

https://github.com/myui/hivemall/wiki/KDDCup-2012-track-2-CTR-prediction-dataset

create table training_libsvmfmt
  ROW FORMAT DELIMITED 
    LINES TERMINATED BY "\n"
  STORED AS TEXTFILE
AS
select 
  concat(label," ",features[1],":1.0 ",features[2],":1.0 ",features[3],":1.0 ",features[4],":1.0 ",features[5],":1.0 ",features[6],":1.0 ",features[7],":1.0 ",features[8],":1.0 ",features[9],":1.0 ",features[10],":1.0 ",features[11],":1.0 ",features[12],":1.0") as line
@myui
myui / generate_cv.sh
Last active August 29, 2015 14:14
generate_cv.sh
#!/bin/bash
nfolds=10
evalsql=""
cat /dev/null > generate_cv.sql
echo "-- ${nfolds}-folds cross validation" >> generate_cv.sql
for i in `seq 1 ${nfolds}`;
do
@myui
myui / generate_cv.sql
Last active August 29, 2015 14:14
generate_cv.sql
-- 10-folds cross validation
create or replace view training1
as
select
userid, movieid, rating
from
ratings_groupded
where
gid != 1;
@myui
myui / hadoopsummit14_slideshare.png
Last active August 29, 2015 14:14
hivemall misc
hadoopsummit14_slideshare.png
@myui
myui / file0.txt
Last active August 29, 2015 14:14
HivemallでMatrix Factorization ref: http://qiita.com/myui/items/dccb4f58799f080e24ab
R \approx P^{T}Q
@myui
myui / tez_autoconvertjoin_bug
Last active August 29, 2015 14:15
Vertex failed, vertexName=Map 22, vertexId=vertex_1424697940054_0003_2_36, diagnostics=[Task failed, taskId=task_1424697940054_0003_2_36_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"gid":2,"userid":5323,"movieid":3462,"rating":5}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.S
@myui
myui / autoconvertjoin_bug
Last active August 29, 2015 14:16
2015-02-24 20:13:52,028 INFO [TezChild] exec.MapOperator: Initializing Self MAP[20]
2015-02-24 20:13:52,028 INFO [TezChild] exec.TableScanOperator: Initializing Self TS[8]
2015-02-24 20:13:52,029 INFO [TezChild] exec.TableScanOperator: Operator 8 TS initialized
2015-02-24 20:13:52,029 INFO [TezChild] exec.TableScanOperator: Initializing children of 8 TS
2015-02-24 20:13:52,029 INFO [TezChild] exec.FilterOperator: Initializing child 9 FIL
2015-02-24 20:13:52,029 INFO [TezChild] exec.FilterOperator: Initializing Self FIL[9]
2015-02-24 20:13:52,029 INFO [TezChild] exec.FilterOperator: Operator 9 FIL initialized
2015-02-24 20:13:52,029 INFO [TezChild] exec.FilterOperator: Initializing children of 9 FIL
2015-02-24 20:13:52,029 INFO [TezChild] exec.SelectOperator: Initializing child 10 SEL
2015-02-24 20:13:52,029 INFO [TezChild] exec.SelectOperator: Initializing Self SEL[10]