Skip to content

Instantly share code, notes, and snippets.

@wyukawa
wyukawa / レコメンデーションで似ている人を出すHiveQL
Created July 16, 2011 14:50
みてわかる クラウド マガジン Vol.3の特集4をHiveQLで一部書いてみた
select
*
from
(
select
dis.user as user,
dis.sim as sim
from
(
select
@wyukawa
wyukawa / gist:2559954
Created April 30, 2012 16:48
HiveのソースをEclipseに取り込む手順
http://mt.orz.at/archives/2012/04/hive-1.html
のupdate版
svn checkout http://svn.apache.org/repos/asf/hive/trunk hive-trunk
cd hive-trunk
wget https://issues.apache.org/jira/secure/attachment/12524106/HIVE-2902.3.patch.txt
patch -p0 < HIVE-2902.3.patch.txt
ant clean package eclipse-files #ただしmakeがないとcleanで怒られるのでオレはant package eclipse-filesしかしてない
@wyukawa
wyukawa / gist:2587178
Created May 3, 2012 16:51
select key from src group by key;の実行計画
PREHOOK: query: explain select key from src group by key
PREHOOK: type: QUERY
POSTHOOK: query: explain select key from src group by key
POSTHOOK: type: QUERY
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL key))) (TOK_GROUPBY (TOK_TABLE_OR_COL key))))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
@wyukawa
wyukawa / gist:2667266
Created May 12, 2012 15:51
Hiveのソースを読む際の参考資料
Hive Anatomy
http://www.slideshare.net/nzhang/hive-anatomy
Internal Hive
http://www.slideshare.net/recruitcojp/internal-hive
Indexed Hive
http://www.slideshare.net/NikhilDeshpande/indexed-hive
Hive – A Petabyte Scale Data Warehouse Using Hadoop
@wyukawa
wyukawa / gist:2757110
Created May 20, 2012 07:01
insert selectした場合のexplain実行結果
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME hoge))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME piyo))) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF))))
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-5 depends on stages: Stage-1 , consists of Stage-4, Stage-3
Stage-4
Stage-0 depends on stages: Stage-4, Stage-3
Stage-2 depends on stages: Stage-0
@wyukawa
wyukawa / gist:2792659
Created May 26, 2012 06:58
hive.cli.print.header
hive> show tables;
OK
aaa
Time taken: 0.124 seconds
hive> desc aaa;
OK
a int
b int
Time taken: 0.096 seconds
hive> select * from aaa limit 1;
@wyukawa
wyukawa / gist:2868224
Created June 4, 2012 13:01
グレートギャツビーの冒頭
In my younger and more vulnerable years my father gave me some advice that I’ve been turning over in my mind ever since.
“Whenever you feel like criticizing any one,” he told me, “just remember that all the people in this world haven’t had the advantages that you’ve had.”
僕がまだ年若く、心に傷を負いやすかったころ、父親がひとつ忠告を与えてくれた。その言葉について僕は、ことあるごとに考えをめぐらせてきた。
「誰かのことを批判したくなったときには、こう考えるようにするんだよ。すべての人が、おまえのように恵まれた条件を与えられたわけではないのだと」
@wyukawa
wyukawa / gist:3074672
Created July 9, 2012 06:48
サブクエリで書いた方が良いクエリ
from (
select
...
from
aaa a
left outer join
bbb b
on
a.id=b.id
and
def solve(length, ants):
minimum = 0
maximum = 0
for ant in ants:
min_ant = min(ant, length-ant)
if min_ant > minimum:
minimum = min_ant
@wyukawa
wyukawa / gist:3340447
Created August 13, 2012 13:01
くじびき問題
def solve(m, k):
for i1 in k:
for i2 in k:
for i3 in k:
for i4 in k:
if i1+i2+i3+i4 == m:
print "Yes %d+%d+%d+%d=%d" % (i1, i2, i3, i4, m)
break
else:
continue