hadoop + peacock

gistfile1.txt

[编译libhdfs]

https://github.com/forward/node-hdfs

[编译github.com/zyxar/hdfs]
因为zyxar/hdfs是根据hadoop 1.0.3编写, 需要修改:
1. cgo部分
删除github.com/zyxar/hdfs/hdfs.h, 修改import "C"为:

/*
#cgo linux CFLAGS: -I/data1/home/xueminzhao/soft/hadoop-0.20.9/src/c++/install/include/hadoop -I/usr/local/jdk/include
#cgo linux LDFLAGS: -L/data1/home/xueminzhao/soft/hadoop-0.20.9/src/c++/install/lib -lhdfs -L/usr/local/jdk/jre/lib/amd64/server -ljvm
#include "hdfs.h"
int getlen(char*** ptr) {
    int i = 0;
    while (ptr[i] != NULL) ++i;
    return i;
}
int getl(char*** ptr) {
    int i = 0;
    while (ptr[0][i] != NULL) ++i;
    return i;
}
char* getstring(char*** ptr, int i, int j) {
    return ptr[i][j];
}
*/
import "C"

2. func ConnectAsUser, 修改为:

  group := C.CString("supergroup")
  defer C.free(unsafe.Pointer(group))

  ret, err := C.hdfsConnectAsUser(h, C.tPort(port), u, &group, 1)

编译得到coordinator, data_server, sampling_server

go install -a

[run peacock]
由于olymplic集群中的机器没有libhdfs.so.0，所以需要将so和其他binary同时发布。
export LD_LIBRARY_PATH=$remote_root_dir/bin:$JAVA_HOME/jre/lib/amd64/server
同时export CLASSPATH

CLASSPATH=".:..:$JAVA_HOME/lib:$JAVA_HOME/jre/lib"
for i in $HADOOP_HOME/*.jar ; do
    CLASSPATH=$CLASSPATH:$i
done
for i in $HADOOP_HOME/lib/*.jar ; do
    CLASSPATH=$CLASSPATH:$i
done
CLASSPATH=$CLASSPATH:$HADOOP_HOME/contrib/vaidya/hadoop-vaidya-0.20.9.jar:\
$HADOOP_HOME/contrib/vaidya/hadoop-vaidya-0.20.9.jar:\
$HADOOP_HOME/contrib/thriftfs/hadoop-thriftfs-0.20.9.jar:\
$HADOOP_HOME/contrib/streaming/hadoop-streaming-0.20.9.jar:\
$HADOOP_HOME/contrib/index/hadoop-index-0.20.9.jar:\
$HADOOP_HOME/contrib/hdfsproxy/hdfsproxy-1.0.jar:\
$HADOOP_HOME/contrib/gridmix/hadoop-gridmix-0.20.9.jar:\
$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-0.20.9.jar:\
$HADOOP_HOME/contrib/failmon/hadoop-failmon-0.20.9.jar:\
$HADOOP_HOME/contrib/datajoin/hadoop-datajoin-0.20.9.jar:\
$HADOOP_HOME/contrib/capacity-scheduler/hadoop-capacity-scheduler-0.20.9.jar

注意: 
1. 和hadoop-1.0.3不同, 0.20.9有contrib目录下的一堆jar.
2. olymplic集群的机器上JAVA_HOME和HADOOP_HOME没有导出, 需要在脚本中配置

JAVA_HOME=/usr/java/default
HADOOP_HOME=/data/users/hadoop/hadoop

3. tee命令会同时输出到console, 若不定向到文件, 使用ssh -l远程执行nohup起server会失败.