Skip to content

Instantly share code, notes, and snippets.

View shibacow's full-sized avatar

shibacow shibacow

21 followers · 6 following
View GitHub Profile
@shibacow
shibacow / test.sql
Created January 19, 2016 18:27
select count_distinct(user_id) as count,
date_format(time,'%Y-%m-%d') as date,
world
from [table20150101-201531]
group by date,world
order by date,world;
@shibacow
shibacow / slide.md
Last active December 24, 2015 17:19
http://glide.so/shibacow/6834756

#ニコニコデータセット分析(構築偏) 芝尾幸一郎(@shibacow) #構築手順

  • ニコニコ動画データセット取得
  • コメントデータセットにsmidとファイル名を追記する
  • S3へのデータアップロード
  • AmazonEMRにデータを入れる
  • 使いやすいようにデータをコンバート

#詳しくはこちら

@shibacow
shibacow / gist:5517164
Created May 4, 2013 11:02
Impalad odbc connector odbcinst.ini samplae
[ODBC Drivers]
Hive Driver=Installed
[Hive Driver]
Driver=/usr/lib/libhiveodbc.so.1
Description=Hive Driver
Setup=/usr/lib/libhiveodbc.so.1
APILevel=2
ConnectFunctions=YYY
DriverODBCVer=1.0
FileUsage=0
@shibacow
shibacow / gist:5517157
Last active December 16, 2015 23:59
impalad ODBC Connector Sample ini
[ODBC Data Sources]
;IMPALA-SERVER=Hive ODBC Driver
IMPALA-SERVER=Hive Driver
[IMPALA-SERVER]
Driver=/usr/lib/libhiveodbc.so.1
Description=Hive ODBC Driver
;specify your impalad server
Host=192.168.1.6
Port=21000
Database=
@shibacow
shibacow / gist:5517142
Created May 4, 2013 10:54
impla odbc conect sapmle by python
#!/usr/bin/python
# -*- coding:utf-8 -*-
import pyodbc
def main():
con=pyodbc.connect('DSN=IMPALA-SERVER;UID=;PWD=;Database=Default;')
cursor=con.cursor()
sql='select count(*) from sample_07'
cursor.execute(sql)
for r in cursor.fetchall():
print r
@shibacow
shibacow / gist:5517129
Created May 4, 2013 10:51
odbc connet sample program
<?php
function odbc_test(){
$DSN='IMPALA-SERVER';
$USER='';
$PASS='';
$con=odbc_connect($DSN,$USER,$PASS);
if($con){
$sql='select count(*) from sample_07';
$rs = odbc_exec($con,$sql);
while($row = odbc_fetch_array($rs)){
@shibacow
shibacow / LINPACK BENCH on aws c4.8xlarge ubuntu
Last active November 13, 2015 11:45
LINPACK BENCH executed on AWS c4.8xlarge Ubuntu 14.04
ubuntu@ip-*-*-*-*:~/l_mklb_p_11.3.1.002/benchmarks_11.3.1/linux/mkl/benchmarks/linpack$ ./runme_xeon64
This is a SAMPLE run script for SMP LINPACK. Change it to reflect
the correct number of CPUs/threads, problem input files, etc..
./runme_xeon64: 37: [: -gt: unexpected operator
Fri Nov 13 11:26:24 UTC 2015
Intel(R) Optimized LINPACK Benchmark data
Current date/time: Fri Nov 13 11:26:24 2015
CPU frequency: 3.126 GHz
@shibacow
shibacow / LINPACK BENCH on osx mac book pro
Last active November 13, 2015 11:39
LINPACKを走らせてみる。
2015年 11月13日 金曜日 20時21分05秒 JST
Intel(R) Optimized LINPACK Benchmark data
Current date/time: Fri Nov 13 20:21:05 2015
CPU frequency: 3.091 GHz
Number of CPUs: 1
Number of cores: 2
Number of threads: 4
@shibacow
shibacow / prosess_test.ex
Last active September 23, 2015 16:11
マルチコア環境下でelxirでどの程度マルチコアを利用するか評価する。こちらに記事を書いた。　http://qiita.com/shibacow/items/8f2e8aaff0febb70b934
require Logger
defmodule KV do
def start_link do
Task.start_link(fn -> loop(%{}) end)
end
defp loop(map) do
receive do
{:get,key,caller} ->
send caller,{:get,:ok,Map.get(map,key)}
loop(map)
@shibacow
shibacow / mix.exs
Created September 11, 2015 10:24
elixirプロセステスト
defmodule TestProcess.Mixfile do
use Mix.Project
def project do
[app: :test_process,
version: "0.0.1",
elixir: "~> 1.0",
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
escript: escript,