Skip to content

Instantly share code, notes, and snippets.

View goungoun's full-sized avatar
⚾
...

goungoun

⚾
...
4 followers · 51 following
View GitHub Profile
@goungoun
goungoun / clean_edge.py
Last active June 21, 2019 05:19
vertex에 없는 edge 빼기
import pandas as pd
vtx=pd.read_csv("vtx.tsv", sep=" ")
edge=pd.read_csv("edge.tsv", sep=" ")
edge = edge[edge[':START_ID'].isin(vtx["id:ID"])]
edge = edge[edge[':END_ID'].isin(vtx["id:ID"])]
edge.to_csv("edge2.tsv", sep=" ", index=False, header = True)
# https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf
@goungoun
goungoun / JSONSample.scala
Created October 17, 2018 02:56 — forked from takezoe/JSONSample.scala
Example of scala.util.parsing.json
import scala.util.parsing.json._
val result = JSON.parseFull("""
{"name": "Naoki", "lang": ["Java", "Scala"]}
""")
result match {
case Some(e) => println(e) // => Map(name -> Naoki, lang -> List(Java, Scala))
case None => println("Failed.")
}
@goungoun
goungoun / gist:4bf71b861b4a195200bccb04b3a8b989
Created July 17, 2018 07:33
ngStype ngClass
<h3> <p [ngStyle]="{backgroundColor:getColor()}" [ngClass]="{online: 'online'==='online'}">Oozie Workflow Maker </p></h3>
@goungoun
goungoun / Qwik-lab install_missing.sh
Created May 10, 2018 13:44
#!/bin/bash
sudo apt-get update
sudo apt-get --fix-missing install python-mpltoolkits.basemap python-numpy python-matplotlib
@goungoun
goungoun / GCP-pubsub.py
Created May 5, 2018 12:04
# gcloud beta pubsub topics create sanidego
# gcloud beta pubsub topics publish sandiego "hello"
from google.cloud import pubsub
client = pubsub.Client()
topic = client.topic("sandiego")
topic.create()
@goungoun
goungoun / GCP datalab
Created May 3, 2018 08:31
datalab create mydatalabvm --zone us-central1-b
@goungoun
goungoun / gist:f49795a11755a5aa20226e5b822317ec
Created May 2, 2018 18:51
Google Resources
Compute Engine: https://cloud.google.com/compute/
Storage: https://cloud.google.com/storage/
Pricing: https://cloud.google.com/pricing/
Cloud Launcher: https://cloud.google.com/launcher/
Pricing Philosophy: https://cloud.google.com/pricing/philosophy/
@goungoun
goungoun / gist:e290cd8b63eba7955006e12fdd346a76
Created May 1, 2018 14:25
gcloud dataproc clusters create
gcloud dataproc clusters create <NAME-OF-YOUR-CLUSTER> --subnet default --zone us-central1-b --master-machine-type n1-standard-2 --master-boot-disk-size 500 --num-workers 2 --worker-machine-type n1-standard-2 --worker-boot-disk-size 500 --project <YOUR-PROJECT-ID>
@goungoun
goungoun / big query example.sql
Last active April 25, 2018 17:20
# RULE
# It is a kind of Column Store
# Avoid using * (star) to return all columns, instead use preview
# Check the amount of the processing size by changing query) (500 MB, 1T..)
# Always with LIMIT
# format converts number into string (cannot add)
# cannot use aliased column in where clause like income
# StandardSQL or legacySQL https://cloud.google.com/bigquery/docs/reference/standard-sql/enabling-standard-sql
#standardSQL
@goungoun
goungoun / embulk.md
Created April 10, 2018 04:32
GCP with Embulk

아래 자료는 황장군님의 강의자료를 GCP에서 테스트한 결과입니다.

ETL

embulk를 리눅스에 설치해보자. jar를 copy 하면 됨

curl --create-dirs -o ~/.embulk/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar"