Skip to content

Instantly share code, notes, and snippets.

@goungoun
goungoun / clean_edge.py
Last active Jun 21, 2019
vertex에 없는 edge 빼기
View clean_edge.py
import pandas as pd
vtx=pd.read_csv("vtx.tsv", sep=" ")
edge=pd.read_csv("edge.tsv", sep=" ")
edge = edge[edge[':START_ID'].isin(vtx["id:ID"])]
edge = edge[edge[':END_ID'].isin(vtx["id:ID"])]
edge.to_csv("edge2.tsv", sep=" ", index=False, header = True)
# https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf
@goungoun
goungoun / JSONSample.scala
Created Oct 17, 2018 — forked from takezoe/JSONSample.scala
Example of scala.util.parsing.json
View JSONSample.scala
import scala.util.parsing.json._
val result = JSON.parseFull("""
{"name": "Naoki", "lang": ["Java", "Scala"]}
""")
result match {
case Some(e) => println(e) // => Map(name -> Naoki, lang -> List(Java, Scala))
case None => println("Failed.")
}
View gist:4bf71b861b4a195200bccb04b3a8b989
<h3> <p [ngStyle]="{backgroundColor:getColor()}" [ngClass]="{online: 'online'==='online'}">Oozie Workflow Maker </p></h3>
View Qwik-lab install_missing.sh
#!/bin/bash
sudo apt-get update
sudo apt-get --fix-missing install python-mpltoolkits.basemap python-numpy python-matplotlib
View GCP-pubsub.py
# gcloud beta pubsub topics create sanidego
# gcloud beta pubsub topics publish sandiego "hello"
from google.cloud import pubsub
client = pubsub.Client()
topic = client.topic("sandiego")
topic.create()
View GCP datalab
datalab create mydatalabvm --zone us-central1-b
View gist:f49795a11755a5aa20226e5b822317ec
Compute Engine: https://cloud.google.com/compute/
Storage: https://cloud.google.com/storage/
Pricing: https://cloud.google.com/pricing/
Cloud Launcher: https://cloud.google.com/launcher/
Pricing Philosophy: https://cloud.google.com/pricing/philosophy/
View gist:e290cd8b63eba7955006e12fdd346a76
gcloud dataproc clusters create <NAME-OF-YOUR-CLUSTER> --subnet default --zone us-central1-b --master-machine-type n1-standard-2 --master-boot-disk-size 500 --num-workers 2 --worker-machine-type n1-standard-2 --worker-boot-disk-size 500 --project <YOUR-PROJECT-ID>
View big query example.sql
# RULE
# It is a kind of Column Store
# Avoid using * (star) to return all columns, instead use preview
# Check the amount of the processing size by changing query) (500 MB, 1T..)
# Always with LIMIT
# format converts number into string (cannot add)
# cannot use aliased column in where clause like income
# StandardSQL or legacySQL https://cloud.google.com/bigquery/docs/reference/standard-sql/enabling-standard-sql
#standardSQL
@goungoun
goungoun / embulk.md
Created Apr 10, 2018
GCP with Embulk
View embulk.md

아래 자료는 황장군님의 강의자료를 GCP에서 테스트한 결과입니다.

ETL

embulk를 리눅스에 설치해보자. jar를 copy 하면 됨

curl --create-dirs -o ~/.embulk/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar"