Skip to content

Instantly share code, notes, and snippets.

View Jongbhin's full-sized avatar

Jongbhin Park Jongbhin

  • SK Planet
  • Pangyo
View GitHub Profile
@Jongbhin
Jongbhin / tsv_to_csv.py
Created February 7, 2020 05:40
[tsv to csv] #tsv #csv
# -*- coding: utf-8 -*-
from __future__ import print_function
import os
import csv
output_dir = ''
input_file = os.path.join(output_dir, 'input0000')
output_file = os.path.join(output_dir, 'input0000.tsv')
@Jongbhin
Jongbhin / ssh_key_gen.md
Last active February 7, 2020 06:47
[ssh key gen] #ssh

generate rsa key

ssh-keygen

add to server

ssh-copy-id -i ~/.ssh/id_rsa.pub 1100378@host
@Jongbhin
Jongbhin / make_dey_dict.py
Created February 10, 2020 06:37
[make dict for key value] #dict, #python, #pickle
with open(input_file) as ifd:
csv_reader = csv.reader(ifd, delimiter='\t', quoting=csv.QUOTE_NONE)
token_dict = {}
for counter, row in enumerate(csv_reader):
prd_no = row[0]
token_dict[prd_no] = row[1:]
with open(output_file, mode='w') as wfd:
pickle.dump(token_dict, wfd, protocol=pickle.HIGHEST_PROTOCOL)
@Jongbhin
Jongbhin / python_reference.md
Created March 2, 2020 00:08
python reference

String

get extension

filename, file_extension = os.path.splitext('/path/to/somefile.ext')
@Jongbhin
Jongbhin / python_api_test.py
Created March 2, 2020 07:30
[python api test] #python #api #test
import unittest
import os, sys, time
import json
from subprocess import call
import requests
import csv
parentPath = os.path.abspath("../..")
if parentPath not in sys.path:
sys.path.insert(0, parentPath)
@Jongbhin
Jongbhin / purify.py
Last active March 12, 2020 00:34
[image purify] #image # 11st #url #purify
import csv
import gzip
import os
file_name = 'input0000'
out_file_name = 'input0000.fix'
if os.path.exists(out_file_name):
os.remove(out_file_name)
@Jongbhin
Jongbhin / transformers.md
Last active March 24, 2020 04:39
[Transformers tip] #transformer #bert
ImportError: cannot import name 'WarmupLinearSchedule'
https://github.com/huggingface/transformers/issues/2082
  • 2.1.1을 체크아웃하고 사용하면 해결됨
@Jongbhin
Jongbhin / split_train_dev.py
Last active March 24, 2020 01:34
[Split train dev file] #python # ml #dl
import pandas as pd
from sklearn.model_selection import train_test_split
# split train, dev set and save to file
all_data = pd.read_csv(output_file_balanced, sep='\t')
all_data.sample(frac=1)
train_set, dev_set = train_test_split(all_data, test_size=0.2)
train_set.to_csv(output_file_train, sep='\t', header=False, index=False)
dev_set.to_csv(output_file_dev, sep='\t', header=False, index=False)
@Jongbhin
Jongbhin / git_reference.md
Last active October 28, 2024 01:53
[Git reference] #git #reference

Git add remote

git remote add REMOTE-ID REMOTE-URL
git remote set-url --add --push all REMOTE-URL-1
git remote set-url --add --push all REMOTE-URL-2
# git remote set-url --add --push all git@175.126.56.233:CVPR/clean-scoring-v2-api.git

Git fetch all branches

@Jongbhin
Jongbhin / bash_reference.md
Last active March 30, 2022 02:31
[bash reference] #bash #reference