Skip to content

Instantly share code, notes, and snippets.

Avatar
🎯
Focusing

Ryota rkaneko

🎯
Focusing
View GitHub Profile
View changelog_between_git_tags.sh
#!/bin/bash
PROGNAME=$(basename $0)
VERSION="1.0"
usage() {
echo "Usage: $PROGNAME [OPTIONS] FILE"
echo " This script is ~."
echo
echo "Options:"
@rkaneko
rkaneko / qgis.md
Last active Feb 29, 2020
Install qgis on Ubuntu 18.04
View qgis.md

How to install QGIS on Ubuntu 18.04

$ sudo apt-get install python3-software-properties

$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F42ED6FBAB17C654

$ sudo apt-get update
@rkaneko
rkaneko / print_cpu_usage.py
Created Feb 18, 2020
Profile CPU usage in Python sing psutil.
View print_cpu_usage.py
import time
import multiprocessing as mp
import psutil
from typing import Any, Callable, List, Tuple
def monitor(target: Callable[..., Any], args: Tuple[Any, ...]) -> List[float]:
worker_process = mp.Process(target=target, args=args)
worker_process.start()
p = psutil.Process(worker_process.pid)
@rkaneko
rkaneko / sample.py
Created Feb 17, 2020
Put parquet file on MinIO (S3 compatible storage) using pyarrow and s3fs.
View sample.py
from typing import Any, Dict, List
import s3fs
from pyarrow import Table, parquet as pq
from pandas import DataFrame, Series
def to_df(data: List[Dict[str, Any]]) -> DataFrame:
df = DataFrame()
for item in data:
indexes = []
values = []
@rkaneko
rkaneko / psql-grant.md
Last active Dec 26, 2019
psql grant, logs configurations
View psql-grant.md
# display privileges
SELECT grantee, privilege_type FROM information_schema.role_table_grants WHERE table_name='some_table_name';

  grantee  | privilege_type
-----------+----------------
 root      | INSERT
 root      | SELECT
 root      | UPDATE
 root      | DELETE
@rkaneko
rkaneko / docker-disk-troubles.md
Last active Dec 20, 2018
Docker disk trouble shooting
View docker-disk-troubles.md

Case: No space left on device error

# remove unnecessary images
docker rm $(docker ps -q -f 'status=exited')
docker rmi $(docker images -q -f "dangling=true")

Case: Volumes are full

@rkaneko
rkaneko / README.md
Last active Dec 19, 2018
awk sample for mapping some column values in some .csv file
View README.md

awk command sample for mapping some column values in some .csv file

$ cat origin.csv
id,created,notes
1,1518393600000,blahblah
2,1518393600000,foobar

# Case1: We want to all column2 values (e.g. unix timestamp-ish value) to formatted datetime
$ head -1 origin.csv > /dev/stdout && tail origin.csv -n+2 | awk -F "," '{gsub($2, strftime("%Y-%m-%d %H:%M:%S", substr($2, 0, 10))); print}' >> /dev/stdout
@rkaneko
rkaneko / test_connection.py
Created Nov 25, 2018
Verify to get Postgresql connection
View test_connection.py
import logging
import psycopg2
import sys
import traceback
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__file__)
log.setLevel(logging.INFO)
@rkaneko
rkaneko / .bashrc.local
Last active Sep 12, 2018
Selecting command history extension using peco on bash
View .bashrc.local
export HISTCONTROL="ignoredups"
_peco_select_history() {
local l=$(HISTTIMEFORMAT= history | tac | sed -e 's/^\s*[0-9]*\+\s\+//' | peco --query "$READLINE_LINE")
READLINE_LINE="$l"
READLINE_POINT=${#l}
}
bind -x '"\C-r":_peco_select_history'
View README.md

Usage

Boot tmp Kafka

$ docker run --rm -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 spotify/kafka

Create topic