Skip to content

Instantly share code, notes, and snippets.

Avatar
🚴‍♂️

takuma-saito

🚴‍♂️
View GitHub Profile
View parallel-link-checker.sh
View gem-installer.sh
# client
gem lists > gem-lists.txt
# server
gem install -v $(cat gem-lists.txt | cut -d' ' -f1 | tr '\n' ' ')
View parallel-domain-checker.sh
#!/bin/bash
SERVER='8.8.8.8'
cat $1 | sed -E 's;^https?://(.*)/;\1;g' | xargs -I{} dig +short +time=5 @$SERVER {}
@takuma-saito
takuma-saito / change-ip-by-eip.sh
Last active Nov 16, 2015
change-ip-by-eip.sh
View change-ip-by-eip.sh
#!/bin/zsh
InstanceId="$1"
# allocation id of current ip
OldAllocationId=$(aws ec2 describe-addresses |
jq -r '.Addresses[] | select(.InstanceId == "'$InstanceId'") | .AllocationId')
# create address
NewAllocationId=$(aws ec2 allocate-address | jq -r '.AllocationId')
View google-crawler.rb
# coding: utf-8
require 'nokogiri'
require 'open-uri'
require 'uri'
instanceID = 'xxxxxx'
## if you are blocked by google, you have to change your ip address.
## You can change it with "elastic ip address" provided by ec2.
View create-instance-by-eip.sh
#!/bin/zsh
AMI="ami-xxxxxxx" # Official CentOS 7.0 AMI in ap-northeast-1 region
ALOCID="eipalloc-xxxxxxx" # Allocating ip address
SIZE="m3.large"
KEYNAME="jobfind"
SEC_GROUP="default"
InstanceId=$(aws ec2 run-instances \
--count 1 \
@takuma-saito
takuma-saito / gist:1b531593bfc99a245f91
Last active Mar 3, 2016
並列化クローリングのメモ.txt
View gist:1b531593bfc99a245f91
* GIL (Global Interepter Lock) が存在するために ruby, python ではマルチコアを生かせない(コア数に比例してクローリングの数が増えない)
⇒ 対策: マルチコアを生かすには GIL が存在しない java や C++ か、メッセージバッシングの行える Scala, Go, Closure を使う
⇒ 今回はあまり癖がなく Garbage Collector が付いている Go を選択する
### 20 (page/second) の壁
* ファイルディスクリプタの枯渇で死ぬ
⇒ 対策
* ulimit -u 40000
View dns-benchmark.sh
dnsperf -d domain-1000000.txt -s 127.0.0.1 -p 53 -c 128 -t 10 -q 5000 -Q 150000 -S 1 2>&1 | grep -v -E 'Timeout|Warning'
@takuma-saito
takuma-saito / google-public-dns-benchmark.sh
Last active Aug 10, 2016
benchmark of google public cache DNS server
View google-public-dns-benchmark.sh
curl 'http://s3.amazonaws.com/alexa-static/top-1m.csv.zip' | zcat -d | awk -F',' '{print $2}' | paste -d' ' - <(yes A | head -n 200000) > queries-200000.txt
dnsperf -d queries-10000.txt -s 8.8.8.8 -p 53 -c 128 -t 10 -q 1000 -Q 1500 -S 1 2>&1 | grep -v -E 'Timeout|Warning'
dnsperf -d queries-200000.tx -s 127.0.0.1 -p 53 -c 120000 -t 10 -q 20000 -Q 20000 -S 1
@takuma-saito
takuma-saito / tech.txt
Last active Oct 18, 2018
useful shell commands
View tech.txt
** メモ **
# コメント付きで keygen
$ ssh-keygen -C taku@hacker -t rsa -f ~/.ssh/github
# 通信している ARP アドレスの確認
$ arp -a
# ネットワークの統計
You can’t perform that action at this time.