Skip to content

Instantly share code, notes, and snippets.

Do the right thing

Mour mylamour

Do the right thing
View GitHub Profile
mylamour /
Created Dec 31, 2016
from musicbrainz Url Schema to extract some useful infomation and export csv , with parse to json. there is a amusing way to decide source due to the url type was various.
import csv
import json
import os
from urlparse import urlparse
csvfile = open('url.csv', 'r')
jsonfile = open('test.json', 'w')
fieldnames = ("@id","sourceUrl")
reader = csv.DictReader( csvfile, fieldnames)
mylamour /
Created Jan 1, 2017
use luigi to Backup Postgresql, there has a problem, it's not flexible. but i still think that's workflow should be
import luigi
import psycopg2
class QueryBackToTmp(luigi.Task):
def run(self):
conn_string = "host='' " \
"dbname='musicbrainz' " \
"user='postgres' " \
mylamour /
Last active Jan 2, 2017
use crontab to define a event and exec every second , it's simply like a log count
  • use crontab -e to define a event every minuter,due to the script was exahust 1 min , so it's mean this log can be ouput every second
  • and you can see what you want , just change the , you can easyliy get file info , and something else
  1. just insert this string : */1 * * * * /bin/bash ~/ >> cat.log
  2. just make a test, in your dest dir : wget -m -p -c
  3. log view (In a new terminal window) : tail -f cat.log
mylamour /
Created Jan 2, 2017
use crontab to define a event and exec every second , it's simply like a log count
for (( i = 0; i < 60; i=(i+step) )); do
# ls ~/tmp/ | wc -l
# ls ~/tmp/ | grep -E "html\."
ls -R ~/jav/ | wc -l
sleep $step
mylamour /
Last active Jan 3, 2017
progress bar with shell .
sleep 0.1
echo $i | dialog --title 'Copy' --gauge 'Backp file from postgresql!' 10 70 0
mylamour /
Last active Jan 12, 2017
Different Way to Open a HTTP SERVER (temp or not )


  • python2 -m SimpleHTTPServer
  • python3 -m http.server
  • twistd -n web -p 8000 --path .


  • ruby -rwebrick -e " => 8888, :DocumentRoot => Dir.pwd).start"
  • ruby -run -ehttpd . -p8000
mylamour / uncompelete regex
Last active Jan 18, 2017
Regex USEFUL Regex FROM : , different programe language was uncompelete support the regex features, you should take care about it.
View uncompelete regex
  • name="UniProt+Fastaheader"
`/^>[^\|]*\|([^\|]*)\|.*OS=([^=]*).*GN=([^ ]*).*$/g`

Matches UniProt accessionnumber, genename and organism in a UniProt fasta header

  • name="E-mail+validator+for+International+Domain"
mylamour /
Last active Jan 28, 2017
conert single line json with unicode to normal, 解决读取人名乱码问题
# s.decode("UTF-8").encode("GBK") 之前爬虫时可以用,但是现在这个场景不适合。
import json
with open('/home/ubuntu/origin/musicgroup.json') as origin, open('/home/ubuntu/json/musicgroup.json','w') as dest:
for i in origin.readlines():
t = json.loads(i)
dest.write(json.dumps(t, ensure_ascii=False).encode('utf8') + '\n')
import datetime
from time import mktime
import simplejson as json
except ImportError:
import json
class DateTimeEncoder(json.JSONEncoder): # 为 JSONEncoder 进行扩展
def default(self, obj):
if isinstance(obj, datetime.datetime):
mylamour /
Last active Feb 16, 2017
#!/usr/bin/env python
import requests
import cPickle as pickle
s = requests.Session()
with open('./iplist.list') as proxylists,open("needproxy.pkl","a") as usefull:
for proxy in proxylists.readlines():
tmp = {
"http": "http://{}:8080".format(proxy),