Skip to content

Instantly share code, notes, and snippets.

View ayonliu's full-sized avatar

Lyon Liu ayonliu

View GitHub Profile
@ayonliu
ayonliu / middlewares.py
Created October 27, 2015 03:52 — forked from pkmishra/middlewares.py
Scrapy middlewares for random agent list and proxy server usage.
import os
import random
from scrapy.conf import settings
class RandomUserAgentMiddleware(object):
def process_request(self, request, spider):
ua = random.choice(settings.get('USER_AGENT_LIST'))
if ua:
request.headers.setdefault('User-Agent', ua)
class ProxyMiddleware(object):
@ayonliu
ayonliu / gist:28f0742b500e2669016a
Created October 26, 2015 12:18 — forked from hzopak/gist:9573928
nginx config for scrapyd deployment to implement basic auth protection
# Scrapyd local proxy for basic authentication.
# Don't forget iptables rule.
# iptables -A INPUT -p tcp --destination-port 6800 -s ! 127.0.0.1 -j DROP
server {
listen 6801;
location ~ /\.ht {
deny all;
}
@ayonliu
ayonliu / slide.md
Last active August 29, 2015 14:04
slide show
@ayonliu
ayonliu / Mongo LBS.md
Created July 28, 2014 02:57
深入浅出Symfony2 - 结合MongoDB开发LBS应用
@ayonliu
ayonliu / add a git repo on server.md
Last active August 29, 2015 14:04
4.4 服务器上的 Git - 架设服务器
@ayonliu
ayonliu / animation.css
Created July 10, 2014 03:32
背景图片从左到右滚动效果
#header {
width: 100%;
height: 764px;
font-family: 'proxima_nova_rgregular', 'Proxima Nova', 'Helvetica Neue', Helvetica, sans-serif;
color: #fff;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
background: #000 url(../images/bg-new.jpg);
background-size: 100%;
margin: 0 auto;
@ayonliu
ayonliu / MongoDB Geo.md
Created July 9, 2014 10:08
Badvalue $geometry is required for geo near query

原因:需要为坐标建立索引, 坐标值要为 double 型。

db.places.ensureIndex({ "location": "2d" } )
'location'   => ['lng'=>42.241,'lat'=>55.5],
@ayonliu
ayonliu / Update to PHP 5.4 from Remi repo on CentOS 6.5.md
Created July 8, 2014 10:14
Update to PHP 5.4 from Remi repo on CentOS 6.5