This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
class MySQLStorePipeline(object): | |
@classmethod | |
def from_settings(cls, settings): | |
return cls(settings.get('DB_SETTING')) | |
def __init__(self, db_settings): | |
db.init_session(db_settings) | |
def process_item(self, item, spider): |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spider = FollowAllSpider(domain='scrapinghub.com') | |
settings = get_project_settings() | |
crawler = Crawler(settings) | |
crawler.signals.connect(reactor.stop, signal=signals.spider_closed) | |
crawler.configure() | |
crawler.crawl(spider) | |
crawler.start() | |
log.start() | |
reactor.run() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
function isNumber(str) { | |
return !isNaN(+str) | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aaa = {} | |
isinstance(aaa, dict) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dir_names.sort(cmp=lambda x,y: cmp(os.path.getctime('%s/%s' % (base_dir, x)), os.path.getctime('%s/%s' % (base_dir, y))) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
list.sort(cmp=lambda x,y: cmp(int(x), int(y))) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
list.sort( cmp=lambda x, y: cmp(x, y), reverse=True ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
re.sub(u'([ぁ-ゞァ-ヾ一-龠0−9!?]+)', lambda m: urllib2.quote(m.group(0).encode('utf8')), k) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def convert_datetime_to_unixtime(d) | |
time.mktime(d.timetuple()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for k, v in sorted(d.items(), key=lambda x:x[1]): | |
print k, v |
NewerOlder