Skip to content

Instantly share code, notes, and snippets.

This document has moved!

It's now here, in The Programmer's Compendium. The content is the same as before, but being part of the compendium means that it's actively maintained.

@kmike
kmike / robots_mw.py
Last active February 21, 2017 10:47
RobotsCrawlDelayMiddleware
# -*- coding: utf-8 -*-
from __future__ import absolute_import
import logging
import urlparse
from reppy.parser import Rules
from scrapy import log
from scrapy.exceptions import NotConfigured
from scrapy.http import Request
@audreyfeldroy
audreyfeldroy / pypi-release-checklist.md
Last active February 23, 2023 15:03
My PyPI Release Checklist
  • Update HISTORY.md
  • Commit the changes:
git add HISTORY.md
git commit -m "Changelog for upcoming release 0.1.1."
  • Update version number (can also be minor or major)
bumpversion patch