eliasdorneles

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
For those times when you just want a button.

Run and notice the new icon in your notifications area.

Inspired by: https://glyph.twistedmatrix.com/2015/07/just-a-button.html
"""

eliasdorneles

#!/usr/bin/env python
# A toy Python BDD framework


# Decorator for test classes
def _context():
    def add_test_method(cls, method, name):
        def test(self, *args, **kwargs):
            return method(self, self.subject(), *args, **kwargs)
        test.func_name = 'test_' + name

eliasdorneles

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Reservoir sampling for line-based input
"""

from __future__ import print_function
import sys
import random

eliasdorneles

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Lisp parser (future interpreter)
"""

from __future__ import print_function
import sys
import re
from collections import namedtuple

eliasdorneles

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Cookiecutter PyPackage development tasks
"""
# To add a task, create a function with name starting with 'do_' that
# receives one argument (the parsed cmdline arguments object).
# Use a short docstring to serve as help.
#
# Example:
#

eliasdorneles

>>> js_with_html_comments = u'<!-- some... --> alert("oi"); <!-- ...comments -->'
>>> import js2xml
>>> from lxml import etree
>>> def strip_comments(js):
   ....:     node = etree.fromstring(u'<node>%s</node>' % js)
   ....:     etree.strip_tags(node, etree.Comment)
   ....:     return node.text
   ....: 
>>> strip_comments(js_with_html_comments)
' alert("oi"); '

eliasdorneles

Scrapy Release Procedure

This page outlines the procedure used to release a new (major/minor) version of Scrapy. Micro/patch versions are published automatically.

You will need bumpversion installed.

Version and git branch

Here is an example, assuming we are releasing 1.1.0rc1:

eliasdorneles

#!/bin/bash
# This script assumes you're using the directory layout described in the
# documentation at: https://voc.readthedocs.io/en/latest/tutorials/tutorial-0.html

set -e
set -x

abort() {
    echo "$*"; exit 1;
}

eliasdorneles

# How one could go about implementing a check for abstract Python classes
# written in the old style (raise NotImplementedError),
# by looking at the methods bytecode


def is_oldstyle_abstract(cls):
    def has_raise_notimplemented_code(func_code):
        # this is too magic, I know.
        # you can see how it works by disassembling the bytes below:
        #

eliasdorneles

Focus on scraping use cases

• Warn about /table/tbody/tr in the query, if no results (common mistake)
  • Consider using HTML5 parser instead
• Shorter method names
• Guess encoding in the case of bytes input
• Selector vs SelectorList <- could we have only one class, make it list-like? (like jQuery)
• Consider adding a shortcut to parse JSON (demjson, to support liberal input JS-like)
• Add a simple link extraction method