testkernel: automate many linux kernel testing steps - great for a git bisect

testkernel.py

#!/usr/bin/env python
"""
The testkernel command does these things to make linux kernel testing easy:
  looks in the indicated folder on the web
  downloads the .deb files for the kernel of the given type
  installs them locally
  configures grub2 to reboot to the given kernel on the next reboot

TODO:

Revisit matching rules for flavor and arch - use re.compile in findAll, clean up, test more 
Develop some automated tests for all the options
Try it in a virtual machine.

Bugs to fix:
 Sometimes grub-reboot doesn't figure out the right name for the kernel,
   currently just assumes it is in sub-menu, and thus needs leading "2>"
   How to figure out what name of grub menu item is - e.g. need for leading "2>"?
 Pulling kernels from huge archive pool fails, seemingly in beautifulsoup
   e.g. on http://us.archive.ubuntu.com/ubuntu/pool/main/l/linux/
  Perhaps FTP support would be better
   ftp://mirror.anl.gov/pub/ubuntu/pool/main/l/linux/
   via ftputil? http://ftputil.sschwarzer.net/trac/wiki/Documentation#iteration-over-directories
  Really want a new protocol-agnostic "walk" function
   see http://stackoverflow.com/questions/9435529/unified-directory-tree-walking-for-local-files-ftp-http-in-python
 Sub-directory name when using -s should probably include flavor and arch to allow trying different builds on one machine
   Workaround: drop -s, or use a different directory.

Features to implement:
 Adapt to practices in distros besides Ubuntu
 Default to use same architecture and flavor as current machine
 Allow for avoiding work already done.  Have options for each step: download, install, grub-reboot.
 Keep a permanent log somewhere
 Add option to build a new kernel
 Add option to automate bisecting/testing process
 Use xml parsing, not strings to get right kernel
 Improve command-line completion - e.g. list candidates for webfolder argument and others
 Make it easy to delete test kernels, e.g. mainline ones that are so recent they become default
 Catch and notify about install errors like "Error! Bad return status for module build on kernel...."
 Note kernel command line parameter "panic=30" to get kernel to reboot after panic
 Add option to try out latest upstream kernel built for Ubuntu
 Add option to list interesting kernels to try
 Provide upstart / systemd / init scripts to log successful boots and keep track of kernel test results
 Find a better way than running the whole program via sudo to avoid needing to wait for download before
  entering password to do sudo dpkg -i, but yet reduce attack surface by running all but necessary steps
  with less privilege.
 Compare to standard kernel test script ktest.pl in linux/tools/testing/ktest
 Compare to autotest.github.com samples/control.kbuild_and_tests
 Compare to Linux Test Project: http://ltp.sourceforge.net/documentation/how-to/ltp.php
 Compare to kexec: A system call that allows you to put another kernel into memory and reboot without going back to the BIOS, and if it fails, reboot back.
 Recover from 404 errors from urlretrieve somehow
 Document booting with console redirected to serial line for better panic monitoring

%InsertOptionParserUsage%

The example above would fetch, install and try out the kernel defined by these files:
 linux-image-3.2.5-030205-generic-pae_3.2.5-030205.201202061401_i386.deb
 linux-headers-3.2.5-030205-generic-pae_3.2.5-030205.201202061401_i386.deb
 linux-headers-3.2.5-030205_3.2.5-030205.201202061401_all.deb

You'll also need the python package BeautifulSoup:
 apt-get install python-beautifulsoup
"""

import sys
import os
import logging
import re
import urllib2
import urlparse
import urllib
from optparse import OptionParser
from BeautifulSoup import BeautifulSoup
import subprocess
import tempfile

# Try to support completion from the command-line for this command.
try:
    import optcomplete
except ImportError:
    pass

__author__ = "Neal McBurnett <http://neal.mcburnett.org/>"
__version__ = "0.2.0"
__date__ = "2012-02-08"
__copyright__ = "Copyright (c) 2012 Neal McBurnett"
__license__ = "GPL v3"

usage="""
 testkernel [options] webfolder
 e.g. testkernel -f generic-pae -a i386 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.5-precise/

You can get bash completion for the options by installing the
python-optcomplete package, and running
    complete -F _optcomplete testkernel

Note that in order for the grub-reboot command to select the configured
kernel on the next reboot, you need to configure GRUB_DEFAULT=saved.
See https://help.ubuntu.com/community/Grub2
"""
parser = OptionParser(usage=usage, prog="testkernel", version=__version__)

# See https://wiki.ubuntu.com/Kernel/Dev/Flavours
parser.add_option("-f", "--flavor",
  default="generic",
  help="The flavor of kernel you want, e.g. generic, generic-pae, virtual, ti-omap4")

parser.add_option("-a", "--arch",
  default="amd64",
  help="The architecture for the kernel you want, e.g. i386, amd64, armel, powerpc")

opt = parser.add_option("-s", "--save_directory",
  default=None,   # make temporary directory later if not specified
  help="The directory in which you want to archive subdirectories for each of your sets of kernel packages.  By default, a temporary directory is created")
if 'optcomplete' in sys.modules:
    opt.completer = optcomplete.DirCompleter()

parser.add_option("-d", "--debuglevel",
  type="int", default=logging.WARNING,
  help="Set logging level to debuglevel: DEBUG=10, INFO=20,\n WARNING=30 (the default), ERROR=40, CRITICAL=50")

parser.add_option("-n", "--noexecute",
  action="store_true", default=False,
  help="Test: don't actually download the big files or run the commands, just look at the webfolder")

# incorporate OptionParser usage documentation in our docstring
__doc__ = __doc__.replace("%InsertOptionParserUsage%\n", parser.format_help())

kernellists = ['http://kernel.ubuntu.com/~kernel-ppa/mainline/']

if 'optcomplete' in sys.modules:
    optcomplete.autocomplete(parser, optcomplete.ListCompleter(kernellists), optcomplete.NoneCompleter())

# FIXME: use this or get rid of it
linux_deb_re = re.compile("linux(?P<attribs>.*?).deb")

def main(parser):
    "Run testkernel with given OptionParser arguments"

    (options, args) = parser.parse_args()

    #configure the root logger.  Without filename, default is StreamHandler with output to stderr. Default level is WARNING
    logging.basicConfig(level=options.debuglevel)   # ..., format='%(message)s', filename= "/file/to/log/to", filemode='w' )

    logging.debug("options: %s; args: %s", options, args)

    if len(args) != 1:
        print "Number of arguments should be 1 (the web folder to look in), not %d" % len(args)
        parser.print_help()
        sys.exit(-1)

    webfolder = args[0]

    visit(webfolder, options)

def visit(webfolder, options):
    "Visit a webfolder and look for relevant files (or perhaps subfolders)"

    image_re = re.compile("linux-image-(?P<abi>[^_]*?)-%s_(?P<upload>.*?)_%s.deb" % (options.flavor, options.arch))

    try:
        stream = urllib2.urlopen(webfolder)
    except urllib2.HTTPError, e:
        logging.error("Error on url '%s':\n %s" % (webfolder, e))
        sys.exit(1)

    # Get canonical name for folder (e.g. to make sure it ends with a slash, for urljoin)
    webfolder = stream.geturl()

    soup = BeautifulSoup(stream)
    anchors = soup.findAll('a', href=True)

    links = [a['href'] for a in anchors]
    logging.debug("all links:\n %s" % '\n '.join(links))

    for tag in anchors:
        # recurse to any sub-paths that start with a 'v'
        if tag['href'].startswith('v'):
            # FIXME: Disable until I can think some more about this and test some more
            # visit(urlparse.urljoin(webfolder, tag['href']), options)
            pass

    images = [l for l in links if image_re.match(l)]
    if len(images) > 1:
        logging.error("Too many matching images:\n %s" % "\n ".join(images))
        sys.exit(2)
    if len(images) < 1:
        logging.error("no matching linux-image present.\n %s\n Pattern: %s\n Links are:\n %s" % (webfolder, image_re.pattern, "\n ".join(links)))
        sys.exit(1)

    image = images[0]
    match = image_re.match(image)
    abi = match.group('abi')
    upload = match.group('upload')

    headers_all = [l for l in links if l.startswith('linux-headers') and l.endswith('_all.deb')]
    if len(headers_all) != 1:
        logging.error("Can't find linux-headers...._all.deb package")
        sys.exit(3)

    headers = [l for l in links if l.startswith('linux-headers') and l.endswith('_%s.deb' % options.arch) and options.flavor in l]

    if len(headers) != 1:
        logging.error("Can't find linux-headers package matching flavor %s, architecture %s" % (options.flavor, options.arch))
        sys.exit(3)

    if options.save_directory:
        kerneldir = os.path.join(options.save_directory, upload)
        if not options.noexecute:
            try:
                os.mkdir(kerneldir)
            except urllib2.HTTPError, e:
                logging.error("Error on url '%s':\n %s" % (webfolder, e))
                sys.exit(1)
    else:
        kerneldir = tempfile.mkdtemp()

    if not options.noexecute:
        os.chdir(kerneldir)

    debs = [image, headers[0], headers_all[0]]

    print "Retrieve these packages from %s" % webfolder

    for url in debs:
        url = urlparse.urljoin(webfolder, url)
        path = urlparse.urlsplit(url)[2]
        filename = os.path.basename(path)

        print " %s" % (filename)
        if not options.noexecute:
            urllib.urlretrieve(url, kerneldir + "/" + filename)

    if options.noexecute:
        explanation = "Would run: "
    else:
        explanation = "Running: "

    # Look in dpkg output for "replace", "Generating" output, return code
    commandargs = ["sudo", "dpkg", "-i"] + debs

    print explanation, " ".join(commandargs)
    if not options.noexecute:
        returncode = subprocess.call(commandargs)
        if returncode != 0:
            logging.error("return code from dpkg is %d" % returncode)
            sys.exit(returncode)

    commandargs = ["sudo", "grub-reboot", "'2>Ubuntu, with Linux %s-%s'" % (abi, options.flavor)]
    print explanation, " ".join(commandargs)

    if not options.noexecute:
        returncode = subprocess.call(commandargs)

    print "Files are in directory %s" % kerneldir

if __name__ == "__main__":
    main(parser)