Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
testkernel: automate many linux kernel testing steps - great for a git bisect
#!/usr/bin/env python
"""
The testkernel command does these things to make linux kernel testing easy:
looks in the indicated folder on the web
downloads the .deb files for the kernel of the given type
installs them locally
configures grub2 to reboot to the given kernel on the next reboot
TODO:
Revisit matching rules for flavor and arch - use re.compile in findAll, clean up, test more
Develop some automated tests for all the options
Try it in a virtual machine.
Bugs to fix:
Sometimes grub-reboot doesn't figure out the right name for the kernel,
currently just assumes it is in sub-menu, and thus needs leading "2>"
How to figure out what name of grub menu item is - e.g. need for leading "2>"?
Pulling kernels from huge archive pool fails, seemingly in beautifulsoup
e.g. on http://us.archive.ubuntu.com/ubuntu/pool/main/l/linux/
Perhaps FTP support would be better
ftp://mirror.anl.gov/pub/ubuntu/pool/main/l/linux/
via ftputil? http://ftputil.sschwarzer.net/trac/wiki/Documentation#iteration-over-directories
Really want a new protocol-agnostic "walk" function
see http://stackoverflow.com/questions/9435529/unified-directory-tree-walking-for-local-files-ftp-http-in-python
Sub-directory name when using -s should probably include flavor and arch to allow trying different builds on one machine
Workaround: drop -s, or use a different directory.
Features to implement:
Adapt to practices in distros besides Ubuntu
Default to use same architecture and flavor as current machine
Allow for avoiding work already done. Have options for each step: download, install, grub-reboot.
Keep a permanent log somewhere
Add option to build a new kernel
Add option to automate bisecting/testing process
Use xml parsing, not strings to get right kernel
Improve command-line completion - e.g. list candidates for webfolder argument and others
Make it easy to delete test kernels, e.g. mainline ones that are so recent they become default
Catch and notify about install errors like "Error! Bad return status for module build on kernel...."
Note kernel command line parameter "panic=30" to get kernel to reboot after panic
Add option to try out latest upstream kernel built for Ubuntu
Add option to list interesting kernels to try
Provide upstart / systemd / init scripts to log successful boots and keep track of kernel test results
Find a better way than running the whole program via sudo to avoid needing to wait for download before
entering password to do sudo dpkg -i, but yet reduce attack surface by running all but necessary steps
with less privilege.
Compare to standard kernel test script ktest.pl in linux/tools/testing/ktest
Compare to autotest.github.com samples/control.kbuild_and_tests
Compare to Linux Test Project: http://ltp.sourceforge.net/documentation/how-to/ltp.php
Compare to kexec: A system call that allows you to put another kernel into memory and reboot without going back to the BIOS, and if it fails, reboot back.
Recover from 404 errors from urlretrieve somehow
Document booting with console redirected to serial line for better panic monitoring
%InsertOptionParserUsage%
The example above would fetch, install and try out the kernel defined by these files:
linux-image-3.2.5-030205-generic-pae_3.2.5-030205.201202061401_i386.deb
linux-headers-3.2.5-030205-generic-pae_3.2.5-030205.201202061401_i386.deb
linux-headers-3.2.5-030205_3.2.5-030205.201202061401_all.deb
You'll also need the python package BeautifulSoup:
apt-get install python-beautifulsoup
"""
import sys
import os
import logging
import re
import urllib2
import urlparse
import urllib
from optparse import OptionParser
from BeautifulSoup import BeautifulSoup
import subprocess
import tempfile
# Try to support completion from the command-line for this command.
try:
import optcomplete
except ImportError:
pass
__author__ = "Neal McBurnett <http://neal.mcburnett.org/>"
__version__ = "0.2.0"
__date__ = "2012-02-08"
__copyright__ = "Copyright (c) 2012 Neal McBurnett"
__license__ = "GPL v3"
usage="""
testkernel [options] webfolder
e.g. testkernel -f generic-pae -a i386 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.5-precise/
You can get bash completion for the options by installing the
python-optcomplete package, and running
complete -F _optcomplete testkernel
Note that in order for the grub-reboot command to select the configured
kernel on the next reboot, you need to configure GRUB_DEFAULT=saved.
See https://help.ubuntu.com/community/Grub2
"""
parser = OptionParser(usage=usage, prog="testkernel", version=__version__)
# See https://wiki.ubuntu.com/Kernel/Dev/Flavours
parser.add_option("-f", "--flavor",
default="generic",
help="The flavor of kernel you want, e.g. generic, generic-pae, virtual, ti-omap4")
parser.add_option("-a", "--arch",
default="amd64",
help="The architecture for the kernel you want, e.g. i386, amd64, armel, powerpc")
opt = parser.add_option("-s", "--save_directory",
default=None, # make temporary directory later if not specified
help="The directory in which you want to archive subdirectories for each of your sets of kernel packages. By default, a temporary directory is created")
if 'optcomplete' in sys.modules:
opt.completer = optcomplete.DirCompleter()
parser.add_option("-d", "--debuglevel",
type="int", default=logging.WARNING,
help="Set logging level to debuglevel: DEBUG=10, INFO=20,\n WARNING=30 (the default), ERROR=40, CRITICAL=50")
parser.add_option("-n", "--noexecute",
action="store_true", default=False,
help="Test: don't actually download the big files or run the commands, just look at the webfolder")
# incorporate OptionParser usage documentation in our docstring
__doc__ = __doc__.replace("%InsertOptionParserUsage%\n", parser.format_help())
kernellists = ['http://kernel.ubuntu.com/~kernel-ppa/mainline/']
if 'optcomplete' in sys.modules:
optcomplete.autocomplete(parser, optcomplete.ListCompleter(kernellists), optcomplete.NoneCompleter())
# FIXME: use this or get rid of it
linux_deb_re = re.compile("linux(?P<attribs>.*?).deb")
def main(parser):
"Run testkernel with given OptionParser arguments"
(options, args) = parser.parse_args()
#configure the root logger. Without filename, default is StreamHandler with output to stderr. Default level is WARNING
logging.basicConfig(level=options.debuglevel) # ..., format='%(message)s', filename= "/file/to/log/to", filemode='w' )
logging.debug("options: %s; args: %s", options, args)
if len(args) != 1:
print "Number of arguments should be 1 (the web folder to look in), not %d" % len(args)
parser.print_help()
sys.exit(-1)
webfolder = args[0]
visit(webfolder, options)
def visit(webfolder, options):
"Visit a webfolder and look for relevant files (or perhaps subfolders)"
image_re = re.compile("linux-image-(?P<abi>[^_]*?)-%s_(?P<upload>.*?)_%s.deb" % (options.flavor, options.arch))
try:
stream = urllib2.urlopen(webfolder)
except urllib2.HTTPError, e:
logging.error("Error on url '%s':\n %s" % (webfolder, e))
sys.exit(1)
# Get canonical name for folder (e.g. to make sure it ends with a slash, for urljoin)
webfolder = stream.geturl()
soup = BeautifulSoup(stream)
anchors = soup.findAll('a', href=True)
links = [a['href'] for a in anchors]
logging.debug("all links:\n %s" % '\n '.join(links))
for tag in anchors:
# recurse to any sub-paths that start with a 'v'
if tag['href'].startswith('v'):
# FIXME: Disable until I can think some more about this and test some more
# visit(urlparse.urljoin(webfolder, tag['href']), options)
pass
images = [l for l in links if image_re.match(l)]
if len(images) > 1:
logging.error("Too many matching images:\n %s" % "\n ".join(images))
sys.exit(2)
if len(images) < 1:
logging.error("no matching linux-image present.\n %s\n Pattern: %s\n Links are:\n %s" % (webfolder, image_re.pattern, "\n ".join(links)))
sys.exit(1)
image = images[0]
match = image_re.match(image)
abi = match.group('abi')
upload = match.group('upload')
headers_all = [l for l in links if l.startswith('linux-headers') and l.endswith('_all.deb')]
if len(headers_all) != 1:
logging.error("Can't find linux-headers...._all.deb package")
sys.exit(3)
headers = [l for l in links if l.startswith('linux-headers') and l.endswith('_%s.deb' % options.arch) and options.flavor in l]
if len(headers) != 1:
logging.error("Can't find linux-headers package matching flavor %s, architecture %s" % (options.flavor, options.arch))
sys.exit(3)
if options.save_directory:
kerneldir = os.path.join(options.save_directory, upload)
if not options.noexecute:
try:
os.mkdir(kerneldir)
except urllib2.HTTPError, e:
logging.error("Error on url '%s':\n %s" % (webfolder, e))
sys.exit(1)
else:
kerneldir = tempfile.mkdtemp()
if not options.noexecute:
os.chdir(kerneldir)
debs = [image, headers[0], headers_all[0]]
print "Retrieve these packages from %s" % webfolder
for url in debs:
url = urlparse.urljoin(webfolder, url)
path = urlparse.urlsplit(url)[2]
filename = os.path.basename(path)
print " %s" % (filename)
if not options.noexecute:
urllib.urlretrieve(url, kerneldir + "/" + filename)
if options.noexecute:
explanation = "Would run: "
else:
explanation = "Running: "
# Look in dpkg output for "replace", "Generating" output, return code
commandargs = ["sudo", "dpkg", "-i"] + debs
print explanation, " ".join(commandargs)
if not options.noexecute:
returncode = subprocess.call(commandargs)
if returncode != 0:
logging.error("return code from dpkg is %d" % returncode)
sys.exit(returncode)
commandargs = ["sudo", "grub-reboot", "'2>Ubuntu, with Linux %s-%s'" % (abi, options.flavor)]
print explanation, " ".join(commandargs)
if not options.noexecute:
returncode = subprocess.call(commandargs)
print "Files are in directory %s" % kerneldir
if __name__ == "__main__":
main(parser)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.