public
Last active

Conversion from Launchpad bugs to GitHub ones

  • Download Gist
lp2gh-issues.py
Python
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369
#!/usr/bin/env python
"""Launchpad to github bug migration script.
 
There's a ton of code from Hydrazine copied here:
https://launchpad.net/hydrazine
 
WARNING: this code was written for the Github issues v2 API, and has *not* been ported to v3. If anyone finds it useful and ports it, please drop me a pull request.
 
Usage
-----
 
This code is meant to port a bug database for a project from Launchpad to
GitHub. It was used to port the IPython bug history.
 
The code is meant to be used interactively. I ran it multiple times in one long
IPython session, until the data structures I was getting from Launchpad looked
right. Then I turned off (see 'if 0' markers below) the Launchpad part, and ran
it again with the github part executing and using the 'bugs' variable from my
interactive namespace (via"%run -i" in IPython).
 
This code is NOT fire and forget, it's meant to be used with some intelligent
supervision at the wheel. Start by making a test repository (I made one called
ipython/BugsTest) and upload only a few issues into that. Once you are sure
that everything is OK, run it against your real repo with all your issues.
 
You should read all the code below and roughly understand what's going on
before using this. Since I didn't intend to use this more than once, it's not
particularly robust or documented. It got the job done and I've never used it
again.
 
Configuration
-------------
 
To pull things off LP, you need to log in first (see the Hydrazine docs). Your
Hydrazine credentials will be cached locally and this script can reuse them.
 
To push to GH, you need to set below the GH repository owner, API token and
repository name you wan to push issues into. See the GH section for the
necessary variables.
"""
 
import collections
import os.path
import subprocess
import sys
import time
 
from pprint import pformat
 
import launchpadlib
from launchpadlib.credentials import Credentials
from launchpadlib.launchpad import (
Launchpad, STAGING_SERVICE_ROOT, EDGE_SERVICE_ROOT )
 
#-----------------------------------------------------------------------------
# Launchpad configuration
#-----------------------------------------------------------------------------
# The official LP project name
PROJECT_NAME = 'ipython'
 
# How LP marks your bugs, I don't know where this is stored, but they use it to
# generate bug descriptions and we need to split on this string to create
# shorter Github bug titles
PROJECT_ID = 'IPython'
 
# Default Launchpad server, see their docs for details
service_root = EDGE_SERVICE_ROOT
 
#-----------------------------------------------------------------------------
# Code copied/modified from Hydrazine (https://launchpad.net/hydrazine)
#-----------------------------------------------------------------------------
 
# Constants for the names in LP of certain
lp_importances = ['Critical', 'High', 'Medium', 'Low', 'Wishlist', 'Undecided']
 
lp_status = ['Confirmed', 'Triaged', 'Fix Committed', 'Fix Released',
'In Progress',"Won't Fix", "Incomplete", "Invalid", "New"]
 
def squish(a):
return a.lower().replace(' ', '_').replace("'",'')
 
lp_importances_c = set(map(squish, lp_importances))
lp_status_c = set(map(squish, lp_status))
 
def trace(s):
sys.stderr.write(s + '\n')
 
 
def create_session():
lplib_cachedir = os.path.expanduser("~/.cache/launchpadlib/")
hydrazine_cachedir = os.path.expanduser("~/.cache/hydrazine/")
rrd_dir = os.path.expanduser("~/.cache/hydrazine/rrd")
for d in [lplib_cachedir, hydrazine_cachedir, rrd_dir]:
if not os.path.isdir(d):
os.makedirs(d, mode=0700)
 
hydrazine_credentials_filename = os.path.join(hydrazine_cachedir,
'credentials')
if os.path.exists(hydrazine_credentials_filename):
credentials = Credentials()
credentials.load(file(
os.path.expanduser("~/.cache/hydrazine/credentials"),
"r"))
trace('loaded existing credentials')
return Launchpad(credentials, service_root,
lplib_cachedir)
# TODO: handle the case of having credentials that have expired etc
else:
launchpad = Launchpad.get_token_and_login(
'Hydrazine',
service_root,
lplib_cachedir)
trace('saving credentials...')
launchpad.credentials.save(file(
hydrazine_credentials_filename,
"w"))
return launchpad
 
def canonical_enum(entered, options):
entered = squish(entered)
return entered if entered in options else None
 
def canonical_importance(from_importance):
return canonical_enum(from_importance, lp_importances_c)
 
def canonical_status(entered):
return canonical_enum(entered, lp_status_c)
 
#-----------------------------------------------------------------------------
# Functions and classes
#-----------------------------------------------------------------------------
 
class Base(object):
def __str__(self):
a = dict([(k,v) for (k,v) in self.__dict__.iteritems()
if not k.startswith('_')])
return pformat(a)
 
__repr__ = __str__
 
 
class Message(Base):
def __init__(self, m):
self.content = m.content
o = m.owner
self.owner = o.name
self.owner_name = o.display_name
self.date = m.date_created
 
class Bug(Base):
def __init__(self, bt):
# Cache a few things for which launchpad will make a web request each
# time.
bug = bt.bug
o = bt.owner
a = bt.assignee
dupe = bug.duplicate_of
# Store from the launchpadlib bug objects only what we want, and as
# local data
self.id = bug.id
self.lp_url = 'https://bugs.launchpad.net/%s/+bug/%i' % \
(PROJECT_NAME, self.id)
self.title = bt.title
self.description = bug.description
# Every bug has an owner (who created it)
self.owner = o.name
self.owner_name = o.display_name
# Not all bugs have been assigned to someone yet
try:
self.assignee = a.name
self.assignee_name = a.display_name
except AttributeError:
self.assignee = self.assignee_name = None
# Store status/importance in canonical format
self.status = canonical_status(bt.status)
self.importance = canonical_importance(bt.importance)
self.tags = bug.tags
# Store the bug discussion messages, but skip m[0], which is the same
# as the bug description we already stored
self.messages = map(Message, list(bug.messages)[1:])
self.milestone = getattr(bt.milestone, 'name', None)
 
# Duplicate handling disabled, since the default query already filters
# out the duplicates. Keep the code here in case we ever want to look
# into this...
if 0:
# Track duplicates conveniently
try:
self.duplicate_of = dupe.id
self.is_duplicate = True
except AttributeError:
self.duplicate_of = None
self.is_duplicate = False
 
# dbg dupe info
if bug.number_of_duplicates > 0:
self.duplicates = [b.id for b in bug.duplicates]
else:
self.duplicates = []
 
# tmp - debug
self._bt = bt
self._bug = bug
 
#-----------------------------------------------------------------------------
# Main script
#-----------------------------------------------------------------------------
 
#-----------------------------------------------------------------------------
# Launchpad part
#-----------------------------------------------------------------------------
if 0:
launchpad = create_session()
project = launchpad.projects[PROJECT_NAME]
# Note: by default, this will give us all bugs except duplicates and those
# with status "won't fix" or 'invalid'
bug_tasks = project.searchTasks()
 
if 0:
bugs = {}
for bt in bug_tasks:
b = Bug(bt)
bugs[b.id] = b
print b.title
sys.stdout.flush()
 
#-----------------------------------------------------------------------------
# Github part
#-----------------------------------------------------------------------------
#http://pypi.python.org/pypi/github2
#http://github.com/ask/python-github2
# Github libraries
from github2 import core, issues, client
#for mod in (core, issues, client):
# reload(mod)
 
 
def format_title(bug):
return bug.title.split('{0}: '.format(PROJECT_ID), 1)[1].strip('"')
 
 
def format_body(bug):
body = \
"""Original Launchpad bug {bug.id}: {bug.lp_url}
Reported by: {bug.owner} ({owner_name}).
 
{description}""".format(bug=bug, owner_name=bug.owner_name.encode('utf-8'),
description=bug.description.encode('utf-8'))
return body
 
 
def format_message(num, m):
body = \
"""[ LP comment {num} by: {owner_name}, on {m.date!s} ]
 
{content}""".format(num=num, m=m, owner_name=m.owner_name.encode('utf-8'),
content=m.content.encode('utf-8'))
return body
 
 
# Config
user = 'ipython'
token= 'PUT YOUR GITHUB TOKEN HERE'
 
repo = 'ipython/BugsTest'
#repo = 'ipython/ipython'
 
# Skip bugs with this status:
to_skip = set([u'fix_committed', u'incomplete'])
 
# Only label these importance levels:
gh_importances = set([u'critical', u'high', u'low', u'medium', u'wishlist'])
 
# Start script
gh = client.Github(username=user, api_token=token)
 
# Filter out the full LP bug dict to process only the ones we want
bugs_todo = dict( (id, b) for (id, b) in bugs.iteritems()
if not b.status in to_skip )
 
# Select which bug ids to run
#bids = bugs_todo.keys()[50:100]
bids = bugs_todo.keys()[12:]
 
bids = bugs_todo.keys()
#bids = bids[:5]+[502787]
 
# Start loop over bug ids and file them on Github
nbugs = len(bids)
gh_issues = [] # for reporting at the end
for n,bug_id in enumerate(bids):
bug = bugs[bug_id]
title = format_title(bug)
body = format_body(bug)
 
print
if len(title)<65:
print bug.id, '[{0}/{1}]'.format(n+1, nbugs), title
else:
print bug.id, title[:65]+'...'
# still check bug.status, in case we manually added other bugs to the list
# above (mostly during testing)
if bug.status in to_skip:
print '--- Skipping - status:',bug.status
continue
print '+++ Filing...',
sys.stdout.flush()
# Create github issue for this bug
issue = gh.issues.open(repo, title=title, body=body)
print 'created GitHub #', issue.number
gh_issues.append(issue.number)
sys.stdout.flush()
# Mark status as a label
#status = 'status-{0}'.format(b.status)
#gh.issues.add_label(repo, issue.number, status)
 
# Mark any extra tags we might have as labels
for tag in b.tags:
label = 'tag-{0}'.format(tag)
gh.issues.add_label(repo, issue.number, label)
 
# If bug has assignee, add it as label
if bug.assignee:
gh.issues.add_label(repo, issue.number,
#bug.assignee
# Github bug, gets confused with dots in labels.
bug.assignee.replace('.','_')
)
 
if bug.importance in gh_importances:
if bug.importance == 'wishlist':
label = bug.importance
else:
label = 'prio-{0}'.format(bug.importance)
gh.issues.add_label(repo, issue.number, label)
 
if bug.milestone:
label = 'milestone-{0}'.format(bug.milestone).replace('.','_')
gh.issues.add_label(repo, issue.number, label)
# Add original message thread
for num, message in enumerate(bug.messages):
# Messages on LP are numbered from 1
comment = format_message(num+1, message)
gh.issues.comment(repo, issue.number, comment)
time.sleep(0.5) # soft sleep after each message to prevent gh block
 
# too many fast requests and gh will block us, so sleep for a while
# I just eyeballed these values by trial and error.
time.sleep(1) # soft sleep after each request
# And longer one after every batch
batch_size = 10
tsleep = 60
if (len(gh_issues) % batch_size)==0:
print
print '*** SLEEPING for {0} seconds to avoid github blocking... ***'.format(tsleep)
sys.stdout.flush()
time.sleep(tsleep)
 
# Summary report
print
print '*'*80
print 'Summary of GitHub issues filed:'
print gh_issues
print 'Total:', len(gh_issues)

Hm, I guess we cant pull request on gists.

I did a little update for github3 here: https://gist.github.com/mleinart/5134236/revisions

Unfortunately I found this gist a bit too late as I'd already started on my own very similar tool: https://github.com/mleinart/launchpad2github/blob/master/launchpad2github.py

This was quite helpful to see though, thanks for posting!

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.