generate changelogs for cdda

cddachangelog.py

import requests
from datetime import datetime
import urllib
from concurrent.futures.thread import ThreadPoolExecutor 
from concurrent.futures import as_completed

# If you don't have a github api key, you are likely to be rate limited.
# get one here https://github.com/settings/tokens
# should look like this ghp_cCx5F6xTSn07hbSRxZW2pbFsNFyiQPCx5K19

API_KEY =  'INSERT_KEY_HERE'
REPO_API = 'https://api.github.com/repos/CleverRaven/Cataclysm-DDA/'
CHANGELOG_FILE = 'cddachangelog.md'

def github_fetch(path,parms={}):
    header = {'Authorization': f'token {API_KEY}'}
    querystring = urllib.parse.urlencode(parms)
    r = requests.get(f"{REPO_API}{path}?{querystring}",headers=header)
    return r.json()

def get_releases(page=1):
    # print('Fetching ...')
    return github_fetch('releases',{'page':page,'per_page':100})

def match_commit_to_pr(hash):
    json = github_fetch(f'commits/{hash}/pulls')
    if len(json) == 0: # likely being rate limited
        print('Maybe you are being rate limited?')
        raise Exception()
    return json[0]

def get_cat(body):
    # I tried using regular expression, but it kept throwing exceptions.
    VALID_SUMMARY_CATEGORIES = (
        'content',
        'features',
        'interface',
        'mods',
        'balance',
        'i18n',
        'bugfixes',
        'performance',
        'build',
        'infrastructure',
        'none',
    )
    body = body.lower().replace('\r\n',' ').replace("\n"," ").replace(':','')
    s = [x for x in body.split(' ') if x != '']
    for a,b in zip(s,s[1:]):
        if a == 'summary':
            if b in VALID_SUMMARY_CATEGORIES:
                return b
    return 'unable to determine category'

def get_commits_in_time_range(starting_date,ending_date):
    current_timestamp = ending_date
    page_num = 1

    commits_hashes = []

    while current_timestamp > starting_date:
        releases = get_releases(page_num)
        for release in releases:
            commits_hashes.append(release['target_commitish'])
            current_timestamp = datetime.fromisoformat(release['published_at'][:-1])
            if current_timestamp < starting_date:
                break
        page_num+=1
    
    return commits_hashes


def generate_changelogs(starting_date,ending_date=None):
    if ending_date == None:
        ending_date = datetime.today()

    commits_hashes = get_commits_in_time_range(starting_date,ending_date)
    print(f"{len(commits_hashes)} found")

    d = dict()

    with ThreadPoolExecutor() as executor: # multi threading
        res = [executor.submit(match_commit_to_pr, hash) for hash in commits_hashes]
        for future in as_completed(res):
            info = future.result()
            author = info['user']
            txt_line = f"[{info['title']}]({info['html_url']}) by [{author['login']}]({author['html_url']})"
            cat = get_cat(info['body'])

            if cat not in d:
                d[cat] = []
            d[cat].append(txt_line)

    with open(CHANGELOG_FILE,'w+') as f:
        f.write(f'Logs generated from {starting_date} til {ending_date}\n')
        for k in d:
            f.write(f'# {k[0].upper()}{k[1:]}\n')
            for i in d[k]:
                f.write(i+'\n\n')


if __name__ =='__main__':
    starting_time = '2021-10-20'
    generate_changelogs(datetime.fromisoformat(starting_time))

First off: Thanks for this! It's always exciting to see more folks with a desire to pitch in!

For this particular implementation, there's some stuff we'll also need to account for if the goal is to have this code see regular usage:

1. There's a desire to generate these in an automated fashion, which means figuring out how to execute stuff like this via Github Actions as part of a build/release. Working in that environment, there's a few things you'd need to change to make this script compatible:
   • API_KEY would be present in an environment variable, so you could query for that or pass it in as an argument rather than having it hard coded
   • CHANGELOG_FILE would need to be written as a build/release artifact or posted to the relevant Github Release via the API. You might also be able to get the markdown file bundled in with the build artifacts so it ends up part of the ZIP file folks download when grabbing a new version of the game.
   • starting_time likewise would need to become an argument. It might also be worth changing starting_time to instead be a reference to a Git ref as a starting point to avoid timezone issues and other ambiguity about what changes to include.
2. It'd be great if you could call out what version of Python you built this against and what version of the requests package you're using. A common issue I'm fighting right now is that we don't have much consistency in what Python and package versions folks are building against. If you can call out what you're using I can make efforts to ensure your code continues to work once everything is integrated.
3. Consider writing script output to STDOUT rather than a file, and log messages to STDERR. This'll make the script a lot more portable between local dev and CI environments, as well as allow easy integration into build scripts. Offloading the writing of the text in STDOUT to the caller of the script means you also don't need to care about what the state of the local filesystem is.

Hope this is welcome feedback, feel free to reach out to me on the CDDA Discord if you'd like to collaborate more on stuff like this! (Damien)