Github Actions Workflow to comment on a Rasa repo PR with cross-validation results

README.md

Comment on a GitHub PR with Rasa NLU cross-validation results

Instructions

Save the yaml file at .github/workflows/comment_crossval_results.yml

Add the format_results.py script at the root of your repo. Your repo should contain a Rasa project following the usual file structure created by rasa init.

Ensure GitHub Actions are enabled for your repo.

When you open a PR to your repo you should get a comment showing cross-validation results. This is helpful if you have added new training data and want to check performance hasn't dropped.

comment_crossval_results.yml

name: Build and Deploy
on: [push]

jobs:
  build-model:
    name: Build and upload model
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install rasa
        pip install pytablewriter==0.46.1
    - name: Cross-validate NLU model
      run: |
        rasa test nlu -f 5 --cross-validation
        python format_results.py
    - name: post cross-val results to PR
      uses: amn41/comment-on-pr@comment-file-contents
      if: github.event_name == 'pull_request'
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        msg: results.md

format_results.py

from pytablewriter import MarkdownTableWriter
import json

writer = MarkdownTableWriter()
writer.table_name = "Intent Cross-Validation Results (5 folds)"

with open('results/intent_report.json', 'r') as f:
    data = json.loads(f.read())
writer.headers = ["class"] + list(data['micro avg'].keys())

classes = list(data.keys())
classes.sort(key = lambda x: data[x]['support'], reverse=True)

writer.value_matrix = [
    [c] + [data[c][k] for k in data[c].keys()]
    for c in classes
]

writer.dump('results.md')

I think another thing that might be cool to add would be if there are confused with options, we show the intent_errors.json info to show which phrases/intents maybe the confusion is with and their confidence so you know from the PR what is causing it in case you aren't running this locally. I'll see if I can find time to help on that.

yes that's a great idea for an enhancement