Skip to content

Instantly share code, notes, and snippets.

@amn41
Last active May 28, 2020 13:33
Show Gist options
  • Save amn41/de555c93913a01fbd56df2e2d211862c to your computer and use it in GitHub Desktop.
Save amn41/de555c93913a01fbd56df2e2d211862c to your computer and use it in GitHub Desktop.
Github Actions Workflow to comment on a Rasa repo PR with cross-validation results

Comment on a GitHub PR with Rasa NLU cross-validation results

image

Instructions

Save the yaml file at .github/workflows/comment_crossval_results.yml

Add the format_results.py script at the root of your repo. Your repo should contain a Rasa project following the usual file structure created by rasa init.

Ensure GitHub Actions are enabled for your repo.

When you open a PR to your repo you should get a comment showing cross-validation results. This is helpful if you have added new training data and want to check performance hasn't dropped.

name: Build and Deploy
on: [push]
jobs:
build-model:
name: Build and upload model
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install rasa
pip install pytablewriter==0.46.1
- name: Cross-validate NLU model
run: |
rasa test nlu -f 5 --cross-validation
python format_results.py
- name: post cross-val results to PR
uses: amn41/comment-on-pr@comment-file-contents
if: github.event_name == 'pull_request'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
msg: results.md
from pytablewriter import MarkdownTableWriter
import json
writer = MarkdownTableWriter()
writer.table_name = "Intent Cross-Validation Results (5 folds)"
with open('results/intent_report.json', 'r') as f:
data = json.loads(f.read())
writer.headers = ["class"] + list(data['micro avg'].keys())
classes = list(data.keys())
classes.sort(key = lambda x: data[x]['support'], reverse=True)
writer.value_matrix = [
[c] + [data[c][k] for k in data[c].keys()]
for c in classes
]
writer.dump('results.md')
@btotharye
Copy link

I think another thing that might be cool to add would be if there are confused with options, we show the intent_errors.json info to show which phrases/intents maybe the confusion is with and their confidence so you know from the PR what is causing it in case you aren't running this locally. I'll see if I can find time to help on that.

@amn41
Copy link
Author

amn41 commented Feb 12, 2020

yes that's a great idea for an enhancement

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment