Skip to content

Instantly share code, notes, and snippets.

@ilmarikyl
ilmarikyl / finetune_mT5_hlqg.py
Last active September 1, 2022 18:52
Script for fine-tuning a multilingual T5 model (mT5-base) for Finnish QG
import torch, os, json, argparse, sys
import numpy as np
import pandas as pd
from collections import Counter, defaultdict
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler
from transformers import MT5Tokenizer, MT5ForConditionalGeneration
from rich.table import Column, Table
from rich import box
from rich.console import Console