Python implementation of the Nadeau and Bengio correction of dependent Student's t-test

corrected_dependent_ttest.py

# Python implementation of the Nadeau and Bengio correction of dependent Student's t-test
# using the equation stated in https://www.cs.waikato.ac.nz/~eibe/pubs/bouckaert_and_frank.pdf

from scipy.stats import t
from math import sqrt
from statistics import stdev

def corrected_dependent_ttest(data1, data2, n_training_samples, n_test_samples, alpha):
    n = len(data1)
    differences = [(data1[i]-data2[i]) for i in range(n)]
    sd = stdev(differences)
    divisor = 1 / n * sum(differences)
    test_training_ratio = n_test_folds / n_training_folds  
    denominator = sqrt(1 / n + test_training_ratio) * sd
    t_stat = divisor / denominator
    # degrees of freedom
    df = n - 1
    # calculate the critical value
    cv = t.ppf(1.0 - alpha, df)
    # calculate the p-value
    p = (1.0 - t.cdf(abs(t_stat), df)) * 2.0
    # return everything
    return t_stat, df, cv, p

Hi! What do you intend n_training_folds and n_test_folds to be here? In the Bouckaert and Frank paper (line 2 in your gist) n_1 and n_2 are the number of examples in the training and test folds respectively. The naming here seems to suggest something different, as if some folds were only used in training and some others only in testing.

Thank you for sharing this, BTW!

Ah yes, you are totally correct! If your folds are of the same size, this should yield the same result, but they certainly don't have to!

Thanks for your answer! Just to be sure, do we agree that your n_training_folds is n_1 in the paper and n_test_folds is n_2?

@Pibborn, yes you are correct that n_training_folds is n_1 and n_test_folds is n_2. The author of the gist just made a typo in the variable names, "test_training_ratio = n_test_folds / n_training_folds " should be corrected to "test_training_ratio = n_test_samples / n_training_samples"