Analysis of duplicated data within bAbi Tasks v1.2

uniq_tasks_10k.txt

Unique samples in tasks_1-20_v1-2/en-10k/qa10_indefinite-knowledge_{}.txt
Train length: 9989
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa11_basic-coreference_{}.txt
Train length: 9827
Test length: 997
Intersection: 25
Unique samples in tasks_1-20_v1-2/en-10k/qa12_conjunction_{}.txt
Train length: 9991
Test length: 1000
Intersection: 2
Unique samples in tasks_1-20_v1-2/en-10k/qa13_compound-coreference_{}.txt
Train length: 9978
Test length: 1000
Intersection: 11
Unique samples in tasks_1-20_v1-2/en-10k/qa14_time-reasoning_{}.txt
Train length: 9987
Test length: 1000
Intersection: 1
Unique samples in tasks_1-20_v1-2/en-10k/qa15_basic-deduction_{}.txt
Train length: 6861
Test length: 678
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa16_basic-induction_{}.txt
Train length: 10000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa17_positional-reasoning_{}.txt
Train length: 5812
Test length: 632
Intersection: 97
Unique samples in tasks_1-20_v1-2/en-10k/qa18_size-reasoning_{}.txt
Train length: 6388
Test length: 602
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa19_path-finding_{}.txt
Train length: 10000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt
Train length: 9908
Test length: 999
Intersection: 16
Unique samples in tasks_1-20_v1-2/en-10k/qa20_agents-motivations_{}.txt
Train length: 8310
Test length: 906
Intersection: 110
Unique samples in tasks_1-20_v1-2/en-10k/qa2_two-supporting-facts_{}.txt
Train length: 9996
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa3_three-supporting-facts_{}.txt
Train length: 10000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en-10k/qa4_two-arg-relations_{}.txt
Train length: 5213
Test length: 933
Intersection: 691
Unique samples in tasks_1-20_v1-2/en-10k/qa5_three-arg-relations_{}.txt
Train length: 9998
Test length: 1000
Intersection: 1
Unique samples in tasks_1-20_v1-2/en-10k/qa6_yes-no-questions_{}.txt
Train length: 9989
Test length: 1000
Intersection: 4
Unique samples in tasks_1-20_v1-2/en-10k/qa7_counting_{}.txt
Train length: 9942
Test length: 999
Intersection: 11
Unique samples in tasks_1-20_v1-2/en-10k/qa8_lists-sets_{}.txt
Train length: 9955
Test length: 1000
Intersection: 7
Unique samples in tasks_1-20_v1-2/en-10k/qa9_simple-negation_{}.txt
Train length: 9963
Test length: 999
Intersection: 9

uniq_tasks_1k.txt

Unique samples in tasks_1-20_v1-2/en/qa10_indefinite-knowledge_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa11_basic-coreference_{}.txt
Train length: 997
Test length: 997
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa12_conjunction_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa13_compound-coreference_{}.txt
Train length: 1000
Test length: 1000
Intersection: 1
Unique samples in tasks_1-20_v1-2/en/qa14_time-reasoning_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa15_basic-deduction_{}.txt
Train length: 695
Test length: 678
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa16_basic-induction_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa17_positional-reasoning_{}.txt
Train length: 627
Test length: 632
Intersection: 11
Unique samples in tasks_1-20_v1-2/en/qa18_size-reasoning_{}.txt
Train length: 654
Test length: 602
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa19_path-finding_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa1_single-supporting-fact_{}.txt
Train length: 997
Test length: 999
Intersection: 2
Unique samples in tasks_1-20_v1-2/en/qa20_agents-motivations_{}.txt
Train length: 909
Test length: 906
Intersection: 39
Unique samples in tasks_1-20_v1-2/en/qa2_two-supporting-facts_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa3_three-supporting-facts_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa4_two-arg-relations_{}.txt
Train length: 919
Test length: 933
Intersection: 124
Unique samples in tasks_1-20_v1-2/en/qa5_three-arg-relations_{}.txt
Train length: 1000
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa6_yes-no-questions_{}.txt
Train length: 999
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa7_counting_{}.txt
Train length: 998
Test length: 999
Intersection: 1
Unique samples in tasks_1-20_v1-2/en/qa8_lists-sets_{}.txt
Train length: 999
Test length: 1000
Intersection: 0
Unique samples in tasks_1-20_v1-2/en/qa9_simple-negation_{}.txt
Train length: 1000
Test length: 999
Intersection: 0