Skip to content

Instantly share code, notes, and snippets.

@mlabonne
Created March 31, 2024 19:59
Show Gist options
  • Save mlabonne/719d5e106eefbcffb951b65616dcbec4 to your computer and use it in GitHub Desktop.
Save mlabonne/719d5e106eefbcffb951b65616dcbec4 to your computer and use it in GitHub Desktop.
Model AGIEval GPT4All TruthfulQA Bigbench Average
Zebrafish-7B 44.92 77.18 78.25 49.28 62.41

AGIEval

Task Version Metric Value Stderr
agieval_aqua_rat 0 acc 28.35 ± 2.83
acc_norm 25.59 ± 2.74
agieval_logiqa_en 0 acc 39.32 ± 1.92
acc_norm 40.09 ± 1.92
agieval_lsat_ar 0 acc 23.04 ± 2.78
acc_norm 22.61 ± 2.76
agieval_lsat_lr 0 acc 53.33 ± 2.21
acc_norm 54.31 ± 2.21
agieval_lsat_rc 0 acc 66.54 ± 2.88
acc_norm 65.43 ± 2.91
agieval_sat_en 0 acc 77.18 ± 2.93
acc_norm 77.67 ± 2.91
agieval_sat_en_without_passage 0 acc 42.23 ± 3.45
acc_norm 42.72 ± 3.45
agieval_sat_math 0 acc 34.09 ± 3.20
acc_norm 30.91 ± 3.12

Average: 44.92%

GPT4All

Task Version Metric Value Stderr
arc_challenge 0 acc 67.49 ± 1.37
acc_norm 67.83 ± 1.37
arc_easy 0 acc 86.24 ± 0.71
acc_norm 80.01 ± 0.82
boolq 1 acc 87.31 ± 0.58
hellaswag 0 acc 69.94 ± 0.46
acc_norm 87.51 ± 0.33
openbookqa 0 acc 39.80 ± 2.19
acc_norm 49.60 ± 2.24
piqa 0 acc 83.19 ± 0.87
acc_norm 85.47 ± 0.82
winogrande 0 acc 82.56 ± 1.07

Average: 77.18%

TruthfulQA

Task Version Metric Value Stderr
truthfulqa_mc 1 mc1 62.91 ± 1.69
mc2 78.25 ± 1.37

Average: 78.25%

Bigbench

Task Version Metric Value Stderr
bigbench_causal_judgement 0 multiple_choice_grade 58.95 ± 3.58
bigbench_date_understanding 0 multiple_choice_grade 60.98 ± 2.54
bigbench_disambiguation_qa 0 multiple_choice_grade 51.16 ± 3.12
bigbench_geometric_shapes 0 multiple_choice_grade 22.84 ± 2.22
exact_str_match 0.00 ± 0.00
bigbench_logical_deduction_five_objects 0 multiple_choice_grade 32.80 ± 2.10
bigbench_logical_deduction_seven_objects 0 multiple_choice_grade 24.71 ± 1.63
bigbench_logical_deduction_three_objects 0 multiple_choice_grade 59.33 ± 2.84
bigbench_movie_recommendation 0 multiple_choice_grade 52.80 ± 2.23
bigbench_navigate 0 multiple_choice_grade 57.00 ± 1.57
bigbench_reasoning_about_colored_objects 0 multiple_choice_grade 68.90 ± 1.04
bigbench_ruin_names 0 multiple_choice_grade 57.14 ± 2.34
bigbench_salient_translation_error_detection 0 multiple_choice_grade 39.18 ± 1.55
bigbench_snarks 0 multiple_choice_grade 72.93 ± 3.31
bigbench_sports_understanding 0 multiple_choice_grade 74.95 ± 1.38
bigbench_temporal_sequences 0 multiple_choice_grade 52.10 ± 1.58
bigbench_tracking_shuffled_objects_five_objects 0 multiple_choice_grade 22.96 ± 1.19
bigbench_tracking_shuffled_objects_seven_objects 0 multiple_choice_grade 18.97 ± 0.94
bigbench_tracking_shuffled_objects_three_objects 0 multiple_choice_grade 59.33 ± 2.84

Average: 49.28%

Average score: 62.41%

Elapsed time: 03:34:52

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment