Skip to content

Instantly share code, notes, and snippets.

@mlabonne
Last active April 20, 2024 11:06
Show Gist options
  • Save mlabonne/f41dad371d1781d0434a4672fd6f0b82 to your computer and use it in GitHub Desktop.
Save mlabonne/f41dad371d1781d0434a4672fd6f0b82 to your computer and use it in GitHub Desktop.
Model AGIEval GPT4All TruthfulQA Bigbench Average
OrpoLlama-3-8B 31.56 70.19 48.11 37.17 46.76

AGIEval

Task Version Metric Value Stderr
agieval_aqua_rat 0 acc 20.47 ± 2.54
acc_norm 23.23 ± 2.65
agieval_logiqa_en 0 acc 28.11 ± 1.76
acc_norm 33.49 ± 1.85
agieval_lsat_ar 0 acc 18.26 ± 2.55
acc_norm 21.74 ± 2.73
agieval_lsat_lr 0 acc 36.27 ± 2.13
acc_norm 30.98 ± 2.05
agieval_lsat_rc 0 acc 44.61 ± 3.04
acc_norm 34.20 ± 2.90
agieval_sat_en 0 acc 59.22 ± 3.43
acc_norm 45.63 ± 3.48
agieval_sat_en_without_passage 0 acc 38.83 ± 3.40
acc_norm 28.64 ± 3.16
agieval_sat_math 0 acc 41.82 ± 3.33
acc_norm 34.55 ± 3.21

Average: 31.56%

GPT4All

Task Version Metric Value Stderr
arc_challenge 0 acc 50.17 ± 1.46
acc_norm 53.84 ± 1.46
arc_easy 0 acc 80.51 ± 0.81
acc_norm 77.99 ± 0.85
boolq 1 acc 81.71 ± 0.68
hellaswag 0 acc 60.31 ± 0.49
acc_norm 78.99 ± 0.41
openbookqa 0 acc 34.60 ± 2.13
acc_norm 44.60 ± 2.23
piqa 0 acc 79.82 ± 0.94
acc_norm 80.96 ± 0.92
winogrande 0 acc 73.24 ± 1.24

Average: 70.19%

TruthfulQA

Task Version Metric Value Stderr
truthfulqa_mc 1 mc1 30.84 ± 1.62
mc2 48.11 ± 1.45

Average: 48.11%

Bigbench

Task Version Metric Value Stderr
bigbench_causal_judgement 0 multiple_choice_grade 56.84 ± 3.60
bigbench_date_understanding 0 multiple_choice_grade 69.11 ± 2.41
bigbench_disambiguation_qa 0 multiple_choice_grade 31.78 ± 2.90
bigbench_geometric_shapes 0 multiple_choice_grade 22.56 ± 2.21
exact_str_match 0.00 ± 0.00
bigbench_logical_deduction_five_objects 0 multiple_choice_grade 27.20 ± 1.99
bigbench_logical_deduction_seven_objects 0 multiple_choice_grade 20.43 ± 1.52
bigbench_logical_deduction_three_objects 0 multiple_choice_grade 43.00 ± 2.86
bigbench_movie_recommendation 0 multiple_choice_grade 31.40 ± 2.08
bigbench_navigate 0 multiple_choice_grade 52.00 ± 1.58
bigbench_reasoning_about_colored_objects 0 multiple_choice_grade 57.10 ± 1.11
bigbench_ruin_names 0 multiple_choice_grade 29.24 ± 2.15
bigbench_salient_translation_error_detection 0 multiple_choice_grade 25.15 ± 1.37
bigbench_snarks 0 multiple_choice_grade 46.41 ± 3.72
bigbench_sports_understanding 0 multiple_choice_grade 51.01 ± 1.59
bigbench_temporal_sequences 0 multiple_choice_grade 25.90 ± 1.39
bigbench_tracking_shuffled_objects_five_objects 0 multiple_choice_grade 21.12 ± 1.15
bigbench_tracking_shuffled_objects_seven_objects 0 multiple_choice_grade 15.77 ± 0.87
bigbench_tracking_shuffled_objects_three_objects 0 multiple_choice_grade 43.00 ± 2.86

Average: 37.17%

Average score: 46.76%

Elapsed time: 06:20:48

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment