gemma-2b-it-Nous.md

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
gemma-2b-it	23.76	43.6	47.64	29.41	36.1

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	23.62	±	2.67
		acc_norm	23.62	±	2.67
agieval_logiqa_en	0	acc	23.66	±	1.67
		acc_norm	24.88	±	1.70
agieval_lsat_ar	0	acc	18.26	±	2.55
		acc_norm	19.13	±	2.60
agieval_lsat_lr	0	acc	22.35	±	1.85
		acc_norm	23.33	±	1.87
agieval_lsat_rc	0	acc	20.82	±	2.48
		acc_norm	20.45	±	2.46
agieval_sat_en	0	acc	32.52	±	3.27
		acc_norm	32.52	±	3.27
agieval_sat_en_without_passage	0	acc	24.76	±	3.01
		acc_norm	24.76	±	3.01
agieval_sat_math	0	acc	25.00	±	2.93
		acc_norm	21.36	±	2.77

Average: 23.76%

GPT4All

Task	Version	Metric	Value		Stderr
arc_challenge	0	acc	21.59	±	1.20
		acc_norm	24.06	±	1.25
arc_easy	0	acc	37.37	±	0.99
		acc_norm	36.91	±	0.99
boolq	1	acc	65.66	±	0.83
hellaswag	0	acc	34.56	±	0.47
		acc_norm	40.35	±	0.49
openbookqa	0	acc	15.40	±	1.62
		acc_norm	27.00	±	1.99
piqa	0	acc	60.23	±	1.14
		acc_norm	60.39	±	1.14
winogrande	0	acc	50.83	±	1.41

Average: 43.6%

TruthfulQA

Task	Version	Metric	Value		Stderr
truthfulqa_mc	1	mc1	30.35	±	1.61
		mc2	47.64	±	1.61

Average: 47.64%

Bigbench

Task	Version	Metric	Value		Stderr
bigbench_causal_judgement	0	multiple_choice_grade	50.00	±	3.64
bigbench_date_understanding	0	multiple_choice_grade	22.76	±	2.19
bigbench_disambiguation_qa	0	multiple_choice_grade	36.05	±	3.00
bigbench_geometric_shapes	0	multiple_choice_grade	7.24	±	1.37
		exact_str_match	0.00	±	0.00
bigbench_logical_deduction_five_objects	0	multiple_choice_grade	24.80	±	1.93
bigbench_logical_deduction_seven_objects	0	multiple_choice_grade	19.43	±	1.50
bigbench_logical_deduction_three_objects	0	multiple_choice_grade	34.67	±	2.75
bigbench_movie_recommendation	0	multiple_choice_grade	26.20	±	1.97
bigbench_navigate	0	multiple_choice_grade	50.00	±	1.58
bigbench_reasoning_about_colored_objects	0	multiple_choice_grade	26.90	±	0.99
bigbench_ruin_names	0	multiple_choice_grade	23.44	±	2.00
bigbench_salient_translation_error_detection	0	multiple_choice_grade	16.93	±	1.19
bigbench_snarks	0	multiple_choice_grade	54.70	±	3.71
bigbench_sports_understanding	0	multiple_choice_grade	49.59	±	1.59
bigbench_temporal_sequences	0	multiple_choice_grade	26.80	±	1.40
bigbench_tracking_shuffled_objects_five_objects	0	multiple_choice_grade	14.80	±	1.00
bigbench_tracking_shuffled_objects_seven_objects	0	multiple_choice_grade	10.40	±	0.73
bigbench_tracking_shuffled_objects_three_objects	0	multiple_choice_grade	34.67	±	2.75

Average: 29.41%

Average score: 36.1%

Elapsed time: 01:45:32