gemma-7b-Nous.md

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
gemma-7b	20.64	38.49	46.61	28.51	33.56

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	22.44	±	2.62
		acc_norm	19.69	±	2.50
agieval_logiqa_en	0	acc	23.96	±	1.67
		acc_norm	26.88	±	1.74
agieval_lsat_ar	0	acc	17.39	±	2.50
		acc_norm	17.39	±	2.50
agieval_lsat_lr	0	acc	14.31	±	1.55
		acc_norm	22.35	±	1.85
agieval_lsat_rc	0	acc	21.93	±	2.53
		acc_norm	15.61	±	2.22
agieval_sat_en	0	acc	25.24	±	3.03
		acc_norm	20.39	±	2.81
agieval_sat_en_without_passage	0	acc	21.84	±	2.89
		acc_norm	22.33	±	2.91
agieval_sat_math	0	acc	25.91	±	2.96
		acc_norm	20.45	±	2.73

Average: 20.64%

GPT4All

Task	Version	Metric	Value		Stderr
arc_challenge	0	acc	19.54	±	1.16
		acc_norm	22.70	±	1.22
arc_easy	0	acc	28.24	±	0.92
		acc_norm	28.54	±	0.93
boolq	1	acc	62.14	±	0.85
hellaswag	0	acc	25.93	±	0.44
		acc_norm	27.45	±	0.45
openbookqa	0	acc	15.40	±	1.62
		acc_norm	27.00	±	1.99
piqa	0	acc	55.50	±	1.16
		acc_norm	52.07	±	1.17
winogrande	0	acc	49.57	±	1.41

Average: 38.49%

TruthfulQA

Task	Version	Metric	Value		Stderr
truthfulqa_mc	1	mc1	21.18	±	1.43
		mc2	46.61	±	1.69

Average: 46.61%

Bigbench

Task	Version	Metric	Value		Stderr
bigbench_causal_judgement	0	multiple_choice_grade	52.11	±	3.63
bigbench_date_understanding	0	multiple_choice_grade	16.53	±	1.94
bigbench_disambiguation_qa	0	multiple_choice_grade	31.78	±	2.90
bigbench_geometric_shapes	0	multiple_choice_grade	11.14	±	1.66
		exact_str_match	0.00	±	0.00
bigbench_logical_deduction_five_objects	0	multiple_choice_grade	20.60	±	1.81
bigbench_logical_deduction_seven_objects	0	multiple_choice_grade	14.57	±	1.33
bigbench_logical_deduction_three_objects	0	multiple_choice_grade	32.67	±	2.71
bigbench_movie_recommendation	0	multiple_choice_grade	29.00	±	2.03
bigbench_navigate	0	multiple_choice_grade	50.00	±	1.58
bigbench_reasoning_about_colored_objects	0	multiple_choice_grade	16.50	±	0.83
bigbench_ruin_names	0	multiple_choice_grade	31.03	±	2.19
bigbench_salient_translation_error_detection	0	multiple_choice_grade	16.83	±	1.18
bigbench_snarks	0	multiple_choice_grade	51.38	±	3.73
bigbench_sports_understanding	0	multiple_choice_grade	49.70	±	1.59
bigbench_temporal_sequences	0	multiple_choice_grade	23.30	±	1.34
bigbench_tracking_shuffled_objects_five_objects	0	multiple_choice_grade	19.36	±	1.12
bigbench_tracking_shuffled_objects_seven_objects	0	multiple_choice_grade	14.06	±	0.83
bigbench_tracking_shuffled_objects_three_objects	0	multiple_choice_grade	32.67	±	2.71

Average: 28.51%

Average score: 33.56%

Elapsed time: 03:09:04