Hermes-2-Pro-Llama-3-8B-Nous.md

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
Hermes-2-Pro-Llama-3-8B	42.52	72.64	57.8	43.53	54.12

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	24.80	±	2.72
		acc_norm	25.59	±	2.74
agieval_logiqa_en	0	acc	35.48	±	1.88
		acc_norm	36.25	±	1.89
agieval_lsat_ar	0	acc	18.26	±	2.55
		acc_norm	19.13	±	2.60
agieval_lsat_lr	0	acc	54.90	±	2.21
		acc_norm	52.75	±	2.21
agieval_lsat_rc	0	acc	64.31	±	2.93
		acc_norm	60.97	±	2.98
agieval_sat_en	0	acc	73.30	±	3.09
		acc_norm	70.39	±	3.19
agieval_sat_en_without_passage	0	acc	40.29	±	3.43
		acc_norm	36.89	±	3.37
agieval_sat_math	0	acc	39.55	±	3.30
		acc_norm	38.18	±	3.28

Average: 42.52%

GPT4All

Task	Version	Metric	Value		Stderr
arc_challenge	0	acc	55.12	±	1.45
		acc_norm	58.62	±	1.44
arc_easy	0	acc	83.42	±	0.76
		acc_norm	81.19	±	0.80
boolq	1	acc	85.87	±	0.61
hellaswag	0	acc	62.66	±	0.48
		acc_norm	80.52	±	0.40
openbookqa	0	acc	38.00	±	2.17
		acc_norm	46.00	±	2.23
piqa	0	acc	79.98	±	0.93
		acc_norm	81.12	±	0.91
winogrande	0	acc	75.14	±	1.21

Average: 72.64%

TruthfulQA

Task	Version	Metric	Value		Stderr
truthfulqa_mc	1	mc1	41.13	±	1.72
		mc2	57.80	±	1.57

Average: 57.8%

Bigbench

Task	Version	Metric	Value		Stderr
bigbench_causal_judgement	0	multiple_choice_grade	57.37	±	3.60
bigbench_date_understanding	0	multiple_choice_grade	66.67	±	2.46
bigbench_disambiguation_qa	0	multiple_choice_grade	31.78	±	2.90
bigbench_geometric_shapes	0	multiple_choice_grade	17.55	±	2.01
		exact_str_match	0.00	±	0.00
bigbench_logical_deduction_five_objects	0	multiple_choice_grade	30.80	±	2.07
bigbench_logical_deduction_seven_objects	0	multiple_choice_grade	20.29	±	1.52
bigbench_logical_deduction_three_objects	0	multiple_choice_grade	55.00	±	2.88
bigbench_movie_recommendation	0	multiple_choice_grade	43.00	±	2.22
bigbench_navigate	0	multiple_choice_grade	49.80	±	1.58
bigbench_reasoning_about_colored_objects	0	multiple_choice_grade	70.15	±	1.02
bigbench_ruin_names	0	multiple_choice_grade	46.65	±	2.36
bigbench_salient_translation_error_detection	0	multiple_choice_grade	19.74	±	1.26
bigbench_snarks	0	multiple_choice_grade	74.03	±	3.27
bigbench_sports_understanding	0	multiple_choice_grade	54.26	±	1.59
bigbench_temporal_sequences	0	multiple_choice_grade	53.10	±	1.58
bigbench_tracking_shuffled_objects_five_objects	0	multiple_choice_grade	22.96	±	1.19
bigbench_tracking_shuffled_objects_seven_objects	0	multiple_choice_grade	15.37	±	0.86
bigbench_tracking_shuffled_objects_three_objects	0	multiple_choice_grade	55.00	±	2.88

Average: 43.53%

Average score: 54.12%

Elapsed time: 04:11:59