gblazex

IMPORTANT

Please duplicate this radar for a Safari fix! 
This will clean up a 50-line workaround.

  rdar://22376037 (https://openradar.appspot.com/radar?id=4965070979203072)

//////////////////////////////////////////////////////////////////////////////

(Now available as a standalone repo.)

gblazex

# 1. Install sglang: https://github.com/sgl-project/sglang?tab=readme-ov-file#install

# 2. MT-bench setup, follow steps here: https://github.com/sgl-project/sglang/tree/main/benchmark/mtbench

# (Benchmark code is here: https://github.com/sgl-project/sglang/blob/main/benchmark/mtbench/bench_sglang.py)

# 3. Only thing missing to be representative is temperature based on category:

# ...also change this line if you need in that file: 
# "num_gpus": 1,

gblazex

// ==UserScript==
// @name         Keyboard Shortcut Scripts
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  Run scripts with keyboard shortcuts
// @match        *://*/*
// @grant        none
// ==/UserScript==

// Parameters

gblazex

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
openchat_3.5	42.67	72.92	47.27	42.51	51.34

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	24.02	±	2.69
		acc_norm	24.80	±	2.72
agieval_logiqa_en	0	acc	38.86	±	1.91

gblazex

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
mistral-ft-optimized-1218	44.74	75.6	59.89	47.17	56.85

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	25.20	±	2.73
		acc_norm	24.02	±	2.69
agieval_logiqa_en	0	acc	39.32	±	1.92

gblazex

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
zephyr-7b-beta	37.33	71.83	55.1	39.7	50.99

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	21.26	±	2.57
		acc_norm	20.47	±	2.54
agieval_logiqa_en	0	acc	33.33	±	1.85

gblazex

[
  {
    "dataset":"helpful_base",
    "instruction":"What are the names of some famous actors that started their careers on Broadway?",
    "output":"1. Meryl Streep\n2. Angela Lansbury\n3. Audra McDonald\n4. Bernadette Peters\n5. Idina Menzel\n6. Patti LuPone\n7. Hugh Jackman\n8. James Earl Jones\n9. Liza Minnelli\n10. Nathan Lane",
    "generator":"dolphin-2.2.1-mistral-7b"
  },
  {
    "dataset":"helpful_base",
    "instruction":"How did US states get their names?",

gblazex

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
neuronovo-7B-v0.2	44.95	76.49	71.57	47.48	60.12

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	25.98	±	2.76
		acc_norm	25.59	±	2.74
agieval_logiqa_en	0	acc	37.48	±	1.90

gblazex

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
distilabeled-Marcoro14-7B-slerp	45.38	76.48	65.68	48.18	58.93

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	27.56	±	2.81
		acc_norm	25.98	±	2.76
agieval_logiqa_en	0	acc	39.17	±	1.91

gblazex

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
openchat-3.5-1210	42.62	72.84	53.21	43.88	53.14

AGIEval

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	22.44	±	2.62
		acc_norm	24.41	±	2.70
agieval_logiqa_en	0	acc	41.17	±	1.93