codelion

providers:
  # Local server 1 - Fastest local, gets first priority
  - name: gemma_server_1
    base_url: http://192.168.10.145:11111/v1
    api_key: dummy
    weight: 100  # Highest weight - will get requests first
    max_concurrent: 1  # This server can only handle 1 request at a time

  # Local server 2 - Slower local, gets second priority  
  - name: gemma_server_2

codelion

import numpy as np
from sentence_transformers import SentenceTransformer
import anthropic
from typing import List, Dict, Optional
import logging
from dataclasses import dataclass
import uuid

logger = logging.getLogger(__name__)

codelion

Model	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K	Average
mera-mix-4x7B-v1	46.67	69.64	39.65	46.78	68.27	2.73	45.62

ARC

Task	Version	Metric	Value		Stderr
arc_challenge	1	acc,none	0.44
		acc_stderr,none	0.01
		acc_norm,none	0.47

codelion

Model	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K	Average
mera-mix-4x7B	72.01	88.82	63.67	77.45	84.61	71.65	76.37

ARC

Task	Version	Metric	Value		Stderr
arc_challenge	1	acc,none	0.70
		acc_stderr,none	0.01
		acc_norm,none	0.72

codelion

Model	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K
mera-mix-4x7B	65.7	84.73	Error: File does not exist	51.03	79.48	66.34

ARC

Task	Version	Metric	Value		Stderr
arc_challenge	1	acc,none	0.62
		acc_stderr,none	0.01
		acc_norm,none	0.66

codelion

Model	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K
mera-mix-4x7B	Error: File does not exist	Error: File does not exist	Error: File does not exist	Error: File does not exist	Error: File does not exist	Error: File does not exist

ARC

Average: Error: File does not exist%

HellaSwag

codelion

Model	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K
mera-mix-4x7B	Error: File does not exist	Error: File does not exist	Error: File does not exist	Error: File does not exist	Error: File does not exist	Error: File does not exist

ARC

Average: Error: File does not exist%

HellaSwag

codelion

struct node{
 int val;
 struct node* next;
};

/*@
list<> == self=null or self::node<_,q> * q::list<>; 
*/

int length(struct node* p)

codelion

struct node{
 int val;
 struct node* next;
};

/*@
list<> == self=null or self::node<_,q> * q::list<>; 
*/

int length(struct node* p)

codelion

default-src 'self'; font-src 'self' https://maxcdn.bootstrapcdn.com; form-action 'self'; frame-ancestors 'none'; report-uri https://report-uri.io/report/camjackson; script-src 'self' https://oss.maxcdn.com; style-src 'self' https://maxcdn.bootstrapcdn.com https://cdn.jsdelivr.net