LLaVa 1.5 MMMU Validation Results

{'Overall-Art and Design': {'num': 120, 'mmmu_acc': 0.25}, 'Art': {'num': 30, 'mmmu_acc': 0.333}, 'Art_Theory': {'num': 30, 'mmmu_acc': 0.233}, 'Design': {'num': 30, 'mmmu_acc': 0.167}, 'Music': {'num': 30, 'mmmu_acc': 0.267}, 'Overall-Business': {'num': 150, 'mmmu_acc': 0.233}, 'Accounting': {'num': 30, 'mmmu_acc': 0.4}, 'Economics': {'num': 30, 'mmmu_acc': 0.233}, 'Finance': {'num': 30, 'mmmu_acc': 0.167}, 'Manage': {'num': 30, 'mmmu_acc': 0.133}, 'Marketing': {'num': 30, 'mmmu_acc': 0.233}, 'Overall-Science': {'num': 150, 'mmmu_acc': 0.193}, 'Biology': {'num': 30, 'mmmu_acc': 0.267}, 'Chemistry': {'num': 30, 'mmmu_acc': 0.1}, 'Geography': {'num': 30, 'mmmu_acc': 0.2}, 'Math': {'num': 30, 'mmmu_acc': 0.233}, 'Physics': {'num': 30, 'mmmu_acc': 0.167}, 'Overall-Health and Medicine': {'num': 150, 'mmmu_acc': 0.227}, 'Basic_Medical_Science': {'num': 30, 'mmmu_acc': 0.3}, 'Clinical_Medicine': {'num': 30, 'mmmu_acc': 0.2}, 'Diagnostics_and_Laboratory_Medicine': {'num': 30, 'mmmu_acc': 0.133}, 'Pharmacy': {'num': 30, 'mmmu_acc': 0.333}, 'Public_Health': {'num': 30, 'mmmu_acc': 0.167}, 'Overall-Humanities and Social Science': {'num': 120, 'mmmu_acc': 0.233}, 'History': {'num': 30, 'mmmu_acc': 0.167}, 'Literature': {'num': 30, 'mmmu_acc': 0.233}, 'Sociology': {'num': 30, 'mmmu_acc': 0.3}, 'Psychology': {'num': 30, 'mmmu_acc': 0.233}, 'Overall-Tech and Engineering': {'num': 210, 'mmmu_acc': 0.214}, 'Agriculture': {'num': 30, 'mmmu_acc': 0.167}, 'Architecture_and_Engineering': {'num': 30, 'mmmu_acc': 0.3}, 'Computer_Science': {'num': 30, 'mmmu_acc': 0.2}, 'Electronics': {'num': 30, 'mmmu_acc': 0.033}, 'Energy_and_Power': {'num': 30, 'mmmu_acc': 0.3}, 'Materials': {'num': 30, 'mmmu_acc': 0.133}, 'Mechanical_Engineering': {'num': 30, 'mmmu_acc': 0.367}, 'Overall': {'num': 900, 'mmmu_acc': 0.223}}
2024-06-17:07:19:16,069 INFO     [evaluation_tracker.py:237] Output path not provided, skipping saving results aggregated
llava (), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
| Tasks  |Version|Filter|n-shot|     Metric      |       |Value|   |Stderr|
|--------|-------|------|-----:|-----------------|------:|-----|---|------|
|mmmu_val|Yaml   |none  |     0|mmmu_acc_num     |900.000|±    |N/A|      |
|mmmu_val|Yaml   |none  |     0|mmmu_acc_mmmu_acc|  0.223|±    |N/A|      |