{{ message }}

Instantly share code, notes, and snippets.

# sile/_main.md

Last active Dec 26, 2019

# Benchmark Result Report

• Report ID: 4ba0c3ff34e08ddaac04387c13628701e21fd683a713b496b3f78c3d541beac9
• Kurobako Version: 0.1.3
• Number of Solvers: 3
• Number of Problems: 4
• Metrics Precedence: `best value -> AUC`

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

## Overall Results

Solver Borda Firsts
Hyperband (n_brackets=3) 1 4
Hyperband (n_brackets=4) 1 4
Hyperband (n_brackets=5) 0 3

## Individual Results

### (1) Problem: HPO-Bench-Parkinson

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.010274 +- 0.002866 0.593 +- 0.168 77.627 +- 4.032
1 Hyperband (n_brackets=3) (study) 0.010998 +- 0.004389 0.610 +- 0.216 78.957 +- 4.106
1 Hyperband (n_brackets=5) (study) 0.010452 +- 0.003867 0.683 +- 0.214 76.188 +- 3.679

### (2) Problem: HPO-Bench-Naval

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.000088 +- 0.000083 0.015 +- 0.016 78.439 +- 3.745
1 Hyperband (n_brackets=3) (study) 0.000113 +- 0.000151 0.023 +- 0.034 81.952 +- 5.304
1 Hyperband (n_brackets=5) (study) 0.000095 +- 0.000095 0.023 +- 0.026 76.634 +- 3.675

### (3) Problem: HPO-Bench-Protein

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.229968 +- 0.008488 10.355 +- 0.517 77.086 +- 4.335
1 Hyperband (n_brackets=3) (study) 0.229168 +- 0.006514 10.320 +- 0.377 78.119 +- 4.569
3 Hyperband (n_brackets=5) (study) 0.235814 +- 0.011201 10.631 +- 0.555 75.984 +- 3.760

### (4) Problem: HPO-Bench-Slice

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.000360 +- 0.000145 0.021 +- 0.008 77.230 +- 5.050
1 Hyperband (n_brackets=3) (study) 0.000315 +- 0.000083 0.018 +- 0.006 79.189 +- 5.027
1 Hyperband (n_brackets=5) (study) 0.000310 +- 0.000119 0.022 +- 0.011 75.585 +- 5.596

## Solvers

### ID: e7f3a145b144a9a3b2bc99b714e4e9f698e8a67f82ece410de77e2abd496c4de

recipe:

```{
"name": "Hyperband (n_brackets=3)",
"command": {
"path": "python",
"args": [
"hyperband-solver.py",
"--n-brackets",
"3"
]
}
}```

specification:

```{
"name": "Hyperband (n_brackets=3)",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=0.19.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}```

### ID: a04abee366e2d572f34e07fbf80e1796ab42e10a0872755487a1de10a9946a5b

recipe:

```{
"name": "Hyperband (n_brackets=4)",
"command": {
"path": "python",
"args": [
"hyperband-solver.py",
"--n-brackets",
"4"
]
}
}```

specification:

```{
"name": "Hyperband (n_brackets=4)",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=0.19.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}```

### ID: f94c12441e4b8915be38df76527f83b9094787327a83115a835e8a20e97385e3

recipe:

```{
"name": "Hyperband (n_brackets=5)",
"command": {
"path": "python",
"args": [
"hyperband-solver.py",
"--n-brackets",
"5"
]
}
}```

specification:

```{
"name": "Hyperband (n_brackets=5)",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=0.19.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}```

## Problems

### ID: 880b007e3c61f4aecb7e9b0aa2f9be5fea9f491a076853f68f402769aa254034

recipe:

```{
"hpobench": {
"dataset": "./fcnet_naval_propulsion_data.hdf5"
}
}```

specification:

```{
"name": "HPO-Bench-Naval",
"attrs": {
"github": "https://github.com/automl/nas_benchmarks",
"paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
"version": "kurobako_problems=0.1.3"
},
"params_domain": [
{
"name": "activation_fn_1",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "activation_fn_2",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "batch_size",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 4
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "init_lr",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "lr_schedule",
"range": {
"type": "CATEGORICAL",
"choices": [
"cosine",
"const"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
}
],
"values_domain": [
{
"name": "Validation MSE",
"range": {
"type": "CONTINUOUS",
"low": 0.0
},
"distribution": "UNIFORM",
"constraint": null
}
],
"steps": 100
}```

### ID: 445bfa45fdbb8ec6ae6d4dba1909114f3948fa67b47209258db9291480b405b5

recipe:

```{
"hpobench": {
"dataset": "./fcnet_parkinsons_telemonitoring_data.hdf5"
}
}```

specification:

```{
"name": "HPO-Bench-Parkinson",
"attrs": {
"github": "https://github.com/automl/nas_benchmarks",
"paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
"version": "kurobako_problems=0.1.3"
},
"params_domain": [
{
"name": "activation_fn_1",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "activation_fn_2",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "batch_size",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 4
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "init_lr",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "lr_schedule",
"range": {
"type": "CATEGORICAL",
"choices": [
"cosine",
"const"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
}
],
"values_domain": [
{
"name": "Validation MSE",
"range": {
"type": "CONTINUOUS",
"low": 0.0
},
"distribution": "UNIFORM",
"constraint": null
}
],
"steps": 100
}```

recipe:

```{
"hpobench": {
"dataset": "./fcnet_protein_structure_data.hdf5"
}
}```

specification:

```{
"name": "HPO-Bench-Protein",
"attrs": {
"github": "https://github.com/automl/nas_benchmarks",
"paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
"version": "kurobako_problems=0.1.3"
},
"params_domain": [
{
"name": "activation_fn_1",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "activation_fn_2",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "batch_size",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 4
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "init_lr",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "lr_schedule",
"range": {
"type": "CATEGORICAL",
"choices": [
"cosine",
"const"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
}
],
"values_domain": [
{
"name": "Validation MSE",
"range": {
"type": "CONTINUOUS",
"low": 0.0
},
"distribution": "UNIFORM",
"constraint": null
}
],
"steps": 100
}```

### ID: d88e1704447639bde17f236f7af47f93274d1f02bc8ec66733146ff9cdf50196

recipe:

```{
"hpobench": {
"dataset": "./fcnet_slice_localization_data.hdf5"
}
}```

specification:

```{
"name": "HPO-Bench-Slice",
"attrs": {
"github": "https://github.com/automl/nas_benchmarks",
"paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
"version": "kurobako_problems=0.1.3"
},
"params_domain": [
{
"name": "activation_fn_1",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "activation_fn_2",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "batch_size",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 4
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "init_lr",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "lr_schedule",
"range": {
"type": "CATEGORICAL",
"choices": [
"cosine",
"const"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
}
],
"values_domain": [
{
"name": "Validation MSE",
"range": {
"type": "CONTINUOUS",
"low": 0.0
},
"distribution": "UNIFORM",
"constraint": null
}
],
"steps": 100
}```