Benchmark Result Report
- Report ID: ecb7024f149c2653a906f600d048eac2658f94bc2cfc8e50bfd7356c0fb1e29b
- Kurobako Version: 0.1.3
- Number of Solvers: 4
- Number of Problems: 2
- Metrics Precedence:
best value -> AUC
Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.
Table of Contents
Overall Results
Solver | Borda | Firsts |
---|---|---|
RandomSampler | 0 | 0 |
TPESampler | 2 | 1 |
TPESampler with MedianPruner | 2 | 1 |
TPESampler with SuccessiveHalvingPruner | 4 | 2 |
Individual Results
HPO-Bench-Parkinson
(1) Problem:Ranking | Solver | Best (avg +- sd) | AUC (avg +- sd) | Elapsed (avg +- sd) |
---|---|---|---|---|
1 | TPESampler with MedianPruner (study) | 0.010070 +- 0.004299 | 0.803 +- 0.299 | 132.749 +- 15.810 |
1 | TPESampler (study) | 0.009882 +- 0.003529 | 0.945 +- 0.354 | 2.775 +- 0.349 |
1 | TPESampler with SuccessiveHalvingPruner (study) | 0.011867 +- 0.003252 | 0.879 +- 0.279 | 26.767 +- 10.526 |
4 | RandomSampler (study) | 0.015558 +- 0.005880 | 1.304 +- 0.444 | 0.614 +- 0.083 |
HPO-Bench-Protein
(2) Problem:Ranking | Solver | Best (avg +- sd) | AUC (avg +- sd) | Elapsed (avg +- sd) |
---|---|---|---|---|
1 | TPESampler with SuccessiveHalvingPruner (study) | 0.231011 +- 0.009908 | 17.172 +- 0.744 | 21.536 +- 8.320 |
2 | TPESampler with MedianPruner (study) | 0.232737 +- 0.010008 | 17.874 +- 0.884 | 131.829 +- 12.683 |
2 | TPESampler (study) | 0.236102 +- 0.014100 | 18.529 +- 1.390 | 2.781 +- 0.370 |
4 | RandomSampler (study) | 0.255230 +- 0.016135 | 19.817 +- 1.112 | 0.624 +- 0.080 |
Solvers
ID: fba765577d406565f676869bca11c65117476c35f2dbf3d0fc5d9025ffda9a43
recipe:
{
"name": "RandomSampler",
"optuna": {
"sampler": "random",
"pruner": "nop"
}
}
specification:
{
"name": "RandomSampler",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=1.0.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}
ID: 870a5088fd88d2fb78e09b23b5a123f76e562a194c038eccd5b5e51bd95931cf
recipe:
{
"name": "TPESampler",
"optuna": {
"pruner": "nop"
}
}
specification:
{
"name": "TPESampler",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=1.0.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}
ID: 679fa4abbe034c1371f96080e14c0796e5257021868dc8e7c666cc25929bca75
recipe:
{
"name": "TPESampler with MedianPruner",
"optuna": {
"median_warmup_steps": 4
}
}
specification:
{
"name": "TPESampler with MedianPruner",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=1.0.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}
ID: cc04a6c5ae0a5d76bbfd007aa9f3ca55b672b949760b3f92a4b390bfc52195c0
recipe:
{
"name": "TPESampler with SuccessiveHalvingPruner",
"optuna": {
"pruner": "asha",
"asha_min_resource": 4
}
}
specification:
{
"name": "TPESampler with SuccessiveHalvingPruner",
"attrs": {
"github": "https://github.com/optuna/optuna",
"paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
"version": "optuna=1.0.0, kurobako-py=0.1.1"
},
"capabilities": [
"UNIFORM_CONTINUOUS",
"UNIFORM_DISCRETE",
"LOG_UNIFORM_CONTINUOUS",
"CATEGORICAL",
"CONDITIONAL",
"CONCURRENT"
]
}
Problems
ID: 4f3ab413a790160a528bbc3790c3a332868e4bd2268005108644e09e2e3968b8
recipe:
{
"hpobench": {
"dataset": "fcnet_parkinsons_telemonitoring_data.hdf5"
}
}
specification:
{
"name": "HPO-Bench-Parkinson",
"attrs": {
"github": "https://github.com/automl/nas_benchmarks",
"paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
"version": "kurobako_problems=0.1.3"
},
"params_domain": [
{
"name": "activation_fn_1",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "activation_fn_2",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "batch_size",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 4
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "init_lr",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "lr_schedule",
"range": {
"type": "CATEGORICAL",
"choices": [
"cosine",
"const"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
}
],
"values_domain": [
{
"name": "Validation MSE",
"range": {
"type": "CONTINUOUS",
"low": 0.0
},
"distribution": "UNIFORM",
"constraint": null
}
],
"steps": 100
}
ID: 81bde99ec2ab16438bf4c7145468df6993c53913a034bd3ee4ee65fdbf1a3bcf
recipe:
{
"hpobench": {
"dataset": "fcnet_protein_structure_data.hdf5"
}
}
specification:
{
"name": "HPO-Bench-Protein",
"attrs": {
"github": "https://github.com/automl/nas_benchmarks",
"paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
"version": "kurobako_problems=0.1.3"
},
"params_domain": [
{
"name": "activation_fn_1",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "activation_fn_2",
"range": {
"type": "CATEGORICAL",
"choices": [
"tanh",
"relu"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "batch_size",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 4
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "dropout_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 3
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "init_lr",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "lr_schedule",
"range": {
"type": "CATEGORICAL",
"choices": [
"cosine",
"const"
]
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_1",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
},
{
"name": "n_units_2",
"range": {
"type": "DISCRETE",
"low": 0,
"high": 6
},
"distribution": "UNIFORM",
"constraint": null
}
],
"values_domain": [
{
"name": "Validation MSE",
"range": {
"type": "CONTINUOUS",
"low": 0.0
},
"distribution": "UNIFORM",
"constraint": null
}
],
"steps": 100
}
Studies
ID: 12e94aa3b91bc181e091a67a1725adbc21d6f6fc838f8eaaa1a4fb1d3b3cbadf
- problem: HPO-Bench-Parkinson
- solver: RandomSampler
- budget: 80
- repeats: 30
- concurrency: 1
ID: 431414b3e295d2647ebca78d922daa39b5091de27f73d648986e5589a8c4a222
- problem: HPO-Bench-Parkinson
- solver: TPESampler
- budget: 80
- repeats: 30
- concurrency: 1
ID: 09968fca2def713da7c876682aa0fee4ab350fdec71388d2890bb8ac4106168e
- problem: HPO-Bench-Parkinson
- solver: TPESampler with MedianPruner
- budget: 80
- repeats: 30
- concurrency: 1
ID: 4ddb19c7790a16c70fa771b63ff831577b3db6aef7c12fdb75d319b8a6457922
- problem: HPO-Bench-Parkinson
- solver: TPESampler with SuccessiveHalvingPruner
- budget: 80
- repeats: 30
- concurrency: 1
ID: 28a919018ae3856bd6ba62cb273fabd521827b078b2b482ccaf2db1acda6b977
- problem: HPO-Bench-Protein
- solver: RandomSampler
- budget: 80
- repeats: 30
- concurrency: 1
ID: d8e15e76a9cd34a3ca46921edc1d6afdda0c3f1a0dc3b3060ec0bf445576aebe
- problem: HPO-Bench-Protein
- solver: TPESampler
- budget: 80
- repeats: 30
- concurrency: 1
ID: feb526ce52a481282f4074285bc3c6ddcecccb6b2902325c162a9d750697bec8
- problem: HPO-Bench-Protein
- solver: TPESampler with MedianPruner
- budget: 80
- repeats: 30
- concurrency: 1
ID: ec52b06a2a3e5529ceeab40e6e7ac97cda7afd3621ee62901b9d29c105348298
- problem: HPO-Bench-Protein
- solver: TPESampler with SuccessiveHalvingPruner
- budget: 80
- repeats: 30
- concurrency: 1