Benchmark Result Report

## report.md

      
          Raw
        
        
              report.md
            
          
    Benchmark Result Report

Report ID: ecb7024f149c2653a906f600d048eac2658f94bc2cfc8e50bfd7356c0fb1e29b
Kurobako Version: 0.1.3
Number of Solvers: 4
Number of Problems: 2
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.
Table of Contents

Overall Results
Individual Results
Solvers
Problems
Studies

Overall Results


Solver
Borda
Firsts


RandomSampler
0
0


TPESampler
2
1


TPESampler with MedianPruner
2
1


TPESampler with SuccessiveHalvingPruner
4
2


Individual Results
(1) Problem: HPO-Bench-Parkinson


Ranking
Solver
Best (avg +- sd)
AUC (avg +- sd)
Elapsed (avg +- sd)


1
TPESampler with MedianPruner (study)
0.010070 +- 0.004299
0.803 +- 0.299
132.749 +- 15.810


1
TPESampler (study)
0.009882 +- 0.003529
0.945 +- 0.354
2.775 +- 0.349


1
TPESampler with SuccessiveHalvingPruner (study)
0.011867 +- 0.003252
0.879 +- 0.279
26.767 +- 10.526


4
RandomSampler (study)
0.015558 +- 0.005880
1.304 +- 0.444
0.614 +- 0.083


(2) Problem: HPO-Bench-Protein


Ranking
Solver
Best (avg +- sd)
AUC (avg +- sd)
Elapsed (avg +- sd)


1
TPESampler with SuccessiveHalvingPruner (study)
0.231011 +- 0.009908
17.172 +- 0.744
21.536 +- 8.320


2
TPESampler with MedianPruner (study)
0.232737 +- 0.010008
17.874 +- 0.884
131.829 +- 12.683


2
TPESampler (study)
0.236102 +- 0.014100
18.529 +- 1.390
2.781 +- 0.370


4
RandomSampler (study)
0.255230 +- 0.016135
19.817 +- 1.112
0.624 +- 0.080


Solvers
ID: fba765577d406565f676869bca11c65117476c35f2dbf3d0fc5d9025ffda9a43
recipe:
{
  "name": "RandomSampler",
  "optuna": {
    "sampler": "random",
    "pruner": "nop"
  }
}
specification:
{
  "name": "RandomSampler",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}
ID: 870a5088fd88d2fb78e09b23b5a123f76e562a194c038eccd5b5e51bd95931cf
recipe:
{
  "name": "TPESampler",
  "optuna": {
    "pruner": "nop"
  }
}
specification:
{
  "name": "TPESampler",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}
ID: 679fa4abbe034c1371f96080e14c0796e5257021868dc8e7c666cc25929bca75
recipe:
{
  "name": "TPESampler with MedianPruner",
  "optuna": {
    "median_warmup_steps": 4
  }
}
specification:
{
  "name": "TPESampler with MedianPruner",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}
ID: cc04a6c5ae0a5d76bbfd007aa9f3ca55b672b949760b3f92a4b390bfc52195c0
recipe:
{
  "name": "TPESampler with SuccessiveHalvingPruner",
  "optuna": {
    "pruner": "asha",
    "asha_min_resource": 4
  }
}
specification:
{
  "name": "TPESampler with SuccessiveHalvingPruner",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.0.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}
Problems
ID: 4f3ab413a790160a528bbc3790c3a332868e4bd2268005108644e09e2e3968b8
recipe:
{
  "hpobench": {
    "dataset": "fcnet_parkinsons_telemonitoring_data.hdf5"
  }
}
specification:
{
  "name": "HPO-Bench-Parkinson",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}
ID: 81bde99ec2ab16438bf4c7145468df6993c53913a034bd3ee4ee65fdbf1a3bcf
recipe:
{
  "hpobench": {
    "dataset": "fcnet_protein_structure_data.hdf5"
  }
}
specification:
{
  "name": "HPO-Bench-Protein",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}
Studies
ID: 12e94aa3b91bc181e091a67a1725adbc21d6f6fc838f8eaaa1a4fb1d3b3cbadf

problem: HPO-Bench-Parkinson
solver: RandomSampler
budget: 80
repeats: 30
concurrency: 1

ID: 431414b3e295d2647ebca78d922daa39b5091de27f73d648986e5589a8c4a222

problem: HPO-Bench-Parkinson
solver: TPESampler
budget: 80
repeats: 30
concurrency: 1

ID: 09968fca2def713da7c876682aa0fee4ab350fdec71388d2890bb8ac4106168e

problem: HPO-Bench-Parkinson
solver: TPESampler with MedianPruner
budget: 80
repeats: 30
concurrency: 1

ID: 4ddb19c7790a16c70fa771b63ff831577b3db6aef7c12fdb75d319b8a6457922

problem: HPO-Bench-Parkinson
solver: TPESampler with SuccessiveHalvingPruner
budget: 80
repeats: 30
concurrency: 1

ID: 28a919018ae3856bd6ba62cb273fabd521827b078b2b482ccaf2db1acda6b977

problem: HPO-Bench-Protein
solver: RandomSampler
budget: 80
repeats: 30
concurrency: 1

ID: d8e15e76a9cd34a3ca46921edc1d6afdda0c3f1a0dc3b3060ec0bf445576aebe

problem: HPO-Bench-Protein
solver: TPESampler
budget: 80
repeats: 30
concurrency: 1

ID: feb526ce52a481282f4074285bc3c6ddcecccb6b2902325c162a9d750697bec8

problem: HPO-Bench-Protein
solver: TPESampler with MedianPruner
budget: 80
repeats: 30
concurrency: 1

ID: ec52b06a2a3e5529ceeab40e6e7ac97cda7afd3621ee62901b9d29c105348298

problem: HPO-Bench-Protein
solver: TPESampler with SuccessiveHalvingPruner
budget: 80
repeats: 30
concurrency: 1
Solver	Borda	Firsts
RandomSampler	0	0
TPESampler	2	1
TPESampler with MedianPruner	2	1
TPESampler with SuccessiveHalvingPruner	4	2
Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	TPESampler with MedianPruner (study)	0.010070 +- 0.004299	0.803 +- 0.299	132.749 +- 15.810
1	TPESampler (study)	0.009882 +- 0.003529	0.945 +- 0.354	2.775 +- 0.349
1	TPESampler with SuccessiveHalvingPruner (study)	0.011867 +- 0.003252	0.879 +- 0.279	26.767 +- 10.526
4	RandomSampler (study)	0.015558 +- 0.005880	1.304 +- 0.444	0.614 +- 0.083
Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	TPESampler with SuccessiveHalvingPruner (study)	0.231011 +- 0.009908	17.172 +- 0.744	21.536 +- 8.320
2	TPESampler with MedianPruner (study)	0.232737 +- 0.010008	17.874 +- 0.884	131.829 +- 12.683
2	TPESampler (study)	0.236102 +- 0.014100	18.529 +- 1.390	2.781 +- 0.370
4	RandomSampler (study)	0.255230 +- 0.016135	19.817 +- 1.112	0.624 +- 0.080