Skip to content

Instantly share code, notes, and snippets.

@sile

sile/_main.md

Last active December 26, 2019 04:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sile/d9140bf45609a773f6edfbff94c5934b to your computer and use it in GitHub Desktop.
Save sile/d9140bf45609a773f6edfbff94c5934b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
_main.md

Benchmark Result Report

  • Report ID: 4ba0c3ff34e08ddaac04387c13628701e21fd683a713b496b3f78c3d541beac9
  • Kurobako Version: 0.1.3
  • Number of Solvers: 3
  • Number of Problems: 4
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Hyperband (n_brackets=3) 1 4
Hyperband (n_brackets=4) 1 4
Hyperband (n_brackets=5) 0 3

Individual Results

(1) Problem: HPO-Bench-Parkinson

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.010274 +- 0.002866 0.593 +- 0.168 77.627 +- 4.032
1 Hyperband (n_brackets=3) (study) 0.010998 +- 0.004389 0.610 +- 0.216 78.957 +- 4.106
1 Hyperband (n_brackets=5) (study) 0.010452 +- 0.003867 0.683 +- 0.214 76.188 +- 3.679

(2) Problem: HPO-Bench-Naval

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.000088 +- 0.000083 0.015 +- 0.016 78.439 +- 3.745
1 Hyperband (n_brackets=3) (study) 0.000113 +- 0.000151 0.023 +- 0.034 81.952 +- 5.304
1 Hyperband (n_brackets=5) (study) 0.000095 +- 0.000095 0.023 +- 0.026 76.634 +- 3.675

(3) Problem: HPO-Bench-Protein

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.229968 +- 0.008488 10.355 +- 0.517 77.086 +- 4.335
1 Hyperband (n_brackets=3) (study) 0.229168 +- 0.006514 10.320 +- 0.377 78.119 +- 4.569
3 Hyperband (n_brackets=5) (study) 0.235814 +- 0.011201 10.631 +- 0.555 75.984 +- 3.760

(4) Problem: HPO-Bench-Slice

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Hyperband (n_brackets=4) (study) 0.000360 +- 0.000145 0.021 +- 0.008 77.230 +- 5.050
1 Hyperband (n_brackets=3) (study) 0.000315 +- 0.000083 0.018 +- 0.006 79.189 +- 5.027
1 Hyperband (n_brackets=5) (study) 0.000310 +- 0.000119 0.022 +- 0.011 75.585 +- 5.596

Solvers

ID: e7f3a145b144a9a3b2bc99b714e4e9f698e8a67f82ece410de77e2abd496c4de

recipe:

{
  "name": "Hyperband (n_brackets=3)",
  "command": {
    "path": "python",
    "args": [
      "hyperband-solver.py",
      "--n-brackets",
      "3"
    ]
  }
}

specification:

{
  "name": "Hyperband (n_brackets=3)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=0.19.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: a04abee366e2d572f34e07fbf80e1796ab42e10a0872755487a1de10a9946a5b

recipe:

{
  "name": "Hyperband (n_brackets=4)",
  "command": {
    "path": "python",
    "args": [
      "hyperband-solver.py",
      "--n-brackets",
      "4"
    ]
  }
}

specification:

{
  "name": "Hyperband (n_brackets=4)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=0.19.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: f94c12441e4b8915be38df76527f83b9094787327a83115a835e8a20e97385e3

recipe:

{
  "name": "Hyperband (n_brackets=5)",
  "command": {
    "path": "python",
    "args": [
      "hyperband-solver.py",
      "--n-brackets",
      "5"
    ]
  }
}

specification:

{
  "name": "Hyperband (n_brackets=5)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=0.19.0, kurobako-py=0.1.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 880b007e3c61f4aecb7e9b0aa2f9be5fea9f491a076853f68f402769aa254034

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_naval_propulsion_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Naval",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: 445bfa45fdbb8ec6ae6d4dba1909114f3948fa67b47209258db9291480b405b5

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_parkinsons_telemonitoring_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Parkinson",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: add73d4788d7900b34988a8b91cde43e820cac99f9e354e1e71b0ea0be3ef4a6

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_protein_structure_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Protein",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

ID: d88e1704447639bde17f236f7af47f93274d1f02bc8ec66733146ff9cdf50196

recipe:

{
  "hpobench": {
    "dataset": "./fcnet_slice_localization_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Slice",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.3"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

Studies

ID: 1ec54571e58d20e8e37e8f6278873dfbd3e2d394a5bbc6836ac2b46475ea2efc

ID: db7879f140bd6bbd39e5dc3da4d2359977b2590a0b25b1de6ba18538b54bf423

ID: 7e08a69a7d165bf1eb3343c72d6aae203c3a42997161a528a0b7414699aa8ba2

ID: 2d2379ebcdd6c97581709feca19edcf633f0199e4f162bb5d05a5774c5485d75

ID: f2017cf186fd4f7977b8027fda44d7f345ac36bdcb03fd3d8c0420af66c5eb2f

ID: ccf2040fe63eb90c4cdac1ec7f502d24cf602878ad1bc7ace3d9912510397f5d

ID: bf18cc1fdd3930db2a02e8cedd6e01ef9a884f2a0b6db768b0825464eb5a1d29

ID: c3d7e3ac7ef91c2f119ec2b985e5bff193012a5c51d2695f38f8afa2c483de88

ID: 5a1f46e2d15ebf39196563225d9fc43d76756a127e5f4d0815e0ff7db30025f2

ID: 51e26ad115a26c2b7fc086be41a7137036b22b4b5775ed318e6643332db7b3ce

ID: 8610e788ca37c18a6b098df57b392a63fa8da7d9e18a7d173abd209040d044d1

ID: 1f2c8bfe46db290a36a96b630310be009198e2d9305396d21376c1cc21c6c4e8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment