Skip to content

Instantly share code, notes, and snippets.

Takeru Ohta sile

View GitHub Profile
@sile
sile / config.gin
Last active Apr 8, 2020
Example: gin-config with optuna
View config.gin
train.batch_size = 10
train.learning_rate = 0.1
@sile
sile / random.py
Last active Jan 27, 2020
Kurobako blog: random.py
View random.py
# A solver implementation based on Random Search algorithm.
from kurobako import problem
from kurobako import solver
import numpy as np
class RandomSolverFactory(solver.SolverFactory):
def specification(self):
return solver.SolverSpec(name='Random Search')
def create_solver(self, seed, problem):
@sile
sile / sa.py
Created Jan 27, 2020
Kurobako blog: sa.py
View sa.py
# A solver implementation based on Simulated Annealing algorithm.
from kurobako import solver
from kurobako.solver.optuna import OptunaSolverFactory
import optuna
class SimulatedAnnealingSampler(optuna.BaseSampler):
# Please refer to
# https://github.com/optuna/optuna/blob/v1.0.0/examples/samplers/simulated_annealing_sampler.py
# for the implementation.
...
@sile
sile / custom-solvers.sh
Created Jan 27, 2020
Kurobako blog: custom-solvers.sh
View custom-solvers.sh
$ kurobako solver command python random.py > solvers.json
$ kurobako solver command python sa.py >> solvers.json
$ kurobako studies --problems $(cat problems.json) --solvers $(cat solvers.json) | kurobako run > result.json
@sile
sile / report.md
Created Jan 27, 2020
Kurobako blog: report.md
View report.md

Benchmark Result Report

  • Report ID: ecb7024f149c2653a906f600d048eac2658f94bc2cfc8e50bfd7356c0fb1e29b
  • Kurobako Version: 0.1.3
  • Number of Solvers: 4
  • Number of Problems: 2
  • Metrics Precedence: best value -> AUC

Please refer to ["A Strategy for Ranking Optimizers using Multiple Criteria"][Dewancker, Ian, et al., 2016] for the ranking strategy used in this report.

@sile
sile / basic-benchmark.sh
Last active Apr 15, 2020
kurobako blog: basic-benchmark.sh
View basic-benchmark.sh
# 1. Download kurobako binary.
$ curl -L https://github.com/sile/kurobako/releases/download/0.1.4/kurobako-0.1.4.linux-amd64 -o kurobako
$ chmod +x kurobako && sudo mv kurobako /usr/local/bin/
# 2. Download the data file for HPOBench (note that the file size is about 700MB).
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/
# 3. Specify problems used in this benchmark.
#
View _main.md

Benchmark Result Report

  • Report ID: 4ba0c3ff34e08ddaac04387c13628701e21fd683a713b496b3f78c3d541beac9
  • Kurobako Version: 0.1.3
  • Number of Solvers: 3
  • Number of Problems: 4
  • Metrics Precedence: best value -> AUC

Please refer to ["A Strategy for Ranking Optimizers using Multiple Criteria"][Dewancker, Ian, et al., 2016] for the ranking strategy used in this report.

View _main.md

The aim of this benchmark is to compare the performances of Optuna's pruners (i.e., NopPruner, MedianPruner, SuccessiveHalvingPruner and the ongoing HyperbandPruner). All of the pruners were used by the default settings in this benchmark.

The commands to execute this benchmark.

// (1) Downloads `kurobako` (BBO benchmark tool) binary.
$ curl -L https://github.com/sile/kurobako/releases/download/0.1.3/kurobako-0.1.3.linux-amd64 -o kurobako
$ chmod +x kurobako && sudo mv kurobako /usr/local/bin/

// (2) Downloads data files of HPOBench. (notice that the total size is over 700MB)
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
@sile
sile / kurobako-usage.md
Last active Sep 14, 2019
kurobakoの使い方メモ
View kurobako-usage.md
$ cargo install kurobako
// or (only linux)
$ wget https://github.com/sile/kurobako/releases/download/0.0.15/kurobako-0.0.15.linux-amd64 -o kurobako && chmod +x kurobako
// or 
$ git clone git://github.com/sile/kurobako.git && cd kurobako && git checkout 0.0.14 && cargo install --path .

// 独自サンプラの場合
$ kurobako benchmark --problems (kurobako problem-suite sigopt auc) --solvers (kurobako solver command -- python3 /tmp/optuna_solver_example.py ) --budget 100 --iterations 10 | kurobako run > /tmp/sigopt-my-sampler.json
@sile
sile / multi_process_storage.py
Last active Aug 29, 2019
An Optuna storage that uses multiprocessing module for inter process communication
View multi_process_storage.py
import copy
from datetime import datetime
import multiprocessing
from optuna import distributions # NOQA
from optuna.storages import base
from optuna.storages.base import DEFAULT_STUDY_NAME_PREFIX
from optuna import structs
from optuna import type_checking
You can’t perform that action at this time.