To be posted in: https://forums.fast.ai/c/fastai-users/fastai-v2/
Title: Proposed workflow to compare & monitor models using WandbCallback
Content:
Hi,
I’ve been working on WandbCallback for the past few months (with a lot of help from @sgugger) and I'm very excited to show how it works!
This is still in very active development so I’d love all the feedback you have regarding bugs or features.
To use it:
import wandb
from fastai2.callback.wandb import *
# start logging a wandb run
wandb.init() # optional -> wandb.init(project='my_project')
# just add WandbCallback to your learner
learn.fit(..., cbs=WandbCallback())It let you:
- quickly compare models -> I used it to debug and check
GradientAccumulation - watch long training runs on your phone
- make lots of custom graphs or reports pulling data from your runs
You can test it with your own project or this small demo notebook.
When you run it, you will have access to:
-
your metrics in real time where you can customize graphs (example with
fit_one_cycle)
-
computer resources
-
model topology
-
gradients and parameters histograms
-
trained model weights & biases saved online (if using
SaveModelCallback)
Now what I like most is that if you run the notebook several times trying different parameters (batch size, number of epochs, learning rate, GradientAccumulation callback…), then open your project page, you will see that more than 100 parameters have automatically been logged for you from all the fastai functions you used.
Press the little wand on the top right of your runs summary table, reorganize and hide columns you want, and you get a nice comparative summary.
You can easily create graphs to compare runs.
And finally you can use them to create cool reports.
I'd love any feedback you may have and I'm here to help if you have any questions.