This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pytorch_lightning as pl | |
from pytorch_lightning.loggers import CSVLogger, WandbLogger | |
from pathlib import Path | |
import pandas as pd | |
def read_metrics_csv(metrics_file_path): | |
df_hist = pd.read_csv(metrics_file_path) | |
df_hist["epoch"] = df_hist["epoch"].ffill() | |
df_histe = df_hist.set_index("epoch").groupby("epoch").mean() | |
return df_histe |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Modified from https://github.com/timeseriesAI/tsai/blob/main/tsai/models/InceptionTimePlus.py | |
I've added my favourite modifications | |
- causal padding in the conv | |
- causal padding the max pool | |
- coord=True by default | |
- dilation so it can work over larger context lengths | |
- specify kernels to be [3, 13, 39] instead of the default [13, 19, 39] or whatver. This makes sure we have a small kernel as well which can help performance. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from typing import Callable | |
from loguru import logger | |
import torch | |
from torch import nn | |
def convert_layers(model: nn.Module, original: nn.Module, value: bool): | |
""" | |
Turn dropout on |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Sometimes we want to use quantiles loss in machine learning, but the outputs are not ordered. This is sometimes called the quantile crossover problem. | |
Surely it would help to impose the constraint that the quantiles must be ordered? | |
What's the best way to do this? | |
Well it seems me that we should predict differences from the median, | |
and apply a softplus to make sure the differences are only in one direction. | |
Note this will NOT work for very small target values. Because we are using a softplus the model must output very large | |
logits to get very small numbers. This means it will have difficulty with small y values. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# autoreload import your package | |
%load_ext autoreload | |
%autoreload 2 | |
## secrets | |
from dotenv import load_dotenv | |
load_dotenv() # take environment variables from .env. | |
## numeric, plotting | |
import numpy as np |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
DILATE_fast | |
DILATE cuda implementation | |
DILATE: DIstortion Loss with shApe and tImE | |
WARNING: | |
- does NOT work for larger batch sizes | |
- if you're dumpster diving for loss functions in other peoples dirty gists, then you deserve what you get |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# load mask | |
maskp = '../data/raw/NBIA/ProstateX/PROSTATEx_masks/Files/lesions/Masks/For_dcm2niix_files/ADC/\ | |
ProstateX-0142-Finding3-ep2d_diff_tra_DYNDIST_ADC0_ROI.nii.gz' | |
y = nib.load(ypath).get_fdata() | |
# 1. reverse what nii2dcm did | |
y = y.transpose((1, 0, 2))[::-1, :, ::-1] | |
# load dicom |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from IPython.display import display | |
import pandas as pd | |
def pdshow(df): | |
""" | |
This shows a pandas dataframe in full/ | |
Also consider .to_html() and https://pbpython.com/dataframe-gui-overview.html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Implements on disk caching of transformed dataframes | |
Used on a function that returns a single pandas object, | |
this decorator will execute the function, cache the dataframe as a pickle | |
file using the hash of function and subdirectory, and the arguments and filename. | |
The next time the function runs, if the hashes match what is on disk, the decoratored function will simply load and return | |
the pickled pandas object. | |
This can result in speedups of 10 to 100 times, or more, depending on the | |
complexity of the function that creates the dataframe. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
read dicom header and cache | |
url https://gist.github.com/wassname/a2cdf0b9b511f8a4769cbbe040a87900 | |
""" | |
from diskcache import Cache, JSONDisk | |
import pandas as pd | |
from tqdm.contrib.concurrent import thread_map, process_map | |
import logging | |
from pathlib import Path | |
from functools import partial |