detr_finetune.md

Get pretrained weights:

wget https://dl.fbaipublicfiles.com/detr/detr-r50-e632da11.pth

Remove class weights

checkpoint = torch.load("detr-r50-e632da11.pth", map_location='cpu')
del checkpoint["model"]["class_embed.weight"]
del checkpoint["model"]["class_embed.bias"]
torch.save(checkpoint,"detr-r50_no-class-head.pth")

and make sure to set non-strict weight loading in main.py

model_without_ddp.load_state_dict(checkpoint['model'], strict=False)

Your dataset should ideally be in the COCO-format. Make your own data-builder (alternatively rename your train/valid/annotation file to match the COCO Dataset) In datasets.coco.py add:

def build_your_dataset(image_set, args):
    root = Path(args.coco_path)
    assert root.exists(), f'provided COCO path {root} does not exist'
    mode = 'instances'
    PATHS = {
        "train": (root / "train", root / "annotations" / f'train.json'),
        "val": (root / "valid", root / "annotations" / f'valid.json'),
    }

    img_folder, ann_file = PATHS[image_set]
    dataset = CocoDetection(img_folder, ann_file, transforms=make_coco_transforms(image_set), return_masks=args.masks)
    return dataset

In datasets.__init__.py add your builder as an option:

def build_dataset(image_set, args):
    if args.dataset_file == 'coco':
        return build_coco(image_set, args)
    if args.dataset_file == 'your_dataset':
        return build_your_dataset(image_set, args)
    [...]

And lastly define how many classes you have in models.detr.py

def build(args):
    [...]
    if args.dataset_file == 'your_dataset': num_classes = 4
    [...]

Run your model (example): python main.py --dataset_file your_dataset --coco_path data --epochs 50 --lr=1e-4 --batch_size=2 --num_workers=4 --output_dir="outputs" --resume="detr-r50_no-class-head.pth"

Hi @dsshean I have done the same mentioned above and I am still getting a single bounding box in the same position for all the image. I have trained the model with custom dataset with class label of 1. Have you trained with custom dataset and then using the same collab notebook example for inference?

I've been encountering the same issue for a while. Yesterday I just figured out the correct way to load my custom DETR model, which came exactly for the deadline of my project later in the day :))). Here is the colab notebook with some impressing result. The DETR class I use to load all matched keys lies in hubconfig.py on the original detr repositoy. Hope this help
I saw your notebook and thanks for sharing it, I am wondering how I can find the datasets used in the notebook?

@quangdaist123 How many epochs did you train for using the dataset you've provided in your colab notebook? Thanks in advance.

Hi @quangdaist123! I cannot seem to access the colab notebook you provided for custom dataset training. It says the file is not accessible. Can you please make it accessible again? Thanks in advance.

Hi @dsshean I have done the same mentioned above and I am still getting a single bounding box in the same position for all the image. I have trained the model with custom dataset with class label of 1. Have you trained with custom dataset and then using the same collab notebook example for inference?

I've been encountering the same issue for a while. Yesterday I just figured out the correct way to load my custom DETR model, which came exactly for the deadline of my project later in the day :))). Here is the colab notebook with some impressing result. The DETR class I use to load all matched keys lies in hubconfig.py on the original detr repositoy. Hope this help
I saw your notebook and thanks for sharing it, I am wondering how I can find the datasets used in the notebook?

@quangdaist123 How many epochs did you train for using the dataset you've provided in your colab notebook? Thanks in advance.

I trained the model over 200 epochs but It stopped getting better result at 70th epoch or 80th

Hi @quangdaist123! I cannot seem to access the colab notebook you provided for custom dataset training. It says the file is not accessible. Can you please make it accessible again? Thanks in advance.

I checked the link and didn't see any problem. If you want a direct download link to the note book, I just created one here

Thank you @quangdaist123. The download link is working perfectly fine :)

Hi, I am trying to train on a custom dataset with 38K train images, 6 classes. I fine-tuned the ResNet detr-r50 model for about 15 epochs and the MAP remains at zero. What are the recommended numbers of epochs to train with this dataset?
Since I have just one GPU, training takes about an hour for each epoch and I wanted to make sure I got other parameters right before I train it for longer.

Thanks!

can you please share the inference script for custom trained checkpoint.path for prediction ??
Thanks in advance !

Check it

https://github.com/woctezuma/finetune-detr/blob/master/finetune_detr.ipynb

It works for me.

Hello,
I had made an attempt to change the num_queries to 500 as images have approximately 450 objects.
to which i received following error.

Traceback (most recent call last):
File "main.py", line 248, in
main(args)
File "main.py", line 178, in main
model_without_ddp.load_state_dict(checkpoint['model'], strict=False)
File "/home/rsharma/git/detr/.venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DETR:
size mismatch for query_embed.weight: copying a param with shape torch.Size([100, 256]) from checkpoint, the shape in current model is torch.Size([500, 256]).

Can anybody please help me with this ??

Maybe:

If you're fine-tuning, I don't recommend changing the number of queries on the fly, it is extremely unlikely to work out of the box. In this case you're probably better off retraining from scratch (you can change the --num_queries arg from our training script).

facebookresearch/detr#9 (comment)

@woctezuma
not able to resolve.

Hi,
if you change your number of queries, unfortunately, you will have to pretty much train from scratch (except for the resnet backbone). You cannot use transformer weights with num_queries=100for a transformer with 500, same goes for class_embed and bbox_embed. Basically retraining of everything except for the backbone is required. Sorry and good luck.

@m-klasen thanks for your quick response,
I am having a small dataset ~2k images having 450 max objects if I am not using the transfer learning results are showing 0mAP (no accuracy) with default hyperparametrs, can you suggest something so that I can be able to predict 500 abjects per image with small dataset ??

Transformers are notoriously difficult to train, take a long time to converge (200 Epochs on COCO). Unless you can split your images into smaller subsets which feature >100 detections per crop it is going to be difficult.

just to be clear,

1. by no means I can change the num_queries and use the transfer learning.
2. if I am training from scratch, I can get decent results in 200 epochs (because i see 0 mAP for 300 epochs in this case as well)

See facebookresearch/detr#216 to work around the issue.