DINO Video inference

video_inference.ipynb

Been getting a few errors while trying to run this notebook.

I think the args.input_path and args.output_dir should be changed to these values else there would be an error thrown during inference.
I've also been getting another error while running inference at predict_video(args) seems to throw the following error for me. Would be great if I could have some insight as to correct this.

[FIX] Error: 'VisionTransformer' object has no attribute 'forward_selfattention'

Change the following code in line 21 of def predict_video(args):
attentions = model.forward_selfattention(img.cuda()) -> attentions = model.get_last_selfattention(img.cuda())

Thanks 👍 !! Finally got it working.
Any idea how I could get the attention heat-maps with a transparent background instead of the all black opaque background such that it feels like an overlay on the input video? I essentially want to get an output like so.

Good evening I would like to know if you can help me with this error, I don't know how to call the "vision_transformer" module. I really appreciate it

Good evening I would like to know if you can help me with this error, I don't know how to call the "vision_transformer" module. I really appreciate it

Did you resolve this? If not, be sure to run all of the cells in order and change into the DINO directory beforehand with "%cd dino/" and it might be helpful to import it as "vits" (import vision_transformer as vits) as that is how it is called in the notebook.

Good evening I would like to know if you can help me with this error, I don't know how to call the "vision_transformer" module. I really appreciate it

Did you resolve this? If not, be sure to run all of the cells in order and change into the DINO directory beforehand with "%cd dino/" and it might be helpful to import it as "vits" (import vision_transformer as vits) as that is how it is called in the notebook.

Yes, thank you very much Felipe, as you said I had missed running the% cd dino / code. I was finally able to get to the end of the code but again I have been stuck since I got a new warning that does not let me continue. Please if you can help me with this.

[FIX] File not found '../output/attn-image-%03d.jpg'

Actually there is a typo in there, it should be:
!ffmpeg -framerate 60 -i ../out/attn-img-%03d.jpg ../output.mp4

Good evening I would like to know if you can help me with this error, I don't know how to call the "vision_transformer" module. I really appreciate it

Did you resolve this? If not, be sure to run all of the cells in order and change into the DINO directory beforehand with "%cd dino/" and it might be helpful to import it as "vits" (import vision_transformer as vits) as that is how it is called in the notebook.

Yes, thank you very much Felipe, as you said I had missed running the% cd dino / code. I was finally able to get to the end of the code but again I have been stuck since I got a new warning that does not let me continue. Please if you can help me with this.

Has anyone tried using the other models? I get several size mismatch errors

Has anyone tried using the other models? I get several size mismatch errors

I have not trained it with another network, but I will try and tell you.

Can anyone help me out here? I've tried changing args.input_path to "../input/" and args.output_dir to "../output/" as suggested above but keep running into the following error:

predict_video(args)

IsADirectoryError Traceback (most recent call last)
in ()
----> 1 predict_video(args)

in predict_video(args)
1 def predict_video(args):
2 for frame in sorted(os.listdir(args.image_path)):
----> 3 with open(os.path.join(args.image_path, frame), 'rb') as f:
4 img = Image.open(f)
5 img = img.convert('RGB')

IsADirectoryError: [Errno 21] Is a directory: '../input/.ipynb_checkpoints'