Skip to content

Instantly share code, notes, and snippets.

@andreyryabtsev
Last active June 5, 2024 04:56
Show Gist options
  • Save andreyryabtsev/243aa3eefa6e06891dda7b1583d1d08f to your computer and use it in GitHub Desktop.
Save andreyryabtsev/243aa3eefa6e06891dda7b1583d1d08f to your computer and use it in GitHub Desktop.
BackMatting.ipynb
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@camjac251
Copy link

Is this script still working? I get some errors with video processing

Segmentation completes successfully Done: colab_inputs_video/input
But with both handheld and fixed, it fails

Traceback (most recent call last):
  File "test_pre_process_video.py", line 123, in <module>
    back_align = alignImages(back, image,mask)
  File "test_pre_process_video.py", line 38, in alignImages
    h, mask = cv2.findHomography(points1, points2, cv2.RANSAC)
cv2.error: OpenCV(3.4.5) /io/opencv/modules/calib3d/src/ptsetreg.cpp:169: error: (-215:Assertion failed) count >= 0 && count2 == count in function 'run'
CUDA Device: 0
Using video mode
Traceback (most recent call last):
  File "test_background-matting_image.py", line 121, in <module>
    bbox=get_bbox(rcnn,R=bgr_img0.shape[0],C=bgr_img0.shape[1])
  File "/content/Background-Matting/functions.py", line 38, in get_bbox
    x1, y1 = np.amin(where, axis=1)
  File "<__array_function__ internals>", line 6, in amin
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 2746, in amin
    keepdims=keepdims, initial=initial, where=where)
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

@venkat094
Copy link

I too got the same error ! The below error is for image, Not sure where is the issue !

CUDA Device: 0
Using image mode
Traceback (most recent call last):
  File "test_background-matting_image.py", line 46, in <module>
    back_img10=cv2.imread(args.target_back); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

@jrdeco560
Copy link

errors

@andreyryabtsev
Copy link
Author

@camjac251 I think your issue may be similar to senguptaumd/Background-Matting#33 . In order to pre-process hand-held videos, that code tries to find a homography alignment between each frame, which may fail for featureless backgrounds. Unfortunately, the script doesn't report the errors nicely; but maybe you could try a different video, or a fixed cam (no alignment needed) video to see if that works?

@venkat094 Actually your error is different; they look the same because cv2's low level code reports them similarly. I am fairly sure the target background file you are trying to use does not exist. So when cv2.imread opens it it silently reads no data, but when it tries to convert the color space it realizes the image is not actually there.

@camjac251
Copy link

Thank you for the reply. I'm trying it with the handheld sample video this time so hopefully it works.

I did notice an little error on the
Run the matting algorithm. If your video was captured with a fixed camera, comment the first line and uncomment the second before running.
block
could it be made into

!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-hand-held -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/
# !CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-fixed-cam -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/ -b colab_inputs_video/$BM_SOURCE_STILL

@andreyryabtsev
Copy link
Author

@camjac251 good catch, fixed!

@gchoi
Copy link

gchoi commented May 20, 2020

Hi, I have error for the following code:
!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-hand-held -i colab_inputs/input/ -o colab_inputs/output/ -tb colab_inputs/background/0001.png

CUDA Device: 0 Using image mode Traceback (most recent call last): File "test_background-matting_image.py", line 121, in <module> bbox=get_bbox(rcnn,R=bgr_img0.shape[0],C=bgr_img0.shape[1]) File "/content/drive/My Drive/Colab-Notebooks/TensorFlow-Examples/033-Back-Matting/Background-Matting/functions.py", line 38, in get_bbox x1, y1 = np.amin(where, axis=1) File "<__array_function__ internals>", line 6, in amin File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 2746, in amin keepdims=keepdims, initial=initial, where=where) File "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) ValueError: zero-size array to reduction operation minimum which has no identity

@venkat094
Copy link

venkat094 commented May 20, 2020

@camjac251 I think your issue may be similar to senguptaumd/Background-Matting#33 . In order to pre-process hand-held videos, that code tries to find a homography alignment between each frame, which may fail for featureless backgrounds. Unfortunately, the script doesn't report the errors nicely; but maybe you could try a different video, or a fixed cam (no alignment needed) video to see if that works?

@venkat094 Actually your error is different; they look the same because cv2's low level code reports them similarly. I am fairly sure the target background file you are trying to use does not exist. So when cv2.imread opens it it silently reads no data, but when it tries to convert the color space it realizes the image is not actually there.

Thanks for your quick reply !!
I did check twice, the target background file is present ! I have just uploaded only one file !

@andreyryabtsev
Copy link
Author

@andreyryabtsev
Copy link
Author

In general, colab can be quite finicky (I think someone had their file get lost because the runtime restarted); if anyone has issues like these it may help to either (a) run the repository in a more standard environment or (b) add asserts to double check all the files exist and the segmentations + homography transforms don't fail for whatever reason.

@Flock1
Copy link

Flock1 commented May 24, 2020

I too got the same error ! The below error is for image, Not sure where is the issue !

CUDA Device: 0
Using image mode
Traceback (most recent call last):
  File "test_background-matting_image.py", line 46, in <module>
    back_img10=cv2.imread(args.target_back); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

I faced this error because of the following:

When I use the segmentation model, it needs to create a model by the name "maskDL" and there it stores the segmented image. Hence, it's not able to read the segmented image because it's not there.

@tolchx
Copy link

tolchx commented May 25, 2020

Hello, thank you very much for sharing this project, I am having the following error, using video segmentation.
@Flock1 using the online example colab, how do I create a model with the name 'maskDL'?
Using the demo material: #Uncomment and run this if you want to skip uploading your videos (below) and run on the sample videos instead (choose fixed or handheld) It works well

The error I have with my own videos is the following:

CUDA Device: 0
Using video mode
Traceback (most recent call last):
File "test_background-matting_image.py", line 85, in
bg_im0=cv2.imread(os.path.join(data_path, filename.replace('_img','_back'))); bg_im0=cv2.cvtColor(bg_im0,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

Any help is welcome thanks

@Flock1
Copy link

Flock1 commented May 27, 2020

@tolchx, this error happens because there is no image that is being read. That's happening probably because there is no folder or a file. Kindly check if the required folder or the image is present.

@tolchx
Copy link

tolchx commented May 31, 2020

@tolchx, this error happens because there is no image that is being read. That's happening probably because there is no folder or a file. Kindly check if the required folder or the image is present.

Yes, it was that, thank you very much.

folder and python file paths were in other folders,

@tolchx
Copy link

tolchx commented Jun 1, 2020

Hello, I have an error when I run this section of the code:

#!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-hand-held -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/
!CUDA_VISIBLE_DEVICES=0 python test_background-matting_image.py -m real-fixed-cam -i colab_inputs_video/input/ -o colab_inputs_video/output/ -tb colab_inputs_video/background/ -b colab_inputs_video/$BM_SOURCE_STILL

error:


Traceback (most recent call last):
  File "test_background-matting_image.py", line 92, in <module>
    back_img10=cv2.imread(os.path.join(args.target_back,filename.replace('_img.png','.png'))); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

I did the test with several videos, with different amount of frames, 600/900/1000 frames, for example.
But it always stops at 441 and the error appears.

What could be the cause?

In the folder: Background-Matting / colab_inputs_video / inputs, the remaining frames appear, it does not end at frame 441

Thank you

@Flock1
Copy link

Flock1 commented Jun 4, 2020

@tolchx, I'm not sure but I think there is a problem with arguments. Instead of --tb, just use --back. Also, how are your results? Can you share them with me because I'm not getting that good results. I had to do some post-processing.

@tolchx
Copy link

tolchx commented Jun 10, 2020

@tolchx, I'm not sure but I think there is a problem with arguments. Instead of --tb, just use --back. Also, how are your results? Can you share them with me because I'm not getting that good results. I had to do some post-processing.

I had good results with some videos, I'm experimenting with other material, here I uploaded a video, use the algorithm to segment the dancers, and then apply styletransfer https://www.instagram.com/p/CAsK54qgx_9/
I will try the solution you recommend, thanks

@Flock1
Copy link

Flock1 commented Jun 12, 2020

@tolchx, you are using it for something interesting. Can you elaborate on what you are doing? Also, did you make any changes to the code? Because I had to do some post-processing to get good results.

@zbvictory
Copy link

zbvictory commented Sep 23, 2020

@Flock1 Have you got a good result? Can you share some tips? thanks
and I too got the same error

Done: 154/351
Done: 155/351
Traceback (most recent call last):
File "test_background-matting_image.py", line 92, in
back_img10=cv2.imread(os.path.join(args.target_back,filename.replace('_img.png','.png'))); back_img10=cv2.cvtColor(back_img10,cv2.COLOR_BGR2RGB);
cv2.error: OpenCV(3.4.5) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

@Flock1
Copy link

Flock1 commented Sep 24, 2020

@zbvictory, as I mentioned here, this error happens because the folder doesn't have the image you want to work with. Do check the folder you're getting the image from.

My raw results weren't that good. I'll share the changes I made.

@zbvictory
Copy link

@Flock1 Thank you very much for your reply and look forward to your sharing !

@Sicily-F
Copy link

Sicily-F commented Dec 23, 2020

Hi! I have a small query. I am very interesting in background matting as a technique, however, I don't have photos with a separate background with and without the subject - I have photos of birds in a cage, and I wish to remove the background of the cage bars. I used the background photo for each image as the cropped background of the cage, so the photos are not the same size for the segmentation.

I got this error code
CUDA Device: 0 Using image mode

But it didn't seem like the segmentation actually was performed on the 7 examples of background and image that I provided - any advice would be GREATLY appreciated!

My results when I try and plot are [] instead of any images

@Flock1
Copy link

Flock1 commented Dec 29, 2020

@Sicily-F, from what I can imagine this is probably because it's not reading the images. Can you post the whole error? I might be able to figure out where exactly is the problem

@amrzv
Copy link

amrzv commented Feb 4, 2023

This colab is no longer works because tf<2 is no longer supported in google colaboratory.

@saidulu31
Copy link

I'm getting issue when Setting up the environment:

Screenshot 2024-01-16 172758

can u please explain how to solve this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment