Skip to content

Instantly share code, notes, and snippets.

@LostFan123

LostFan123/question.md

Last active Nov 19, 2019
Embed
What would you like to do?
How to prevent rectified images to be cropped in OpenCV?

This question was originally posted on Stack Overflow by the following link: https://stackoverflow.com/questions/50585255/how-to-prevent-rectified-images-to-be-cropped-in-opencv
Unfortunately, most probably due to my high activity in content moderation, it got several "revenge downvotes" and it was automatically deleted by the system. This question got around 500 views in 1 year, so I assume other people also encounter this issue, hence I decided to save a copy here.
While there is still no answer to the problem of cropping, I think my attempt to solve it which is presented in an edit to the question will be helpful for future readers.

Let's take the following pair of consequent aerial images and do the image rectification (code below):

query.jpg query

train.jpg enter image description here

import cv2
import matplotlib.pyplot as plt
import numpy as np


def main():
    query_image = cv2.imread(filename='query.jpg', flags=cv2.CV_8U)
    train_image = cv2.imread(filename='train.jpg', flags=cv2.CV_8U)

    detector = cv2.xfeatures2d.SURF_create()
    query_keypoints, query_descriptors = detector.detectAndCompute(query_image,
                                                                   mask=None)
    train_keypoints, train_descriptors = detector.detectAndCompute(train_image,
                                                                   mask=None)
    matcher = cv2.BFMatcher_create(normType=cv2.NORM_L1, crossCheck=True)
    matches = matcher.match(queryDescriptors=query_descriptors,
                            trainDescriptors=train_descriptors)
    matches = sorted(matches, key=lambda x: x.distance)

    query_points = np.array([query_keypoints[match.queryIdx].pt
                             for match in matches])
    train_points = np.array([train_keypoints[match.trainIdx].pt
                             for match in matches])

    fundamental_matrix, _ = cv2.findFundamentalMat(points1=query_points,
                                                   points2=train_points,
                                                   method=cv2.FM_RANSAC,
                                                   param1=0.1)
    _, query_homography, _ = cv2.stereoRectifyUncalibrated(
        points1=query_points,
        points2=train_points,
        F=fundamental_matrix,
        imgSize=query_image.shape[::-1])
    map_1, map_2 = cv2.initUndistortRectifyMap(R=query_homography,
                                               cameraMatrix=np.eye(3, 3),
                                               distCoeffs=np.zeros(5),
                                               newCameraMatrix=np.eye(3, 3),
                                               size=query_image.shape[::-1],
                                               m1type=cv2.CV_16SC2)
    plt.imshow(cv2.remap(src=query_image,
                         map1=map_1,
                         map2=map_2,
                         interpolation=cv2.INTER_LINEAR))
    plt.show()


if __name__ == '__main__':
    main()

The resulting rectified query image will look like this:
enter image description here


Question:
How can I avoid cropping the result and fit the whole image in the figure?


What I tried:

  • Increasing size parameter of cv2.initUndistortRectifyMap. Changing it by tuple(2*x for x in query_image.shape[::-1]) will result in the following image: enter image description here

  • Doing manipulations with map_1:

     map_1[:, :, 0] -= 1000
     map_1[:, :, 1] += 1000
    

    This will result in the following image:
    enter image description here

This hack kinda works but I don't want so much black space around the image. Ideally, I want the borders of the image to touch the borders of the figure. I feel like I miss some kind of functionality in OpenCV which would allow the new image not to be cropped, and the sizes of the new figure to be adjusted taking into account rotation of the image. Or maybe I need to extract and use somehow information about transformation from map_1 and map_2...


Edit:
Replying to a comment by @Micka.

change the new camera matrix. There is a function getOptimalNewCameraMatrix according to docs. You can't prevent the black parts around a distortion corrected image, so you can either live with the border or crop the outer parts away. See https://stackoverflow.com/a/21479072/7851470 for one approach.

In my real code I have the following camera matrix and distortion coefficients:

CAMERA_MATRIX = np.array([
    [2425.51170203134142866475, 0, 2035.60834479390314299962],
    [0, 2425.51170203134142866475, 1512.81389897734607075108],
    [0, 0, 1]])

DISTORTION_COEFFICIENTS = np.array([-0.00470114123536235617,
                                    -0.00149541744410850905,
                                    -0.00024420109077626909,
                                    -0.00003711484246148531,
                                    0.00020459075700470246])

From there I calculate new camera matrix like this:

new_camera_matrix, _ = cv2.getOptimalNewCameraMatrix(
    cameraMatrix=CAMERA_MATRIX,
    distCoeffs=DISTORTION_COEFFICIENTS,
    imageSize=query_image.shape[::-1],
    alpha=0,
    centerPrincipalPoint=0)

Then, I use it like this to calculate map_1 and map_2:

rotation_matrix = np.linalg.inv(CAMERA_MATRIX) @ query_homography @ CAMERA_MATRIX
map_1, map_2 = cv2.initUndistortRectifyMap(R=rotation_matrix,
                                           cameraMatrix=CAMERA_MATRIX,
                                           distCoeffs=DISTORTION_COEFFICIENTS,
                                           newCameraMatrix=new_camera_matrix,
                                           size=query_image.shape[::-1],
                                           m1type=cv2.CV_16SC2)

Result is equivalent to the first rectified image I provided above.

And regarding your link, I don't see how it can help me. There they want to crop the image. I, on other hand, don't want to lose any information from the image


Edit Nº2:
Probably, it's worth noting that in the end it is required that both rectified images have to be not only fit in the figure but to be aligned as well. Like here (image was taken from A short tutorial on image rectification by Du Huynh):
enter image description here Currently I am getting the following:
enter image description here


Edit Nº3:
I tried to extract information about how much I should resize the figure and how much I should move the rectified image from inverse maps:

query_rotation_matrix = (np.linalg.inv(CAMERA_MATRIX)
                         @ query_homography
                         @ CAMERA_MATRIX)
inverse_query_rotation_matrix = np.linalg.inv(query_rotation_matrix)

train_rotation_matrix = (np.linalg.inv(CAMERA_MATRIX)
                         @ train_homography
                         @ CAMERA_MATRIX)
inverse_train_rotation_matrix = np.linalg.inv(train_rotation_matrix)

inv_query_map_1, inv_query_map_2 = cv2.initUndistortRectifyMap(
    R=inverse_query_rotation_matrix,
    cameraMatrix=CAMERA_MATRIX,
    distCoeffs=DISTORTION_COEFFICIENTS,
    newCameraMatrix=new_camera_matrix,
    size=query_image.shape[::-1],
    m1type=cv2.CV_16SC2)

inv_train_map_1, inv_train_map_2 = cv2.initUndistortRectifyMap(
    R=inverse_train_rotation_matrix,
    cameraMatrix=CAMERA_MATRIX,
    distCoeffs=DISTORTION_COEFFICIENTS,
    newCameraMatrix=new_camera_matrix,
    size=query_image.shape[::-1],
    m1type=cv2.CV_16SC2)

From here we can get the size of the new rectified image:

extended_x_size = (inv_query_map_1[:, :, 1].max()
                   - inv_query_map_1[:, :, 1].min())
extended_y_size = (inv_query_map_1[:, :, 0].max()
                   - inv_query_map_1[:, :, 0].min())

And how much we should shift the images:

x_shift = np.abs(inv_query_map_1[:, :, 1].min())
y_shift = np.abs(inv_query_map_1[:, :, 0].min())

Also, taking into account that the rectified train image will be shifted relative to the rectified query image, we need to add some more space:

delta_y = max(0, inv_train_map_1[:, :, 0].max() + y_shift - extended_y_size)

Using new dimensions to create maps:

query_map_1, query_map_2 = cv2.initUndistortRectifyMap(
    R=query_rotation_matrix,
    cameraMatrix=CAMERA_MATRIX,
    distCoeffs=DISTORTION_COEFFICIENTS,
    newCameraMatrix=new_camera_matrix,
    size=(extended_y_size + delta_y, extended_x_size),
    m1type=cv2.CV_16SC2)
train_map_1, train_map_2 = cv2.initUndistortRectifyMap(
    R=train_rotation_matrix,
    cameraMatrix=CAMERA_MATRIX,
    distCoeffs=DISTORTION_COEFFICIENTS,
    newCameraMatrix=new_camera_matrix,
    size=(extended_y_size + delta_y, extended_x_size),
    m1type=cv2.CV_16SC2)

And now not the most obvious part. When applying shifts to these maps, image will move not along the axis of the figure but along the axis of the image itself. Applying the shifts:

x_1 = inv_query_map_1[0, 0, 1]
x_2 = inv_query_map_1[0, -1, 1]
y_1 = inv_query_map_1[0, 0, 0]
y_2 = inv_query_map_1[0, -1, 0]
d_1 = x_2 - x_1
d_2 = y_2 - y_1
phi = np.arctan(d_1 / d_2)
psi = phi + np.arctan(y_shift / x_shift)
ro = np.sqrt(x_shift**2 + y_shift**2)

x_shift_image = ro * np.cos(psi)
y_shift_image = ro * np.sin(psi)

query_map_1[:, :, 1] -= x_shift_image.astype(int)
query_map_1[:, :, 0] -= y_shift_image.astype(int)
train_map_1[:, :, 1] -= x_shift_image.astype(int)
train_map_1[:, :, 0] -= y_shift_image.astype(int)

Finally, plotting the results:

rectified_query_image = cv2.remap(src=query_image,
                                  map1=query_map_1,
                                  map2=query_map_2,
                                  interpolation=cv2.INTER_LINEAR)
rectified_train_image = cv2.remap(src=train_image,
                                  map1=train_map_1,
                                  map2=train_map_2,
                                  interpolation=cv2.INTER_LINEAR)

plt.imshow(rectified_query_image)
plt.show()
plt.imshow(rectified_train_image)
plt.show()

rectified_query.jpg
enter image description here
rectified_train.jpg
enter image description here
epipolar_lines.jpg enter image description here

This looks pretty good at first glance, but unfortunately my disparity map becomes broken...

disparity_before.jpg
enter image description here
disparity_after.jpg
enter image description here


Edit Nº4:
I was thinking that maybe the problem with the broken disparity could be because of different inclinations of the rectified images. If so, then shifts should be calculated separately for each image, like this:

from typing import Tuple


def shift_distance(map_: np.ndarray,
                   *,
                   x_shift: int,
                   y_shift: int) -> Tuple[int, int]:
    """
    Converts shifts from figure coordinate system 
    to image coordinate system
    """
    x_1 = map_[0, 0, 1]
    x_2 = map_[0, -1, 1]
    y_1 = map_[0, 0, 0]
    y_2 = map_[0, -1, 0]
    d_1 = x_2 - x_1
    d_2 = y_2 - y_1
    phi = np.arctan(d_1 / d_2)
    psi = phi + np.arctan(y_shift / x_shift)
    ro = np.sqrt(x_shift ** 2 + y_shift ** 2)

    return ro * np.cos(psi), ro * np.sin(psi)

x_shift_image, y_shift_image = shift_distance(inv_query_map_1,
                                              x_shift=x_shift,
                                              y_shift=y_shift)
query_map_1[:, :, 1] -= x_shift_image.astype(int)
query_map_1[:, :, 0] -= y_shift_image.astype(int)

x_shift_image, y_shift_image = shift_distance(inv_train_map_1,
                                              x_shift=x_shift,
                                              y_shift=y_shift)
train_map_1[:, :, 1] -= x_shift_image.astype(int)
train_map_1[:, :, 0] -= y_shift_image.astype(int)

I hoped that this would fix the disparity image, but it doesn't, for some reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.