This question was originally posted on Stack Overflow by the following link: https://stackoverflow.com/questions/50585255/how-to-prevent-rectified-images-to-be-cropped-in-opencv
Unfortunately, most probably due to my high activity in content moderation, it got several "revenge downvotes" and it was automatically deleted by the system. This question got around 500 views in 1 year, so I assume other people also encounter this issue, hence I decided to save a copy here.
While there is still no answer to the problem of cropping, I think my attempt to solve it which is presented in an edit to the question will be helpful for future readers.
Let's take the following pair of consequent aerial images and do the image rectification (code below):
query.jpg
train.jpg
import cv2
import matplotlib.pyplot as plt
import numpy as np
def main():
query_image = cv2.imread(filename='query.jpg', flags=cv2.CV_8U)
train_image = cv2.imread(filename='train.jpg', flags=cv2.CV_8U)
detector = cv2.xfeatures2d.SURF_create()
query_keypoints, query_descriptors = detector.detectAndCompute(query_image,
mask=None)
train_keypoints, train_descriptors = detector.detectAndCompute(train_image,
mask=None)
matcher = cv2.BFMatcher_create(normType=cv2.NORM_L1, crossCheck=True)
matches = matcher.match(queryDescriptors=query_descriptors,
trainDescriptors=train_descriptors)
matches = sorted(matches, key=lambda x: x.distance)
query_points = np.array([query_keypoints[match.queryIdx].pt
for match in matches])
train_points = np.array([train_keypoints[match.trainIdx].pt
for match in matches])
fundamental_matrix, _ = cv2.findFundamentalMat(points1=query_points,
points2=train_points,
method=cv2.FM_RANSAC,
param1=0.1)
_, query_homography, _ = cv2.stereoRectifyUncalibrated(
points1=query_points,
points2=train_points,
F=fundamental_matrix,
imgSize=query_image.shape[::-1])
map_1, map_2 = cv2.initUndistortRectifyMap(R=query_homography,
cameraMatrix=np.eye(3, 3),
distCoeffs=np.zeros(5),
newCameraMatrix=np.eye(3, 3),
size=query_image.shape[::-1],
m1type=cv2.CV_16SC2)
plt.imshow(cv2.remap(src=query_image,
map1=map_1,
map2=map_2,
interpolation=cv2.INTER_LINEAR))
plt.show()
if __name__ == '__main__':
main()The resulting rectified query image will look like this:
Question:
How can I avoid cropping the result and fit the whole image in the figure?
What I tried:
-
Increasing
sizeparameter ofcv2.initUndistortRectifyMap. Changing it bytuple(2*x for x in query_image.shape[::-1])will result in the following image: -
Doing manipulations with
map_1:map_1[:, :, 0] -= 1000 map_1[:, :, 1] += 1000This will result in the following image:
This hack kinda works but I don't want so much black space around the image. Ideally, I want the borders of the image to touch the borders of the figure. I feel like I miss some kind of functionality in OpenCV which would allow the new image not to be cropped, and the sizes of the new figure to be adjusted taking into account rotation of the image. Or maybe I need to extract and use somehow information about transformation from map_1 and map_2...
Edit:
Replying to a comment by @Micka.
change the new camera matrix. There is a function
getOptimalNewCameraMatrixaccording to docs. You can't prevent the black parts around a distortion corrected image, so you can either live with the border or crop the outer parts away. See https://stackoverflow.com/a/21479072/7851470 for one approach.
In my real code I have the following camera matrix and distortion coefficients:
CAMERA_MATRIX = np.array([
[2425.51170203134142866475, 0, 2035.60834479390314299962],
[0, 2425.51170203134142866475, 1512.81389897734607075108],
[0, 0, 1]])
DISTORTION_COEFFICIENTS = np.array([-0.00470114123536235617,
-0.00149541744410850905,
-0.00024420109077626909,
-0.00003711484246148531,
0.00020459075700470246])
From there I calculate new camera matrix like this:
new_camera_matrix, _ = cv2.getOptimalNewCameraMatrix(
cameraMatrix=CAMERA_MATRIX,
distCoeffs=DISTORTION_COEFFICIENTS,
imageSize=query_image.shape[::-1],
alpha=0,
centerPrincipalPoint=0)
Then, I use it like this to calculate map_1 and map_2:
rotation_matrix = np.linalg.inv(CAMERA_MATRIX) @ query_homography @ CAMERA_MATRIX
map_1, map_2 = cv2.initUndistortRectifyMap(R=rotation_matrix,
cameraMatrix=CAMERA_MATRIX,
distCoeffs=DISTORTION_COEFFICIENTS,
newCameraMatrix=new_camera_matrix,
size=query_image.shape[::-1],
m1type=cv2.CV_16SC2)
Result is equivalent to the first rectified image I provided above.
And regarding your link, I don't see how it can help me. There they want to crop the image. I, on other hand, don't want to lose any information from the image
Edit Nº2:
Probably, it's worth noting that in the end it is required that both rectified images have to be not only fit in the figure but to be aligned as well. Like here (image was taken from A short tutorial on image rectification by Du Huynh):
Edit Nº3:
I tried to extract information about how much I should resize the figure and how much I should move the rectified image from inverse maps:
query_rotation_matrix = (np.linalg.inv(CAMERA_MATRIX)
@ query_homography
@ CAMERA_MATRIX)
inverse_query_rotation_matrix = np.linalg.inv(query_rotation_matrix)
train_rotation_matrix = (np.linalg.inv(CAMERA_MATRIX)
@ train_homography
@ CAMERA_MATRIX)
inverse_train_rotation_matrix = np.linalg.inv(train_rotation_matrix)
inv_query_map_1, inv_query_map_2 = cv2.initUndistortRectifyMap(
R=inverse_query_rotation_matrix,
cameraMatrix=CAMERA_MATRIX,
distCoeffs=DISTORTION_COEFFICIENTS,
newCameraMatrix=new_camera_matrix,
size=query_image.shape[::-1],
m1type=cv2.CV_16SC2)
inv_train_map_1, inv_train_map_2 = cv2.initUndistortRectifyMap(
R=inverse_train_rotation_matrix,
cameraMatrix=CAMERA_MATRIX,
distCoeffs=DISTORTION_COEFFICIENTS,
newCameraMatrix=new_camera_matrix,
size=query_image.shape[::-1],
m1type=cv2.CV_16SC2)
From here we can get the size of the new rectified image:
extended_x_size = (inv_query_map_1[:, :, 1].max()
- inv_query_map_1[:, :, 1].min())
extended_y_size = (inv_query_map_1[:, :, 0].max()
- inv_query_map_1[:, :, 0].min())
And how much we should shift the images:
x_shift = np.abs(inv_query_map_1[:, :, 1].min())
y_shift = np.abs(inv_query_map_1[:, :, 0].min())
Also, taking into account that the rectified train image will be shifted relative to the rectified query image, we need to add some more space:
delta_y = max(0, inv_train_map_1[:, :, 0].max() + y_shift - extended_y_size)
Using new dimensions to create maps:
query_map_1, query_map_2 = cv2.initUndistortRectifyMap(
R=query_rotation_matrix,
cameraMatrix=CAMERA_MATRIX,
distCoeffs=DISTORTION_COEFFICIENTS,
newCameraMatrix=new_camera_matrix,
size=(extended_y_size + delta_y, extended_x_size),
m1type=cv2.CV_16SC2)
train_map_1, train_map_2 = cv2.initUndistortRectifyMap(
R=train_rotation_matrix,
cameraMatrix=CAMERA_MATRIX,
distCoeffs=DISTORTION_COEFFICIENTS,
newCameraMatrix=new_camera_matrix,
size=(extended_y_size + delta_y, extended_x_size),
m1type=cv2.CV_16SC2)
And now not the most obvious part. When applying shifts to these maps, image will move not along the axis of the figure but along the axis of the image itself. Applying the shifts:
x_1 = inv_query_map_1[0, 0, 1]
x_2 = inv_query_map_1[0, -1, 1]
y_1 = inv_query_map_1[0, 0, 0]
y_2 = inv_query_map_1[0, -1, 0]
d_1 = x_2 - x_1
d_2 = y_2 - y_1
phi = np.arctan(d_1 / d_2)
psi = phi + np.arctan(y_shift / x_shift)
ro = np.sqrt(x_shift**2 + y_shift**2)
x_shift_image = ro * np.cos(psi)
y_shift_image = ro * np.sin(psi)
query_map_1[:, :, 1] -= x_shift_image.astype(int)
query_map_1[:, :, 0] -= y_shift_image.astype(int)
train_map_1[:, :, 1] -= x_shift_image.astype(int)
train_map_1[:, :, 0] -= y_shift_image.astype(int)
Finally, plotting the results:
rectified_query_image = cv2.remap(src=query_image,
map1=query_map_1,
map2=query_map_2,
interpolation=cv2.INTER_LINEAR)
rectified_train_image = cv2.remap(src=train_image,
map1=train_map_1,
map2=train_map_2,
interpolation=cv2.INTER_LINEAR)
plt.imshow(rectified_query_image)
plt.show()
plt.imshow(rectified_train_image)
plt.show()
rectified_query.jpg
rectified_train.jpg
epipolar_lines.jpg
This looks pretty good at first glance, but unfortunately my disparity map becomes broken...
disparity_before.jpg
disparity_after.jpg
Edit Nº4:
I was thinking that maybe the problem with the broken disparity could be because of different inclinations of the rectified images. If so, then shifts should be calculated separately for each image, like this:
from typing import Tuple
def shift_distance(map_: np.ndarray,
*,
x_shift: int,
y_shift: int) -> Tuple[int, int]:
"""
Converts shifts from figure coordinate system
to image coordinate system
"""
x_1 = map_[0, 0, 1]
x_2 = map_[0, -1, 1]
y_1 = map_[0, 0, 0]
y_2 = map_[0, -1, 0]
d_1 = x_2 - x_1
d_2 = y_2 - y_1
phi = np.arctan(d_1 / d_2)
psi = phi + np.arctan(y_shift / x_shift)
ro = np.sqrt(x_shift ** 2 + y_shift ** 2)
return ro * np.cos(psi), ro * np.sin(psi)
x_shift_image, y_shift_image = shift_distance(inv_query_map_1,
x_shift=x_shift,
y_shift=y_shift)
query_map_1[:, :, 1] -= x_shift_image.astype(int)
query_map_1[:, :, 0] -= y_shift_image.astype(int)
x_shift_image, y_shift_image = shift_distance(inv_train_map_1,
x_shift=x_shift,
y_shift=y_shift)
train_map_1[:, :, 1] -= x_shift_image.astype(int)
train_map_1[:, :, 0] -= y_shift_image.astype(int)
I hoped that this would fix the disparity image, but it doesn't, for some reason.