PMSNet stereo disparity 3 pixel error at test phase

pixel_error.py

"""
KITTI stereo disparity 3 pixel error

REFERENCES:
    * [PMSNet code](https://github.com/JiaRenChang/PSMNet/blob/b6520ff9b77168e3dfe3485f8bf6a2a798361d85/finetune.py)
    * [type converter](https://github.com/traveller59/torchplus/blob/master/tools.py)

MATLAB code:
```
% disp_error.m
function d_err = disp_error (D_gt,D_est,tau)

E = abs(D_gt-D_est);
n_err   = length(find(D_gt>0 & E>tau(1) & E./abs(D_gt)>tau(2)));
n_total = length(find(D_gt>0));
d_err = n_err/n_total;


%main.m
% error threshold
tau = [3 0.05];

% stereo demo
disp('Load and show disparity map ... ');
D_est = disp_read('data/disp_est.png');
D_gt  = disp_read('data/disp_gt.png');
d_err = disp_error(D_gt,D_est,tau);
```
So, we should also include the `equal` of both comparison

FrameWork: PyTorch
"""

import torch
import numpy as np


# [type converter](https://github.com/traveller59/torchplus/blob/master/tools.py)
def np_dtype_to_torch(dtype):
    type_map = {
        np.dtype(np.float16): torch.HalfTensor,
        np.dtype(np.float32): torch.FloatTensor,
        np.dtype(np.float64): torch.DoubleTensor,
        np.dtype(np.int32): torch.IntTensor,
        np.dtype(np.int64): torch.LongTensor,
        np.dtype(np.uint8): torch.ByteTensor,
    }
    return type_map[dtype]


def to_tensor(arg):
    if isinstance(arg, np.ndarray):
        return torch.from_numpy(arg).type(np_dtype_to_torch(arg.dtype))
    elif isinstance(arg, (list, tuple)):
        arg = np.array(arg)
        return torch.from_numpy(arg).type(np_dtype_to_torch(arg.dtype))
    else:
        raise ValueError("unsupported arg type.")


def calc_3pe(disp_pred, disp_true, original=False):
    assert disp_pred.shape == disp_true.shape
    assert disp_pred.dim() == 3

    disp_pred = disp_pred.clone().type(torch.FloatTensor)
    disp_true = disp_true.clone().type(torch.FloatTensor)

    if original:
        true_disp = disp_true
    else:
        true_disp = disp_true.clone()

    index = np.argwhere(true_disp > 0)
    disp_true[index[0][:], index[1][:], index[2][:]] = np.abs(
        true_disp[index[0][:], index[1][:], index[2][:]] -
        disp_pred[index[0][:], index[1][:], index[2][:]])
    correct = (disp_true[index[0][:], index[1][:], index[2][:]] <=
               3) | (disp_true[index[0][:], index[1][:], index[2][:]] <=
                     true_disp[index[0][:], index[1][:], index[2][:]] * 0.05)

    return 1 - (float(torch.sum(correct)) / float(len(index[0])))


def calc_3pe_standalone(disp_src, disp_dst):

    assert disp_src.shape == disp_dst.shape, "{}, {}".format(
        disp_src.shape, disp_dst.shape)
    assert len(disp_src.shape) == 2  # (N*M)

    not_empty = (disp_src > 0) & (~np.isnan(disp_src)) & (disp_dst > 0) & (
        ~np.isnan(disp_dst))

    disp_src_flatten = disp_src[not_empty].flatten().astype(np.float32)
    disp_dst_flatten = disp_dst[not_empty].flatten().astype(np.float32)

    disp_diff_l = abs(disp_src_flatten - disp_dst_flatten)

    accept_3p = (disp_diff_l <= 3) | (disp_diff_l <= disp_dst_flatten * 0.05)
    err_3p = 1 - np.count_nonzero(accept_3p) / len(disp_diff_l)

    return err_3p


if __name__ == '__main__':

    USE_ORIGINAL = False

    np.random.seed(42)
    # bz, h, w = (1, 2, 5)
    bz, h, w = (20, 20, 50)
    # est_disp = np.random.randint(low=10, high=200, size= (bz, h, w), dtype=np.uint8)
    # gt_disp = np.random.randint(low=10, high=200, size= (bz, h, w), dtype=np.uint8)
    est_disp = np.random.randint(low=100, high=2000, size=(bz, h, w)) / 10.0
    gt_disp = np.random.randint(low=100, high=2000, size=(bz, h, w)) / 10.0

    def err_two_ndarr(arr1, arr2):
        tensor1 = to_tensor(arr1)
        tensor2 = to_tensor(arr2)
        # print(tensor1.shape)
        e = calc_3pe(tensor1, tensor2, original=USE_ORIGINAL)
        return e

    def split_arr(arr, step):
        l = []
        for i in range(0, arr.shape[0], step):
            l.append(arr[i:i + step])
        return l

    def clc_split_step(step=1):
        est_l = split_arr(est_disp, step=step)
        gt_l = split_arr(gt_disp, step=step)
        err = 0
        for est, gt in zip(est_l, gt_l):
            err_tmp = err_two_ndarr(est, gt)
            # print('{:.4f}'.format(err_tmp))
            err += err_tmp
        return err / len(est_l)

    step_l = [1, 2, 3, 4, 6]

    print("ORIGINAL:{}".format(USE_ORIGINAL))
    for step in step_l:
        e = clc_split_step(step=step)
        print("step:{}, err:{:.6f}".format(step, e))

    err = 0
    for i in range(len(est_disp)):
        est = est_disp[i]
        gt = gt_disp[i]
        err_3p = calc_3pe_standalone(disp_src=est, disp_dst=gt)  # gt as dst
        # print('{:.4f}'.format(err_3p))
        err += err_3p

    print("STANDALONE: {:.6f}".format(err / len(est_disp)))

• ORIGINAL:False
  step:1, err:0.942450
  step:2, err:0.942450
  step:3, err:0.942429
  step:4, err:0.942450
  step:6, err:0.942375
  STANDALONE: 0.942450

• ORIGINAL:True
  step:1, err:0.969700
  step:2, err:0.969700
  step:3, err:0.969571
  step:4, err:0.969700
  step:6, err:0.969250
  STANDALONE: 0.942450

NOTE ALSO the batch size also matters, choose one divisor of all examples is better.
And, this partially answers one issue Outlier 3-px error value in last batch of validation set #111

WHY?

It is somewhat strange at first glance, it should be the same, right? Let's show with one small example:

o x
x x
o o

o means True and x for False.

With batch size:
bz=3, err = 3/6 = 0.5
bz=1, err = (1/2 + 2/2 + 0/2)/3 = 0.5
bz=2, err = (3/4 + 0/2)/2 = 3/8 = 0.375

And, another one:

o x
x x
x x

With batch size:
bz=3, err = 5/6 = 0.8333
bz=1, err = (1/2 + 2/2 + 2/2)/3 = 5/6
bz=2, err = (3/4 + 2/2)/2 = 7/8 = 0.875

SO, that is it, the last batch are not treated equally with previous ones (there is only two batch for bz=3, but easily extended), it is not so proper to calculate mean of these batches.

Then the problem becomes, how can we deal with it?

• Make sure batch size is the divisor of all examples
  • Yes, it can be frustrating
• Or, instead of calc mean per batch and then mean on batches, record (Correct num, ALL num) per batch and calc mean with these pairs
  • Well, are there better ways?

Note, the analysis is helpful, but not that correct, please refer the correct one.