FCN-16s Fully Convolutional Semantic Segmentation on SIFT Flow

readme.md

name	caffemodel	caffemodel_url	sha1	gist_id
FCN-16s Fully Convolutional Semantic Segmentation on SIFT Flow	fcn-16s-sift-flow.caffemodel	http://dl.caffe.berkeleyvision.org/fcn-16s-sift-flow.caffemodel	a7cea6b351f0460c9a87ff613abe2df58b3bcc5d	f35e3a101e1478f721f5

This is a model from the paper:

Fully Convolutional Networks for Semantic Segmentation
Jonathan Long, Evan Shelhamer, Trevor Darrell
arXiv:1411.4038

This is the two stream, 16 pixel prediction stride version, trained on the SIFT flow dataset.

This model was trained to predict the 33-class segmentation labels, and simultaneously the three geometric labels. The final layers output scores for each class, which may be normalized via softmax or argmaxed to obtain per-pixel labels.

The input is expected in BGR channel order, with the following per-channel mean subtracted:

B 104.00698793 G 116.66876762 R 122.67891434

This is a pre-release: it requires unmerged PRs to run. It should be usable with the branch available at https://github.com/longjon/caffe/tree/future. Training ought to be possible with that code, but the original training scripts have not yet been ported.

This model is slightly different than the one reported in the paper; it obtains 85.7 overall accuracy, 50.2 mean accuracy, and 36.7 mean I/U on the SIFT flow test set.

fcn-16s-sift-flow-deploy.prototxt

name: 'FCN'

input: 'data'
input_dim: 1
input_dim: 3
input_dim: 500
input_dim: 500

layers { bottom: 'data' top: 'conv1_1' name: 'conv1_1' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 64 pad: 100 kernel_size: 3 } }
layers { bottom: 'conv1_1' top: 'conv1_1' name: 'relu1_1' type: RELU }
layers { bottom: 'conv1_1' top: 'conv1_2' name: 'conv1_2' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 64 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv1_2' top: 'conv1_2' name: 'relu1_2' type: RELU }
layers { name: 'pool1' bottom: 'conv1_2' top: 'pool1' type: POOLING
  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layers { name: 'conv2_1' bottom: 'pool1' top: 'conv2_1' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 128 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv2_1' top: 'conv2_1' name: 'relu2_1' type: RELU }
layers { bottom: 'conv2_1' top: 'conv2_2' name: 'conv2_2' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 128 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv2_2' top: 'conv2_2' name: 'relu2_2' type: RELU }
layers { bottom: 'conv2_2' top: 'pool2' name: 'pool2' type: POOLING
  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layers { bottom: 'pool2' top: 'conv3_1' name: 'conv3_1' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 256 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv3_1' top: 'conv3_1' name: 'relu3_1' type: RELU }
layers { bottom: 'conv3_1' top: 'conv3_2' name: 'conv3_2' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 256 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv3_2' top: 'conv3_2' name: 'relu3_2' type: RELU }
layers { bottom: 'conv3_2' top: 'conv3_3' name: 'conv3_3' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 256 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv3_3' top: 'conv3_3' name: 'relu3_3' type: RELU }
layers { bottom: 'conv3_3' top: 'pool3' name: 'pool3' type: POOLING
  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layers { bottom: 'pool3' top: 'conv4_1' name: 'conv4_1' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 512 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv4_1' top: 'conv4_1' name: 'relu4_1' type: RELU }
layers { bottom: 'conv4_1' top: 'conv4_2' name: 'conv4_2' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 512 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv4_2' top: 'conv4_2' name: 'relu4_2' type: RELU }
layers { bottom: 'conv4_2' top: 'conv4_3' name: 'conv4_3' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 512 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv4_3' top: 'conv4_3' name: 'relu4_3' type: RELU }
layers { bottom: 'conv4_3' top: 'pool4' name: 'pool4' type: POOLING
  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layers { bottom: 'pool4' top: 'conv5_1' name: 'conv5_1' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 512 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv5_1' top: 'conv5_1' name: 'relu5_1' type: RELU }
layers { bottom: 'conv5_1' top: 'conv5_2' name: 'conv5_2' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 512 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv5_2' top: 'conv5_2' name: 'relu5_2' type: RELU }
layers { bottom: 'conv5_2' top: 'conv5_3' name: 'conv5_3' type: CONVOLUTION
  convolution_param { engine: CAFFE num_output: 512 pad: 1 kernel_size: 3 } }
layers { bottom: 'conv5_3' top: 'conv5_3' name: 'relu5_3' type: RELU }
layers { bottom: 'conv5_3' top: 'pool5' name: 'pool5' type: POOLING
  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layers { bottom: 'pool5' top: 'fc6' name: 'fc6' type: CONVOLUTION
  convolution_param { engine: CAFFE kernel_size: 7 num_output: 4096 } }
layers { bottom: 'fc6' top: 'fc6' name: 'relu6' type: RELU }
layers { bottom: 'fc6' top: 'fc6' name: 'drop6' type: DROPOUT
  dropout_param { dropout_ratio: 0.5 } }
layers { bottom: 'fc6' top: 'fc7' name: 'fc7' type: CONVOLUTION
  convolution_param { engine: CAFFE kernel_size: 1 num_output: 4096 } }
layers { bottom: 'fc7' top: 'fc7' name: 'relu7' type: RELU }
layers { bottom: 'fc7' top: 'fc7' name: 'drop7' type: DROPOUT
  dropout_param { dropout_ratio: 0.5 } }
layers { name: 'score' type: CONVOLUTION bottom: 'fc7' top: 'score'
  convolution_param { engine: CAFFE num_output: 33 kernel_size: 1 } }
layers { name: 'score-geom' type: CONVOLUTION bottom: 'fc7' top: 'score-geom'
  convolution_param { engine: CAFFE num_output: 3 kernel_size: 1 } }

layers { type: DECONVOLUTION name: 'score2' bottom: 'score' top: 'score2'
  convolution_param { kernel_size: 4 stride: 2 num_output: 33 bias_term: false } }
layers { type: DECONVOLUTION name: 'score2-geom' bottom: 'score-geom' top: 'score2-geom'
  convolution_param { kernel_size: 4 stride: 2 num_output: 3 bias_term: false } }

layers { name: 'score-pool4' type: CONVOLUTION bottom: 'pool4' top: 'score-pool4'
  convolution_param { engine: CAFFE num_output: 33 kernel_size: 1 } }
layers { name: 'score-pool4-geom' type: CONVOLUTION bottom: 'pool4' top: 'score-pool4-geom'
  convolution_param { engine: CAFFE num_output: 3 kernel_size: 1 } }

layers { type: CROP name: 'crop' bottom: 'score-pool4' bottom: 'score2' top: 'score-pool4c' }
layers { type: CROP name: 'crop-geom' bottom: 'score-pool4-geom' bottom: 'score2-geom' top: 'score-pool4c-geom' }

layers { type: ELTWISE name: 'fuse' bottom: 'score2' bottom: 'score-pool4c' top: 'score-fused'
  eltwise_param { operation: SUM } }
layers { type: ELTWISE name: 'fuse-geom' bottom: 'score2-geom' bottom: 'score-pool4c-geom' top: 'score-fused-geom'
  eltwise_param { operation: SUM } }

layers { type: DECONVOLUTION name: 'upsample'
  bottom: 'score-fused' top: 'bigscore'
  convolution_param { num_output: 33 kernel_size: 32 stride: 16 bias_term: false } }
layers { type: DECONVOLUTION name: 'upsample-geom'
  bottom: 'score-fused-geom' top: 'bigscore-geom'
  convolution_param { num_output: 3 kernel_size: 32 stride: 16 bias_term: false } }

layers { type: CROP name: 'crop' bottom: 'bigscore' bottom: 'data' top: 'upscore' }
layers { type: CROP name: 'crop-geom' bottom: 'bigscore-geom' bottom: 'data' top: 'upscore-geom' }

It looks like mean I/U should be 39.1, for the same reason mentioned in the updated arxiv paper. @longjon could you please confirm and update this?