FCN-32s Fully Convolutional Semantic Segmentation on PASCAL-Context

readme.md

See the reference implementation at http://fcn.berkeleyvision.org. This pre-release is deprecated.

@Fchaubard @KimHoon BVLC/caffe#1698 (comment) here is an example, also here https://groups.google.com/forum/#!searchin/caffe-users/segmentation$20lmdb/caffe-users/19XfmJqg34Q/Id_jnm_L0iIJ

@debadeepta Have you figure out why ? I got similar convergence value, around 650000.

Same here. Also made sure to cluster all the classes outside of the 59 set to zero (background). Note that classes 1,2,3,... in the 59 set are not necessarily 1,2,3,... in the full set. You need to map them so you end up with indices in the [0, 59] range. Still getting > 666790 loss with fluctuations that never seem to go below 440K.

I tried describing my procedure in this post.

I've set up Caffe (the future-branch) and successfully run the FCN-32s Fully Convolutional Semantic Segmentation on PASCAL-Context model. However, I'm unable to produce clear labeled images with it. Here are my results:

Input image:

Ground truth (example from the PASCAL webpage):

My result (not very good):
.

Any idea of where I'm going wrong?

@irri please see my answer at stackoverflow

how can I convert JPEG images to data which caffe can process?

I use convert_imageset in caffe/tools, but it need a a list of images as well as their labels(an integer). But in FCN-32s the labels are images.

@Eniac-Xie: you can use a function like this - https://gist.github.com/arunmallya/9b67faf63405389afb83, to load CSV data (segmentation labels) into a datum and then store it in an LMDB database.

My dataset has 2 classes; with 1000 training images of (5,256,256) also corresponding ground truth data (1,256,256) which is a binary image either 0 or 1 to represent the 2 classes.

When training in solve.py you use the existing caffemodel which I assume is 3-channel ; but as I want to implement in on my 5 channel dataset can I use the same model provided ?

I have just tried to run the eval.py using FCN-32s model. But I get the following error

F1105 19:29:04.896656 25800 layer_factory.hpp:77] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Crop (known types: AbsVal, Accuracy, ArgMax, BNLL, Concat, ContrastiveLoss, Convolution, Data, Deconvolution, Dropout, DummyData, Eltwise, EuclideanLoss, Exp, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, LRN, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Pooling, Power, Python, ReLU, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, WindowData)
*** Check failure stack trace: ***
Aborted (core dumped)

which says that the Crop is an unknown layer type. I have no idea of what is going wrong. Could anyone give me some suggestions?

BTW, I use the required caffe-future brach: I download and unzip caffe-future.zip. Then I copy the Makefile.config from caffe-master to it and successfully run make all and make pycaffe.

@Jianchao-ICT update your ld_library_path and python_path to point to new caffe-future branch which has the crop layer

Hello anyone, I execute eval.py, and it returns at the end:

...
I1119 19:59:50.529485  6746 net.cpp:241] conv1_1 does not need backward computation.
I1119 19:59:50.529494  6746 net.cpp:241] data_input_0_split does not need backward computation.
I1119 19:59:50.529501  6746 net.cpp:284] This network produces output score
I1119 19:59:50.529527  6746 net.cpp:298] Network initialization done.
I1119 19:59:50.529536  6746 net.cpp:299] Memory required for data: 1277452160
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 597011289

So I have to increase the limit and recompile the protobuf now? Is there any better solution? Thank you!

I have to use a quick and dirty approach to hack the kDefaultTotalBytesLimit and kDefaultTotalBytesWarningThreshold in the coded_stream.h, recompile, reinstall. So eval.py can execute successfully finally.

I will try to limit the sizes of the big messages now.

Could anybody explain what the 'Crop' layer is for?

What is the train/val split on the PASCAL Context data that is needed to get the 35.1 mean I/U?

There is a train.txt and val.txt in the VOC2010/ImageSets/Segmentation folder of the PASCAL VOC 2010 download, but these only have 964 images in each. The Long et al. paper seems to use half of the 10,103 images available for training and the other half for testing. So, I don't think these are the correct files. The PASCAL Context Dataset download also doesn't have any files for the train/val split.

I was able to get a mean 53.5% accuracy after 10150 iteration with a loss of 67968. This was with a 50/50 random split for train vs test. The accuracy was determined by adding an 'accuracy' layer to train_val.prototxt. How is the mean I/U determined? Is there a similar layer that can be added?

Why the learning rate is so small, can anyone explain it,plz?

Which should I use to train my own data，the solver.py or the solver.prototxt? I used the solver.prototxt to train my data,but the loss didn't decrease. Dose anyone have a solution? Thanks

Hi, anyone know where to download "vgg16fc.caffemodel" in the solver.py?

I would like to share my script for creating a custom data set in lmdb format. It would be very appreciated if you could let me know when you find any bug on this code.

#!/usr/bin/python

import caffe
import glob
import lmdb
import numpy as np
from PIL import Image
import os
import sys

# Variables
img_width = 500
img_height = 500


# Paths
# PNG images
color_dir = './input/color_image_dir'
# PNG images
# Per-pixel labels are stored in a gray image
label_dir = './input/label_image_dir'
output_dir = './lmdb/'


inputs = glob.glob(color_dir + '/*.png')

color_lmdb_name = output_dir + '/color-lmdb'
if not os.path.isdir(color_lmdb_name):
    os.makedirs(color_lmdb_name)
color_in_db = lmdb.open(color_lmdb_name, map_size=int(1e12))

label_lmdb_name = output_dir + '/label-lmdb'
if not os.path.isdir(label_lmdb_name):
    os.makedirs(label_lmdb_name)
label_in_db = lmdb.open(label_lmdb_name, map_size=int(1e12))

num_images = 0;
color_mean_color = np.zeros((3))


with color_in_db.begin(write=True) as color_in_txn:
    with label_in_db.begin(write=True) as label_in_txn:

        for in_idx, in_ in enumerate(inputs):
            img_name = os.path.splitext( os.path.basename(in_))[0]
            color_filename = color_dir + img_name + '.png'
            label_filename = label_dir + img_name + '.png'
            print(str(in_idx + 1) + ' / ' + str(len(inputs)))

            # load image
            im = np.array(Image.open(color_filename)) # or load whatever ndarray you need
            assert im.dtype == np.uint8            
            # RGB to BGR
            im = im[:,:,::-1]
            # in Channel x Height x Width order (switch from H x W x C)
            im = im.transpose((2,0,1))

            # compute mean color image
            for i in range(3):
                color_mean_color[i] += im[i,:,:].mean()
            num_images += 1

            #color_im_dat = caffe.io.array_to_datum(im)
            color_im_dat = caffe.proto.caffe_pb2.Datum()
            color_im_dat.channels, color_im_dat.height, color_im_dat.width = im.shape
            assert color_im_dat.height == img_height
            assert color_im_dat.width == img_width
            color_im_dat.data = im.tostring()
            color_in_txn.put('{:0>12d}'.format(in_idx), color_im_dat.SerializeToString())

            im = np.array(Image.open(label_filename)) # or load whatever ndarray you need
            assert im.dtype == np.uint8
            label_im_dat = caffe.proto.caffe_pb2.Datum()
            label_im_dat.channels = 1
            label_im_dat.height, label_im_dat.width = im.shape
            assert label_im_dat.height == img_height
            assert label_im_dat.width == img_width
            label_im_dat.data = im.tostring()
            label_in_txn.put('{:0>12d}'.format(in_idx), label_im_dat.SerializeToString())

    label_in_db.close()
color_in_db.close()

color_mean_color /= num_images
np.savetxt(output_dir + '/{}.csv'.format('color-mean'), color_mean_color, delimiter=",", fmt='%.4f')

pad: 100 in conv1_1 layer is wrong..

When I tried to run deploy.prototxt, it takes very large memory, and the process crash on CPU.
Can anyone help me to reduce the use of memory or solve this problem?

Thanks in advance.

Hi guys,
how to finetune on different number of classes?
thanks

Can I fine-tune this model for high-resolution image classification? The input sizes of Alexnet/Caffenet/GoogleLenet are too small for my application.

@weiliu89 Hi, I encountered conflicts when I merged PR #2016 , it says "Automatic merge failed". What should I do next? Thank you in advance.

Hi, I tried to use deconv layer with group and bilinear as upsampling instead of using the solver script, but could hardly reproduce the result. Anybody knows the reason?

@acgtyrant
Hi I am getting this error :
I0421 02:38:28.223543 24891 net.cpp:299] Memory required for data: 1277452160
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 597011289

I installed the protobuf using the command given in google tensorflow website
$ pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/protobuf-3.0.0b2.post2-cp27-none-linux_x86_64.whl

However, once I uninstalled that and tried to compile using the github source code from google protobuf, python can't find the google protobuf at all. Can you help me finding a way here?

Thanks in advance

@yjc04
I had same error. I just modified "kDefaultTotalBytesLimit" and "kDefaultTotalBytesWarningThreshold" in the "/usr/include/google/protobuf/io/coded_stream.h". I didn't recompile and reinstall, it works well.

Hello all, Have anyone try to export prediction image using C++, instead of python? I am not the family of python code. Thank all

any c++ for classification or Segmentation would be great.

Hi,
I trained the FCN32 from the scratch. But the output is zero values (a black image). Could someone help what is the reason? Thanks