Skip to content

Instantly share code, notes, and snippets.

@eerwitt
Created January 31, 2016 05:52
  • Star 99 You must be signed in to star a gist
  • Fork 24 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save eerwitt/518b0c9564e500b4b50f to your computer and use it in GitHub Desktop.
Example loading multiple JPEG files with TensorFlow and make them available as Tensors with the shape [[R, G, B], ... ].
# Typical setup to include TensorFlow.
import tensorflow as tf
# Make a queue of file names including all the JPEG images files in the relative
# image directory.
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("./images/*.jpg"))
# Read an entire image file which is required since they're JPEGs, if the images
# are too large they could be split in advance to smaller files or use the Fixed
# reader to split up the file.
image_reader = tf.WholeFileReader()
# Read a whole file from the queue, the first returned value in the tuple is the
# filename which we are ignoring.
_, image_file = image_reader.read(filename_queue)
# Decode the image as a JPEG file, this will turn it into a Tensor which we can
# then use in training.
image = tf.image.decode_jpeg(image_file)
# Start a new session to show example output.
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.initialize_all_variables().run()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run([image])
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
@tianmingdu
Copy link

Hi eerwitt,
It seems that the image_tensor only return an image. How do I get multiple image tensors?

Cheers,
Tianming

@leiup
Copy link

leiup commented Aug 29, 2016

Hi eerwitt,

I have the same question as Tianming. Thank you very much!

best regards,
Lei

@marcbelmont
Copy link

If you want a mini batch, you can add

    image.set_shape((102, 136, 3))

    # Generate batch
    num_preprocess_threads = 1
    min_queue_examples = 256
    images = tf.train.shuffle_batch(
        [image],
        batch_size=batch_size,
        num_threads=num_preprocess_threads,
        capacity=min_queue_examples + 3 * batch_size,
        min_after_dequeue=min_queue_examples)

@prakashjayy
Copy link

OutOfRangeError (see above for traceback): RandomShuffleQueue '_0_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 32, current size 0)
 [[Node: shuffle_batch = QueueDequeueMany[_class=["loc:@shuffle_batch/random_shuffle_queue"], component_types=[DT_UINT8, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

I am getting the following error. can you tell me the possible reason?

@HarshaVardhanP
Copy link

HarshaVardhanP commented Feb 15, 2017

Can you post entire error message ? Are you getting any size mismatch error ? If yes, read on
All the images should be of same size. You might need to do this. [resize and set shape]

image_orig = tf.image.decode_jpeg(image_file)
image = tf.image.resize_images(image_orig, [224, 224])
image.set_shape((224, 224, 3))
batch_size = 50
num_preprocess_threads = 1
min_queue_examples = 256
images = tf.train.shuffle_batch(
[image],
batch_size=batch_size,
num_threads=num_preprocess_threads,
capacity=min_queue_examples + 3 * batch_size,
min_after_dequeue=min_queue_examples)

@rowjay
Copy link

rowjay commented Feb 18, 2017

Thanks, very helpful!

@ilovin
Copy link

ilovin commented Apr 11, 2017

how can I get an array with the data of image_width
when using

image = tf.image.decode_png(image_file)
width = image.shape[1]

the width is always None


edit:

width=tf.shape(image)[1]

width_val = sess.run([width])

and it will get the result

@himat
Copy link

himat commented Apr 16, 2017

This was very helpful.
But in your minibatch example, why did you do image.set_shape((102, 136, 3))

@manselag123
Copy link

how can read multipe directories of images for feeding them into the network

@simicvm
Copy link

simicvm commented May 4, 2017

Hi, thanks for the script. But I have a problem, image loaded is totally black (all zeros in matrix, and size is correct). Tried it with several images in different locations, and it's always the same. Whole script executes without any errors.

@345ishaan
Copy link

Hi All,

I need some inputs in the case where I have stored my data in a tfrecord file.

I have N number of images each of size 2272273 stored in a tf record in a single string format. Hence when I will decode a image raw string , I will get a tensor of size N227227*3 where N is not fixed. And in my tfrecord file, I have M such images.

I want to use tf.train.shuffle to get the batches from these MN images of size (227227*3).

@setare94
Copy link

setare94 commented May 21, 2017

I have the OutOfRangeError, I resize and setshape images but I still have this error :(
what must I do?

Copy link

ghost commented Jun 7, 2017

it raises an OutOfRangeError to me too.

@AravindhKuppusamy
Copy link

If you are getting OutOfRangeError, try initializing the local variables.
Instead of
tf.initialize_all_variables().run() - Deprecated!
try
tf.local_variables_initializer().run()

@kon3m
Copy link

kon3m commented Jul 13, 2017

if your getting this error:tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value Variable
then try:
tf.global_variables_initializer().run()

@anonymousAwesome
Copy link

anonymousAwesome commented Aug 16, 2017

I kept getting weird problems where Tensorflow wouldn't recognize that the decoded data was a tensor, claiming that the type was class list. Turns out, I needed to remove the square brackets around image in line 32 (that's the one that says, "image_tensor = sess.run([image])" . After that, it worked like a charm.

@RyanAkilos
Copy link

Thanks for the example. Just a quick a question. Does this creates the label for each image? If not, how do I create the label for them. Thanks!

@vadakattu
Copy link

@RyanAkilos no this script does not create a label. A possible modification would be to use the folder path to infer the label (assuming that your training images are separated in folders by class)

@greenfigo2015
Copy link

only one image is read too, any solutions? Thanks!

@Edouard360
Copy link

I had the OutOfRange error but sess.run(tf.local_variables_initializer()) worked it out. Thanks @AravindhKuppusamy

@ianholing
Copy link

I load some of them in an array, but my computer is not a super machine and is running out of memory quickly. Is there a way to release them?

@Hariharanmayu
Copy link

2018-02-08 14:30:06.707850: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-02-08 14:30:06.707991: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_should_use.py:175: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "images.py", line 32, in
image_tensor = sess.run([image])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

Caused by op 'ReaderReadV2', defined at:
File "images.py", line 16, in
_, image_file = image_reader.read(filename_queue)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/io_ops.py", line 194, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 423, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

I have this error. anybody can help me??

@Junaid388
Copy link

this is really helpful thank you.
I want to add all the pixel value in csv file after converting each image into grey image how can i do that.

@imnaren142
Copy link

imnaren142 commented Mar 4, 2018

I have loaded a directory of 240 images to the path.But it showing an error when I run this code.
import tensorflow as tf
x = tf.placeholder(tf.float32, [None, 270000])
from tensorflow.examples.tutorials.mnist import input_data
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("/home/iamnaren/DeepLearning/T/*.jpg"))
image_reader = tf.WholeFileReader()
_, image_file = image_reader.read(filename_queue)
image_file.shape
image = tf.image.decode_jpeg(image_file)
print(image_file)
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.global_variables_initializer()

# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)

# Get an image tensor and print its value.
image_tensor = sess.run(image_file)
print(image_tensor)

# Finish off the filename queue coordinator.
coord.request_stop()

coord.join(threads)
THE ERROR IS

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.FailedPreconditionError'>, Attempting to use uninitialized value matching_filenames
[[Node: matching_filenames/read = IdentityT=DT_STRING, _class=["loc:@matching_filenames"], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]


OutOfRangeError Traceback (most recent call last)
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1349 try:
-> 1350 return fn(*args)
1351 except errors.OpError as e:

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1328 feed_dict, fetch_list, target_list,
-> 1329 status, run_metadata)
1330

~/env3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg)
472 compat.as_text(c_api.TF_Message(self.status.status)),
--> 473 c_api.TF_GetCode(self.status.status))
474 # Delete the underlying status object from memory otherwise it stays alive

OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]

During handling of the above exception, another exception occurred:

OutOfRangeError Traceback (most recent call last)
in ()
8
9 # Get an image tensor and print its value.
---> 10 image_tensor = sess.run(image_file)
11 print(image_tensor)
12

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
893 try:
894 result = self._run(None, fetches, feed_dict, options_ptr,
--> 895 run_metadata_ptr)
896 if run_metadata:
897 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1126 if final_fetches or final_targets or (handle and feed_dict_tensor):
1127 results = self._do_run(handle, final_targets, final_fetches,
-> 1128 feed_dict_tensor, options, run_metadata)
1129 else:
1130 results = []

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1342 if handle is None:
1343 return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1344 options, run_metadata)
1345 else:
1346 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1361 except KeyError:
1362 pass
-> 1363 raise type(e)(node_def, op, message)
1364
1365 def _extend_graph(self):

OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]

Caused by op 'ReaderReadV2', defined at:
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/iamnaren/env3/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 486, in start
self.io_loop.start()
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/ioloop.py", line 832, in start
self._run_callback(self._callbacks.popleft())
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
ret = callback()
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 536, in
self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
self._handle_recv()
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2728, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2850, in run_ast_nodes
if self.run_code(code, result):
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 4, in
_, image_file = image_reader.read(filename_queue)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/ops/io_ops.py", line 209, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 678, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]

@wsthub
Copy link

wsthub commented May 7, 2018

I also had the same problem:
"OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)"

However, I found that the error did not occur when I changed
filename_queue = tf.train.string_input_producer ( tf.train.match_filenames_once ( IMAGE_FILE_PATH + '*.JPG' ) )
to
IMAGE_FILE_PATH = './images/'
filenames = [ os.path.join( IMAGE_FILE_PATH, filename ) for filename in os.listdir( IMAGE_FILE_PATH ) ]
filename_queue = tf.train.string_input_producer( filenames )

@flyher
Copy link

flyher commented May 18, 2018

@Hariharanmayu
I also got the same issue!

@flyher
Copy link

flyher commented May 18, 2018

Hi all, the code will work after you change the code from

filename_queue = tf.train.string_input_producer(
    tf.train.match_filenames_once("./images/*.jpg"))

to

filename_queue = tf.train.string_input_producer(['./*.jpg'])

I donnot know why, so who can explain it.

@shivangiiert
Copy link

shivangiiert commented Apr 3, 2019

filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once('./apple2orange/trainA/*.jpg'))
gives me error TypeError: Fetch argument <tensorflow.python.ops.data_flow_ops.FIFOQueue object at 0xb1a840ef0> has invalid type <class 'tensorflow.python.ops.data_flow_ops.FIFOQueue'>, must be a string or Tensor. (Can not convert a FIFOQueue into a Tensor or Operation.)

can anybody tell me why tf.train.match_filenames_once is giving me error of Can not convert a FIFOQueue into a Tensor or Operation?
how to solve this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment