Skip to content

Instantly share code, notes, and snippets.

@eerwitt
Created January 31, 2016 05:52
Show Gist options
  • Star 99 You must be signed in to star a gist
  • Fork 24 You must be signed in to fork a gist
  • Save eerwitt/518b0c9564e500b4b50f to your computer and use it in GitHub Desktop.
Save eerwitt/518b0c9564e500b4b50f to your computer and use it in GitHub Desktop.
Example loading multiple JPEG files with TensorFlow and make them available as Tensors with the shape [[R, G, B], ... ].
# Typical setup to include TensorFlow.
import tensorflow as tf
# Make a queue of file names including all the JPEG images files in the relative
# image directory.
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("./images/*.jpg"))
# Read an entire image file which is required since they're JPEGs, if the images
# are too large they could be split in advance to smaller files or use the Fixed
# reader to split up the file.
image_reader = tf.WholeFileReader()
# Read a whole file from the queue, the first returned value in the tuple is the
# filename which we are ignoring.
_, image_file = image_reader.read(filename_queue)
# Decode the image as a JPEG file, this will turn it into a Tensor which we can
# then use in training.
image = tf.image.decode_jpeg(image_file)
# Start a new session to show example output.
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.initialize_all_variables().run()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run([image])
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
@RyanAkilos
Copy link

Thanks for the example. Just a quick a question. Does this creates the label for each image? If not, how do I create the label for them. Thanks!

@vadakattu
Copy link

@RyanAkilos no this script does not create a label. A possible modification would be to use the folder path to infer the label (assuming that your training images are separated in folders by class)

@greenfigo2015
Copy link

only one image is read too, any solutions? Thanks!

@Edouard360
Copy link

I had the OutOfRange error but sess.run(tf.local_variables_initializer()) worked it out. Thanks @AravindhKuppusamy

@ianholing
Copy link

I load some of them in an array, but my computer is not a super machine and is running out of memory quickly. Is there a way to release them?

@Hariharanmayu
Copy link

2018-02-08 14:30:06.707850: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-02-08 14:30:06.707991: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_should_use.py:175: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "images.py", line 32, in
image_tensor = sess.run([image])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

Caused by op 'ReaderReadV2', defined at:
File "images.py", line 16, in
_, image_file = image_reader.read(filename_queue)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/io_ops.py", line 194, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 423, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

I have this error. anybody can help me??

@Junaid388
Copy link

this is really helpful thank you.
I want to add all the pixel value in csv file after converting each image into grey image how can i do that.

@imnaren142
Copy link

imnaren142 commented Mar 4, 2018

I have loaded a directory of 240 images to the path.But it showing an error when I run this code.
import tensorflow as tf
x = tf.placeholder(tf.float32, [None, 270000])
from tensorflow.examples.tutorials.mnist import input_data
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("/home/iamnaren/DeepLearning/T/*.jpg"))
image_reader = tf.WholeFileReader()
_, image_file = image_reader.read(filename_queue)
image_file.shape
image = tf.image.decode_jpeg(image_file)
print(image_file)
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.global_variables_initializer()

# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)

# Get an image tensor and print its value.
image_tensor = sess.run(image_file)
print(image_tensor)

# Finish off the filename queue coordinator.
coord.request_stop()

coord.join(threads)
THE ERROR IS

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.FailedPreconditionError'>, Attempting to use uninitialized value matching_filenames
[[Node: matching_filenames/read = IdentityT=DT_STRING, _class=["loc:@matching_filenames"], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]


OutOfRangeError Traceback (most recent call last)
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1349 try:
-> 1350 return fn(*args)
1351 except errors.OpError as e:

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1328 feed_dict, fetch_list, target_list,
-> 1329 status, run_metadata)
1330

~/env3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg)
472 compat.as_text(c_api.TF_Message(self.status.status)),
--> 473 c_api.TF_GetCode(self.status.status))
474 # Delete the underlying status object from memory otherwise it stays alive

OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]

During handling of the above exception, another exception occurred:

OutOfRangeError Traceback (most recent call last)
in ()
8
9 # Get an image tensor and print its value.
---> 10 image_tensor = sess.run(image_file)
11 print(image_tensor)
12

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
893 try:
894 result = self._run(None, fetches, feed_dict, options_ptr,
--> 895 run_metadata_ptr)
896 if run_metadata:
897 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1126 if final_fetches or final_targets or (handle and feed_dict_tensor):
1127 results = self._do_run(handle, final_targets, final_fetches,
-> 1128 feed_dict_tensor, options, run_metadata)
1129 else:
1130 results = []

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1342 if handle is None:
1343 return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1344 options, run_metadata)
1345 else:
1346 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1361 except KeyError:
1362 pass
-> 1363 raise type(e)(node_def, op, message)
1364
1365 def _extend_graph(self):

OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]

Caused by op 'ReaderReadV2', defined at:
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/iamnaren/env3/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 486, in start
self.io_loop.start()
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/ioloop.py", line 832, in start
self._run_callback(self._callbacks.popleft())
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
ret = callback()
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 536, in
self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
self._handle_recv()
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2728, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2850, in run_ast_nodes
if self.run_code(code, result):
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 4, in
_, image_file = image_reader.read(filename_queue)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/ops/io_ops.py", line 209, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 678, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]

@wsthub
Copy link

wsthub commented May 7, 2018

I also had the same problem:
"OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)"

However, I found that the error did not occur when I changed
filename_queue = tf.train.string_input_producer ( tf.train.match_filenames_once ( IMAGE_FILE_PATH + '*.JPG' ) )
to
IMAGE_FILE_PATH = './images/'
filenames = [ os.path.join( IMAGE_FILE_PATH, filename ) for filename in os.listdir( IMAGE_FILE_PATH ) ]
filename_queue = tf.train.string_input_producer( filenames )

@flyher
Copy link

flyher commented May 18, 2018

@Hariharanmayu
I also got the same issue!

@flyher
Copy link

flyher commented May 18, 2018

Hi all, the code will work after you change the code from

filename_queue = tf.train.string_input_producer(
    tf.train.match_filenames_once("./images/*.jpg"))

to

filename_queue = tf.train.string_input_producer(['./*.jpg'])

I donnot know why, so who can explain it.

@shivangiiert
Copy link

shivangiiert commented Apr 3, 2019

filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once('./apple2orange/trainA/*.jpg'))
gives me error TypeError: Fetch argument <tensorflow.python.ops.data_flow_ops.FIFOQueue object at 0xb1a840ef0> has invalid type <class 'tensorflow.python.ops.data_flow_ops.FIFOQueue'>, must be a string or Tensor. (Can not convert a FIFOQueue into a Tensor or Operation.)

can anybody tell me why tf.train.match_filenames_once is giving me error of Can not convert a FIFOQueue into a Tensor or Operation?
how to solve this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment