-
-
Save eerwitt/518b0c9564e500b4b50f to your computer and use it in GitHub Desktop.
# Typical setup to include TensorFlow. | |
import tensorflow as tf | |
# Make a queue of file names including all the JPEG images files in the relative | |
# image directory. | |
filename_queue = tf.train.string_input_producer( | |
tf.train.match_filenames_once("./images/*.jpg")) | |
# Read an entire image file which is required since they're JPEGs, if the images | |
# are too large they could be split in advance to smaller files or use the Fixed | |
# reader to split up the file. | |
image_reader = tf.WholeFileReader() | |
# Read a whole file from the queue, the first returned value in the tuple is the | |
# filename which we are ignoring. | |
_, image_file = image_reader.read(filename_queue) | |
# Decode the image as a JPEG file, this will turn it into a Tensor which we can | |
# then use in training. | |
image = tf.image.decode_jpeg(image_file) | |
# Start a new session to show example output. | |
with tf.Session() as sess: | |
# Required to get the filename matching to run. | |
tf.initialize_all_variables().run() | |
# Coordinate the loading of image files. | |
coord = tf.train.Coordinator() | |
threads = tf.train.start_queue_runners(coord=coord) | |
# Get an image tensor and print its value. | |
image_tensor = sess.run([image]) | |
print(image_tensor) | |
# Finish off the filename queue coordinator. | |
coord.request_stop() | |
coord.join(threads) |
Hi eerwitt,
I have the same question as Tianming. Thank you very much!
best regards,
Lei
If you want a mini batch, you can add
image.set_shape((102, 136, 3))
# Generate batch
num_preprocess_threads = 1
min_queue_examples = 256
images = tf.train.shuffle_batch(
[image],
batch_size=batch_size,
num_threads=num_preprocess_threads,
capacity=min_queue_examples + 3 * batch_size,
min_after_dequeue=min_queue_examples)
OutOfRangeError (see above for traceback): RandomShuffleQueue '_0_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 32, current size 0)
[[Node: shuffle_batch = QueueDequeueMany[_class=["loc:@shuffle_batch/random_shuffle_queue"], component_types=[DT_UINT8, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
I am getting the following error. can you tell me the possible reason?
Can you post entire error message ? Are you getting any size mismatch error ? If yes, read on
All the images should be of same size. You might need to do this. [resize and set shape]
image_orig = tf.image.decode_jpeg(image_file)
image = tf.image.resize_images(image_orig, [224, 224])
image.set_shape((224, 224, 3))
batch_size = 50
num_preprocess_threads = 1
min_queue_examples = 256
images = tf.train.shuffle_batch(
[image],
batch_size=batch_size,
num_threads=num_preprocess_threads,
capacity=min_queue_examples + 3 * batch_size,
min_after_dequeue=min_queue_examples)
Thanks, very helpful!
how can I get an array with the data of image_width
when using
image = tf.image.decode_png(image_file)
width = image.shape[1]
the width is always None
edit:
width=tf.shape(image)[1]
width_val = sess.run([width])
and it will get the result
This was very helpful.
But in your minibatch example, why did you do image.set_shape((102, 136, 3))
how can read multipe directories of images for feeding them into the network
Hi, thanks for the script. But I have a problem, image loaded is totally black (all zeros in matrix, and size is correct). Tried it with several images in different locations, and it's always the same. Whole script executes without any errors.
Hi All,
I need some inputs in the case where I have stored my data in a tfrecord file.
I have N number of images each of size 2272273 stored in a tf record in a single string format. Hence when I will decode a image raw string , I will get a tensor of size N227227*3 where N is not fixed. And in my tfrecord file, I have M such images.
I want to use tf.train.shuffle to get the batches from these MN images of size (227227*3).
I have the OutOfRangeError, I resize and setshape images but I still have this error :(
what must I do?
it raises an OutOfRangeError to me too.
If you are getting OutOfRangeError, try initializing the local variables.
Instead of
tf.initialize_all_variables().run() - Deprecated!
try
tf.local_variables_initializer().run()
if your getting this error:tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value Variable
then try:
tf.global_variables_initializer().run()
I kept getting weird problems where Tensorflow wouldn't recognize that the decoded data was a tensor, claiming that the type was class list. Turns out, I needed to remove the square brackets around image in line 32 (that's the one that says, "image_tensor = sess.run([image])" . After that, it worked like a charm.
Thanks for the example. Just a quick a question. Does this creates the label for each image? If not, how do I create the label for them. Thanks!
@RyanAkilos no this script does not create a label. A possible modification would be to use the folder path to infer the label (assuming that your training images are separated in folders by class)
only one image is read too, any solutions? Thanks!
I had the OutOfRange error but sess.run(tf.local_variables_initializer())
worked it out. Thanks @AravindhKuppusamy
I load some of them in an array, but my computer is not a super machine and is running out of memory quickly. Is there a way to release them?
2018-02-08 14:30:06.707850: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-02-08 14:30:06.707991: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/util/tf_should_use.py:175: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer
instead.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "images.py", line 32, in
image_tensor = sess.run([image])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
Caused by op 'ReaderReadV2', defined at:
File "images.py", line 16, in
_, image_file = image_reader.read(filename_queue)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/io_ops.py", line 194, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 423, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
I have this error. anybody can help me??
this is really helpful thank you.
I want to add all the pixel value in csv file after converting each image into grey image how can i do that.
I have loaded a directory of 240 images to the path.But it showing an error when I run this code.
import tensorflow as tf
x = tf.placeholder(tf.float32, [None, 270000])
from tensorflow.examples.tutorials.mnist import input_data
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("/home/iamnaren/DeepLearning/T/*.jpg"))
image_reader = tf.WholeFileReader()
_, image_file = image_reader.read(filename_queue)
image_file.shape
image = tf.image.decode_jpeg(image_file)
print(image_file)
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.global_variables_initializer()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run(image_file)
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
THE ERROR IS
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.FailedPreconditionError'>, Attempting to use uninitialized value matching_filenames
[[Node: matching_filenames/read = IdentityT=DT_STRING, _class=["loc:@matching_filenames"], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
OutOfRangeError Traceback (most recent call last)
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1349 try:
-> 1350 return fn(*args)
1351 except errors.OpError as e:
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1328 feed_dict, fetch_list, target_list,
-> 1329 status, run_metadata)
1330
~/env3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg)
472 compat.as_text(c_api.TF_Message(self.status.status)),
--> 473 c_api.TF_GetCode(self.status.status))
474 # Delete the underlying status object from memory otherwise it stays alive
OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]
During handling of the above exception, another exception occurred:
OutOfRangeError Traceback (most recent call last)
in ()
8
9 # Get an image tensor and print its value.
---> 10 image_tensor = sess.run(image_file)
11 print(image_tensor)
12
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
893 try:
894 result = self._run(None, fetches, feed_dict, options_ptr,
--> 895 run_metadata_ptr)
896 if run_metadata:
897 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1126 if final_fetches or final_targets or (handle and feed_dict_tensor):
1127 results = self._do_run(handle, final_targets, final_fetches,
-> 1128 feed_dict_tensor, options, run_metadata)
1129 else:
1130 results = []
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1342 if handle is None:
1343 return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1344 options, run_metadata)
1345 else:
1346 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)
~/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1361 except KeyError:
1362 pass
-> 1363 raise type(e)(node_def, op, message)
1364
1365 def _extend_graph(self):
OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]
Caused by op 'ReaderReadV2', defined at:
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/iamnaren/env3/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 486, in start
self.io_loop.start()
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/ioloop.py", line 832, in start
self._run_callback(self._callbacks.popleft())
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
ret = callback()
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 536, in
self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
self._handle_recv()
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/iamnaren/env3/lib/python3.5/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2728, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2850, in run_ast_nodes
if self.run_code(code, result):
File "/home/iamnaren/env3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 4, in
_, image_file = image_reader.read(filename_queue)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/ops/io_ops.py", line 209, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 678, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
op_def=op_def)
File "/home/iamnaren/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](WholeFileReaderV2, input_producer)]]
I also had the same problem:
"OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)"
However, I found that the error did not occur when I changed
filename_queue = tf.train.string_input_producer ( tf.train.match_filenames_once ( IMAGE_FILE_PATH + '*.JPG' ) )
to
IMAGE_FILE_PATH = './images/'
filenames = [ os.path.join( IMAGE_FILE_PATH, filename ) for filename in os.listdir( IMAGE_FILE_PATH ) ]
filename_queue = tf.train.string_input_producer( filenames )
@Hariharanmayu
I also got the same issue!
Hi all, the code will work after you change the code from
filename_queue = tf.train.string_input_producer(
tf.train.match_filenames_once("./images/*.jpg"))
to
filename_queue = tf.train.string_input_producer(['./*.jpg'])
I donnot know why, so who can explain it.
filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once('./apple2orange/trainA/*.jpg'))
gives me error TypeError: Fetch argument <tensorflow.python.ops.data_flow_ops.FIFOQueue object at 0xb1a840ef0> has invalid type <class 'tensorflow.python.ops.data_flow_ops.FIFOQueue'>, must be a string or Tensor. (Can not convert a FIFOQueue into a Tensor or Operation.)
can anybody tell me why tf.train.match_filenames_once is giving me error of Can not convert a FIFOQueue into a Tensor or Operation?
how to solve this
Hi eerwitt,
It seems that the image_tensor only return an image. How do I get multiple image tensors?
Cheers,
Tianming