CTC label index out of bound error

CTC error

sathish@sathish-linux-deep:~/gitRepos/OCR/CRNN_Tensorflow$ python3 tools/train_shadownet.py --dataset_dir /home/sathish/gitRepos/OCR/datasets/SynthText/SynthText/tfrecords_dir/
/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
The dict_path -  /home/sathish/gitRepos/OCR/CRNN_Tensorflow/data/char_dict/char_dict.json
The dict_path -  /home/sathish/gitRepos/OCR/CRNN_Tensorflow/data/char_dict/char_dict.json
The labels read -  SparseTensor(indices=Tensor("ParseSingleExample/ParseSingleExample:0", shape=(?, 1), dtype=int64), values=Tensor("Cast:0", shape=(?,), dtype=int32), dense_shape=Tensor("ParseSingleExample/ParseSingleExample:2", shape=(1,), dtype=int64))
The image names -  Tensor("ParseSingleExample/ParseSingleExample:3", shape=(1,), dtype=string)
2018-06-29 10:09:08.875204: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-06-29 10:09:09.445772: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-06-29 10:09:09.446455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.607
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 10.75GiB
2018-06-29 10:09:09.446481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-29 10:09:09.619575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-29 10:09:09.619621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-06-29 10:09:09.619627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-06-29 10:09:09.619784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9496 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO: 06-29 10:09:09: train_shadownet.py:121 * 140532568536832 Training from scratch
The input labels -  SparseTensorValue(indices=array([[ 0,  0],
       [ 0,  1],
       [ 0,  2],
       [ 1,  0],
       [ 1,  1],
       [ 2,  0],
       [ 2,  1],
       [ 2,  2],
       [ 3,  0],
       [ 3,  1],
       [ 3,  2],
       [ 3,  3],
       [ 3,  4],
       [ 4,  0],
       [ 4,  1],
       [ 4,  2],
       [ 4,  3],
       [ 5,  0],
       [ 5,  1],
       [ 6,  0],
       [ 6,  1],
       [ 6,  2],
       [ 7,  0],
       [ 7,  1],
       [ 7,  2],
       [ 8,  0],
       [ 9,  0],
       [ 9,  1],
       [ 9,  2],
       [10,  0],
       [10,  1],
       [10,  2],
       [10,  3],
       [10,  4],
       [10,  5],
       [10,  6],
       [10,  7],
       [11,  0],
       [11,  1],
       [11,  2],
       [12,  0],
       [12,  1],
       [12,  2],
       [12,  3],
       [12,  4],
       [12,  5],
       [13,  0],
       [13,  1],
       [13,  2],
       [14,  0],
       [14,  1],
       [14,  2],
       [15,  0],
       [15,  1],
       [15,  2],
       [16,  0],
       [16,  1],
       [16,  2],
       [17,  0],
       [17,  1],
       [17,  2],
       [17,  3],
       [18,  0],
       [18,  1],
       [18,  2],
       [18,  3],
       [18,  4],
       [18,  5],
       [19,  0],
       [19,  1],
       [20,  0],
       [20,  1],
       [20,  2],
       [21,  0],
       [21,  1],
       [21,  2],
       [22,  0],
       [22,  1],
       [22,  2],
       [23,  0],
       [23,  1],
       [23,  2],
       [24,  0],
       [24,  1],
       [24,  2],
       [25,  0],
       [25,  1],
       [25,  2],
       [26,  0],
       [26,  1],
       [26,  2],
       [26,  3],
       [27,  0],
       [27,  1],
       [27,  2],
       [28,  0],
       [28,  1],
       [28,  2],
       [28,  3],
       [29,  0],
       [29,  1],
       [29,  2],
       [30,  0],
       [30,  1],
       [30,  2],
       [30,  3],
       [30,  4],
       [31,  0],
       [31,  1],
       [31,  2],
       [31,  3]]), values=array([22,  8, 13, 27, 34, 13, 14, 19,  3,  0, 19,  4, 45, 12,  0, 13, 24,
        0, 19, 19,  7,  4,  5, 14, 17,  8, 24, 14, 20,  0, 12,  4, 17,  8,
        2,  0, 13, 24, 14, 20, 11,  8, 13,  4, 18, 45, 12,  0,  6,  7,  8,
       18, 24, 14, 20,  6, 14, 19,  5,  4, 17, 13,  2,  8, 21,  8,  2, 18,
        1, 24,  5, 14, 17, 19, 14, 14,  0, 13,  3, 19,  7,  4,  0, 13,  3,
       10, 20, 46, 22,  8, 11, 11, 12,  0,  6,  0,  4, 17, 14, 19,  7,  4,
       11, 14, 14, 13, 24, 22,  8, 11, 11], dtype=int32), dense_shape=array([32,  8]))
The output labels -  (25, 32, 42)
2018-06-29 10:09:11.027875: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at ctc_loss_op.cc:166 : Invalid argument: Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 42 labels:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 42 labels:
         [[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shadow/LSTMLayers/transpose_time_major/_91, shuffle_batch/TakeManySparseFromTensorsMap, shuffle_batch/TakeManySparseFromTensorsMap:1, CTCLoss/sequence_length)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/train_shadownet.py", line 192, in <module>
    train_shadownet(args.dataset_dir, args.weights_path)
  File "tools/train_shadownet.py", line 136, in train_shadownet
    sess.run(optimizer)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 42 labels:
         [[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shadow/LSTMLayers/transpose_time_major/_91, shuffle_batch/TakeManySparseFromTensorsMap, shuffle_batch/TakeManySparseFromTensorsMap:1, CTCLoss/sequence_length)]]

Caused by op 'CTCLoss', defined at:
  File "tools/train_shadownet.py", line 192, in <module>
    train_shadownet(args.dataset_dir, args.weights_path)
  File "tools/train_shadownet.py", line 71, in train_shadownet
    cost = tf.reduce_mean(tf.nn.ctc_loss(labels=input_labels, inputs=net_out, sequence_length=25*np.ones(32)))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/ctc_ops.py", line 158, in ctc_loss
    ignore_longer_outputs_than_inputs=ignore_longer_outputs_than_inputs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_ctc_ops.py", line 285, in ctc_loss
    name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Saw a non-null label (index >= num_classes - 1) following a null label, batch: 10 num_classes: 42 labels:
         [[Node: CTCLoss = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shadow/LSTMLayers/transpose_time_major/_91, shuffle_batch/TakeManySparseFromTensorsMap, shuffle_batch/TakeManySparseFromTensorsMap:1, CTCLoss/sequence_length)]]

sathish@sathish-linux-deep:~/gitRepos/OCR/CRNN_Tensorflow$