Skip to content

Instantly share code, notes, and snippets.

@enakai00

enakai00/TF2.6 GPU OOM error.txt

Created December 3, 2021 08:53
Show Gist options
  • Save enakai00/6479cfd90d734f613323b5a9071471c4 to your computer and use it in GitHub Desktop.
Save enakai00/6479cfd90d734f613323b5a9071471c4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
TF2.6 GPU OOM error.txt
model = train_and_evaluate(batch_size=32, lrate=0.0001, l1=0, l2=0, num_hidden=128)
=====
[Output]
2021-12-03 08:48:43.805276: W tensorflow/core/common_runtime/bfc_allocator.cc:457] Allocator (GPU_0_bfc) ran out of memory trying to allocate 73.50MiB (rounded to 77070336)requested by op Mul
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2021-12-03 08:48:43.805401: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] BFCAllocator dump for GPU_0_bfc
2021-12-03 08:48:43.805463: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (256): Total Chunks: 33, Chunks in use: 32. 8.2KiB allocated for chunks. 8.0KiB in use in bin. 217B client-requested in use in bin.
2021-12-03 08:48:43.805473: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (512): Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2021-12-03 08:48:43.805481: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-12-03 08:48:43.805489: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2048): Total Chunks: 3, Chunks in use: 3. 7.5KiB allocated for chunks. 7.5KiB in use in bin. 7.5KiB client-requested in use in bin.
2021-12-03 08:48:43.805497: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805505: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805529: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805537: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805544: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805552: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805559: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805567: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805574: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805582: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805589: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805597: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805605: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16777216): Total Chunks: 1, Chunks in use: 0. 18.39MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805613: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805622: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (67108864): Total Chunks: 2, Chunks in use: 2. 147.00MiB allocated for chunks. 147.00MiB in use in bin. 147.00MiB client-requested in use in bin.
2021-12-03 08:48:43.805631: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (134217728): Total Chunks: 2, Chunks in use: 2. 284.34MiB allocated for chunks. 284.34MiB in use in bin. 147.00MiB client-requested in use in bin.
2021-12-03 08:48:43.805639: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-12-03 08:48:43.805653: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Bin for 73.50MiB was 64.00MiB, Chunk State:
2021-12-03 08:48:43.805663: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 471597056
2021-12-03 08:48:43.805681: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000000 of size 1280 next 1
2021-12-03 08:48:43.805696: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000500 of size 256 next 2
2021-12-03 08:48:43.805706: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000600 of size 256 next 3
2021-12-03 08:48:43.805716: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000700 of size 256 next 4
2021-12-03 08:48:43.805725: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000800 of size 256 next 5
2021-12-03 08:48:43.805734: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000900 of size 256 next 6
2021-12-03 08:48:43.805752: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000a00 of size 256 next 7
2021-12-03 08:48:43.805762: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000b00 of size 512 next 8
2021-12-03 08:48:43.805771: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000d00 of size 256 next 11
2021-12-03 08:48:43.805781: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000e00 of size 256 next 12
2021-12-03 08:48:43.805791: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee000f00 of size 256 next 13
2021-12-03 08:48:43.805801: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001000 of size 256 next 14
2021-12-03 08:48:43.805810: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001100 of size 256 next 17
2021-12-03 08:48:43.805821: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001200 of size 256 next 18
2021-12-03 08:48:43.805831: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001300 of size 256 next 19
2021-12-03 08:48:43.805840: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001400 of size 256 next 20
2021-12-03 08:48:43.805850: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001500 of size 256 next 21
2021-12-03 08:48:43.805860: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001600 of size 256 next 22
2021-12-03 08:48:43.805871: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001700 of size 256 next 23
2021-12-03 08:48:43.805881: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001800 of size 256 next 24
2021-12-03 08:48:43.805891: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001900 of size 256 next 25
2021-12-03 08:48:43.805901: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001a00 of size 512 next 26
2021-12-03 08:48:43.805911: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001c00 of size 256 next 28
2021-12-03 08:48:43.805922: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001d00 of size 512 next 30
2021-12-03 08:48:43.805931: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee001f00 of size 256 next 32
2021-12-03 08:48:43.805941: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee002000 of size 256 next 33
2021-12-03 08:48:43.805951: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee002100 of size 256 next 34
2021-12-03 08:48:43.805961: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee002200 of size 256 next 35
2021-12-03 08:48:43.805971: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee002300 of size 256 next 15
2021-12-03 08:48:43.805990: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee002400 of size 2560 next 16
2021-12-03 08:48:43.806000: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55ee002e00 of size 154131712 next 9
2021-12-03 08:48:43.806011: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55f7300b00 of size 77070336 next 10
2021-12-03 08:48:43.806021: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55fbc80b00 of size 2560 next 27
2021-12-03 08:48:43.806032: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f55fbc81500 of size 77070336 next 29
2021-12-03 08:48:43.806042: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5600601500 of size 2560 next 31
2021-12-03 08:48:43.806052: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5600601f00 of size 256 next 36
2021-12-03 08:48:43.806062: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5600602000 of size 256 next 37
2021-12-03 08:48:43.806072: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5600602100 of size 256 next 38
2021-12-03 08:48:43.806082: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f5600602200 of size 19286528 next 48
2021-12-03 08:48:43.806092: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5601866c00 of size 256 next 49
2021-12-03 08:48:43.806102: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5601866d00 of size 256 next 39
2021-12-03 08:48:43.806112: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5601866e00 of size 256 next 52
2021-12-03 08:48:43.806123: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f5601866f00 of size 256 next 47
2021-12-03 08:48:43.806133: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5601867000 of size 256 next 50
2021-12-03 08:48:43.806146: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f5601867100 of size 144019200 next 18446744073709551615
2021-12-03 08:48:43.806156: I tensorflow/core/common_runtime/bfc_allocator.cc:1065] Summary of in-use Chunks by size:
2021-12-03 08:48:43.806169: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 32 Chunks of size 256 totalling 8.0KiB
2021-12-03 08:48:43.806181: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 512 totalling 1.5KiB
2021-12-03 08:48:43.806191: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 1280 totalling 1.2KiB
2021-12-03 08:48:43.806202: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 2560 totalling 7.5KiB
2021-12-03 08:48:43.806214: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 77070336 totalling 147.00MiB
2021-12-03 08:48:43.806226: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 144019200 totalling 137.35MiB
2021-12-03 08:48:43.806237: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 154131712 totalling 146.99MiB
2021-12-03 08:48:43.806248: I tensorflow/core/common_runtime/bfc_allocator.cc:1072] Sum Total of in-use chunks: 431.36MiB
2021-12-03 08:48:43.806259: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] total_region_allocated_bytes_: 471597056 memory_limit_: 471597056 available bytes: 0 curr_region_allocation_bytes_: 943194112
2021-12-03 08:48:43.806276: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] Stats:
Limit: 471597056
InUse: 452310272
MaxInUse: 471577088
NumAllocs: 39543
MaxAllocSize: 154131712
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2021-12-03 08:48:43.806291: W tensorflow/core/common_runtime/bfc_allocator.cc:468] *****************xxxxxxxxxxxxxxx**********************************___*****************xxxxxxxxxxxxxx
2021-12-03 08:48:43.806328: W tensorflow/core/framework/op_kernel.cc:1680] Resource exhausted: failed to allocate memory
---------------------------------------------------------------------------
ResourceExhaustedError Traceback (most recent call last)
/tmp/ipykernel_1490/2654878505.py in <module>
----> 1 model = train_and_evaluate(batch_size=32, lrate=0.0001, l1=0, l2=0, num_hidden=128)
/tmp/ipykernel_1490/1433601672.py in train_and_evaluate(batch_size, lrate, l1, l2, num_hidden)
23 tf.keras.layers.Dense(len(CLASS_NAMES),
24 kernel_regularizer=regularizer,
---> 25 activation='softmax')
26 ])
27 model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=lrate),
/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
528 self._self_setattr_tracking = False # pylint: disable=protected-access
529 try:
--> 530 result = method(self, *args, **kwargs)
531 finally:
532 self._self_setattr_tracking = previous_value # pylint: disable=protected-access
/opt/conda/lib/python3.7/site-packages/keras/engine/sequential.py in __init__(self, layers, name)
132 layers = [layers]
133 for layer in layers:
--> 134 self.add(layer)
135
136 @property
/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
528 self._self_setattr_tracking = False # pylint: disable=protected-access
529 try:
--> 530 result = method(self, *args, **kwargs)
531 finally:
532 self._self_setattr_tracking = previous_value # pylint: disable=protected-access
/opt/conda/lib/python3.7/site-packages/keras/engine/sequential.py in add(self, layer)
215 # If the model is being built continuously on top of an input layer:
216 # refresh its output.
--> 217 output_tensor = layer(self.outputs[0])
218 if len(tf.nest.flatten(output_tensor)) != 1:
219 raise ValueError(SINGLE_LAYER_OUTPUT_ERROR_MSG)
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
975 if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
976 return self._functional_construction_call(inputs, args, kwargs,
--> 977 input_list)
978
979 # Maintains info about the `Layer.call` stack.
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
1113 # Check input assumptions set after layer building, e.g. input shape.
1114 outputs = self._keras_tensor_symbolic_call(
-> 1115 inputs, input_masks, args, kwargs)
1116
1117 if outputs is None:
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
846 return tf.nest.map_structure(keras_tensor.KerasTensor, output_signature)
847 else:
--> 848 return self._infer_output_signature(inputs, args, kwargs, input_masks)
849
850 def _infer_output_signature(self, inputs, args, kwargs, input_masks):
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
884 # overridden).
885 # TODO(kaftan): do we maybe_build here, or have we already done it?
--> 886 self._maybe_build(inputs)
887 inputs = self._maybe_cast_inputs(inputs)
888 outputs = call_fn(inputs, *args, **kwargs)
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _maybe_build(self, inputs)
2657 # operations.
2658 with tf_utils.maybe_init_scope(self):
-> 2659 self.build(input_shapes) # pylint:disable=not-callable
2660 # We must set also ensure that the layer is marked as built, and the build
2661 # shape is stored since user defined build functions may not be calling
/opt/conda/lib/python3.7/site-packages/keras/layers/core.py in build(self, input_shape)
1183 constraint=self.kernel_constraint,
1184 dtype=self.dtype,
-> 1185 trainable=True)
1186 if self.use_bias:
1187 self.bias = self.add_weight(
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint, use_resource, synchronization, aggregation, **kwargs)
661 synchronization=synchronization,
662 aggregation=aggregation,
--> 663 caching_device=caching_device)
664 if regularizer is not None:
665 # TODO(fchollet): in the future, this should be handled at the
/opt/conda/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py in _add_variable_with_custom_getter(self, name, shape, dtype, initializer, getter, overwrite, **kwargs_for_getter)
816 dtype=dtype,
817 initializer=initializer,
--> 818 **kwargs_for_getter)
819
820 # If we set an initializer and the variable processed it, tracking will not
/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer_utils.py in make_variable(name, shape, dtype, initializer, trainable, caching_device, validate_shape, constraint, use_resource, collections, synchronization, aggregation, partitioner)
127 synchronization=synchronization,
128 aggregation=aggregation,
--> 129 shape=variable_shape if variable_shape else None)
130
131
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/variables.py in __call__(cls, *args, **kwargs)
264 def __call__(cls, *args, **kwargs):
265 if cls is VariableV1:
--> 266 return cls._variable_v1_call(*args, **kwargs)
267 elif cls is Variable:
268 return cls._variable_v2_call(*args, **kwargs)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/variables.py in _variable_v1_call(cls, initial_value, trainable, collections, validate_shape, caching_device, name, variable_def, dtype, expected_shape, import_scope, constraint, use_resource, synchronization, aggregation, shape)
225 synchronization=synchronization,
226 aggregation=aggregation,
--> 227 shape=shape)
228
229 def _variable_v2_call(cls,
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/variables.py in <lambda>(**kwargs)
203 shape=None):
204 """Call on Variable class. Useful to force the signature."""
--> 205 previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
206 for _, getter in ops.get_default_graph()._variable_creator_stack: # pylint: disable=protected-access
207 previous_getter = _make_getter(getter, previous_getter)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py in default_variable_creator(next_creator, **kwargs)
2624 synchronization=synchronization,
2625 aggregation=aggregation,
-> 2626 shape=shape)
2627 else:
2628 return variables.RefVariable(
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/variables.py in __call__(cls, *args, **kwargs)
268 return cls._variable_v2_call(*args, **kwargs)
269 else:
--> 270 return super(VariableMetaclass, cls).__call__(*args, **kwargs)
271
272
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py in __init__(self, initial_value, trainable, collections, validate_shape, caching_device, name, dtype, variable_def, import_scope, constraint, distribute_strategy, synchronization, aggregation, shape)
1611 aggregation=aggregation,
1612 shape=shape,
-> 1613 distribute_strategy=distribute_strategy)
1614
1615 def _init_from_args(self,
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py in _init_from_args(self, initial_value, trainable, collections, caching_device, name, dtype, constraint, synchronization, aggregation, distribute_strategy, shape)
1738 with ops.name_scope("Initializer"), device_context_manager(None):
1739 if init_from_fn:
-> 1740 initial_value = initial_value()
1741 if isinstance(initial_value, trackable.CheckpointInitialValue):
1742 self._maybe_initialize_trackable()
/opt/conda/lib/python3.7/site-packages/keras/initializers/initializers_v2.py in __call__(self, shape, dtype, **kwargs)
515 else:
516 limit = math.sqrt(3.0 * scale)
--> 517 return self._random_generator.random_uniform(shape, -limit, limit, dtype)
518
519 def get_config(self):
/opt/conda/lib/python3.7/site-packages/keras/initializers/initializers_v2.py in random_uniform(self, shape, minval, maxval, dtype)
971 op = tf.random.uniform
972 return op(
--> 973 shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
974
975 def truncated_normal(self, shape, mean, stddev, dtype):
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
204 """Call target, and fall back on dispatchers if there is a TypeError."""
205 try:
--> 206 return target(*args, **kwargs)
207 except (TypeError, ValueError):
208 # Note: convert_to_eager_tensor currently raises a ValueError, not a
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py in random_uniform(shape, minval, maxval, dtype, seed, name)
313 result = math_ops.multiply(result, maxval)
314 else:
--> 315 result = math_ops.add(result * (maxval - minval), minval, name=name)
316 # TODO(b/132092188): C++ shape inference inside functional ops does not
317 # cross FuncGraph boundaries since that information is only available in
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py in binary_op_wrapper(x, y)
1365 # r_binary_op_wrapper use different force_same_dtype values.
1366 x, y = maybe_promote_tensors(x, y, force_same_dtype=False)
-> 1367 return func(x, y, name=name)
1368 except (TypeError, ValueError) as e:
1369 # Even if dispatching the op failed, the RHS may be a tensor aware
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py in _mul_dispatch(x, y, name)
1708 return sparse_tensor.SparseTensor(y.indices, new_vals, y.dense_shape)
1709 else:
-> 1710 return multiply(x, y, name=name)
1711
1712
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py in wrapper(*args, **kwargs)
204 """Call target, and fall back on dispatchers if there is a TypeError."""
205 try:
--> 206 return target(*args, **kwargs)
207 except (TypeError, ValueError):
208 # Note: convert_to_eager_tensor currently raises a ValueError, not a
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py in multiply(x, y, name)
528 """
529
--> 530 return gen_math_ops.mul(x, y, name)
531
532
/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py in mul(x, y, name)
6234 return _result
6235 except _core._NotOkStatusException as e:
-> 6236 _ops.raise_from_not_ok_status(e, name)
6237 except _core._FallbackException:
6238 pass
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
6939 message = e.message + (" name: " + name if name is not None else "")
6940 # pylint: disable=protected-access
-> 6941 six.raise_from(core._status_to_exception(e.code, message), None)
6942 # pylint: enable=protected-access
6943
/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)
ResourceExhaustedError: failed to allocate memory [Op:Mul]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment