stdout.txt

(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:33:55.742005: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:33:55.747631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:33:55.747656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:33:55.747662: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.696012: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.701059: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.701437: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/ray/anaconda3/lib/python3.8/site-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.716080: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.716297: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.716317: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
(RayTrainWorker pid=743, ip=172.31.221.11) Skipping registering GPU devices...
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.719398: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
(RayTrainWorker pid=743, ip=172.31.221.11) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.721129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.721146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.721153: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.725123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.725146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.725153: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.745957: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 172.31.221.11:50369, 1 -> 172.31.210.32:52713}
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:33:55.746629: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:411] Started server with target: grpc://172.31.221.11:50369
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:33:55.773961: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 172.31.221.11:50369, 1 -> 172.31.210.32:52713}
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:33:55.774661: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:411] Started server with target: grpc://172.31.210.32:52713
(RayTrainWorker pid=673, ip=172.31.210.32) num of views 5
(RayTrainWorker pid=673, ip=172.31.210.32) backbone choice mobilenet_v3_small
(RayTrainWorker pid=743, ip=172.31.221.11) num of views 5
(RayTrainWorker pid=743, ip=172.31.221.11) backbone choice mobilenet_v3_small
(scheduler +2m16s) Resized to 72 CPUs, 2 GPUs.
(RayTrainWorker pid=673, ip=172.31.210.32) 0 input_3 KerasTensor(type_spec=TensorSpec(shape=(None, 260, 260, 3), dtype=tf.float32, name='input_3'), name='input_3', description="created by layer 'input_3'") (None, 260, 260, 3)
(RayTrainWorker pid=673, ip=172.31.210.32) 1 Conv0 KerasTensor(type_spec=TensorSpec(shape=(None, 130, 130, 16), dtype=tf.float32, name=None), name='Conv0/Conv2D:0', description="created by layer 'Conv0'") (None, 130, 130, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 2 Conv/BatchNorm0 KerasTensor(type_spec=TensorSpec(shape=(None, 130, 130, 16), dtype=tf.float32, name=None), name='Conv/BatchNorm0/FusedBatchNormV3:0', description="created by layer 'Conv/BatchNorm0'") (None, 130, 130, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 3 tf.__operators__.add KerasTensor(type_spec=TensorSpec(shape=(None, 130, 130, 16), dtype=tf.float32, name=None), name='tf.__operators__.add/AddV2:0', description="created by layer 'tf.__operators__.add'") (None, 130, 130, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 4 re_lu KerasTensor(type_spec=TensorSpec(shape=(None, 130, 130, 16), dtype=tf.float32, name=None), name='re_lu/Relu6:0', description="created by layer 're_lu'") (None, 130, 130, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 5 tf.math.multiply KerasTensor(type_spec=TensorSpec(shape=(None, 130, 130, 16), dtype=tf.float32, name=None), name='tf.math.multiply/Mul:0', description="created by layer 'tf.math.multiply'") (None, 130, 130, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 6 multiply KerasTensor(type_spec=TensorSpec(shape=(None, 130, 130, 16), dtype=tf.float32, name=None), name='multiply/mul:0', description="created by layer 'multiply'") (None, 130, 130, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 7 expanded_conv/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 16), dtype=tf.float32, name=None), name='expanded_conv/depthwise/depthwise:0', description="created by layer 'expanded_conv/depthwise'") (None, 65, 65, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 8 expanded_conv/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 16), dtype=tf.float32, name=None), name='expanded_conv/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv/depthwise/BatchNorm'") (None, 65, 65, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 9 re_lu_1 KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 16), dtype=tf.float32, name=None), name='re_lu_1/Relu:0', description="created by layer 're_lu_1'") (None, 65, 65, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 10 expanded_conv/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 16), dtype=tf.float32, name=None), name='expanded_conv/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv/squeeze_excite/AvgPool'") (None, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 11 reshape KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 16), dtype=tf.float32, name=None), name='reshape/Reshape:0', description="created by layer 'reshape'") (None, 1, 1, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 12 expanded_conv/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 8), dtype=tf.float32, name=None), name='expanded_conv/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv/squeeze_excite/Conv'") (None, 1, 1, 8)
(RayTrainWorker pid=673, ip=172.31.210.32) 13 expanded_conv/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 8), dtype=tf.float32, name=None), name='expanded_conv/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv/squeeze_excite/Relu'") (None, 1, 1, 8)
(RayTrainWorker pid=673, ip=172.31.210.32) 14 expanded_conv/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 16), dtype=tf.float32, name=None), name='expanded_conv/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv/squeeze_excite/Conv_1'") (None, 1, 1, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 15 tf.__operators__.add_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 16), dtype=tf.float32, name=None), name='tf.__operators__.add_1/AddV2:0', description="created by layer 'tf.__operators__.add_1'") (None, 1, 1, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 16 re_lu_2 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 16), dtype=tf.float32, name=None), name='re_lu_2/Relu6:0', description="created by layer 're_lu_2'") (None, 1, 1, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 17 tf.math.multiply_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 16), dtype=tf.float32, name=None), name='tf.math.multiply_1/Mul:0', description="created by layer 'tf.math.multiply_1'") (None, 1, 1, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 18 expanded_conv/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 16), dtype=tf.float32, name=None), name='expanded_conv/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv/squeeze_excite/Mul'") (None, 65, 65, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 19 expanded_conv/project KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 16), dtype=tf.float32, name=None), name='expanded_conv/project/Conv2D:0', description="created by layer 'expanded_conv/project'") (None, 65, 65, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 20 expanded_conv/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 16), dtype=tf.float32, name=None), name='expanded_conv/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv/project/BatchNorm'") (None, 65, 65, 16)
(RayTrainWorker pid=673, ip=172.31.210.32) 21 expanded_conv_1/expand KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 72), dtype=tf.float32, name=None), name='expanded_conv_1/expand/Conv2D:0', description="created by layer 'expanded_conv_1/expand'") (None, 65, 65, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 22 expanded_conv_1/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 72), dtype=tf.float32, name=None), name='expanded_conv_1/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_1/expand/BatchNorm'") (None, 65, 65, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 23 re_lu_3 KerasTensor(type_spec=TensorSpec(shape=(None, 65, 65, 72), dtype=tf.float32, name=None), name='re_lu_3/Relu:0', description="created by layer 're_lu_3'") (None, 65, 65, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 24 expanded_conv_1/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 72), dtype=tf.float32, name=None), name='expanded_conv_1/depthwise/depthwise:0', description="created by layer 'expanded_conv_1/depthwise'") (None, 33, 33, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 25 expanded_conv_1/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 72), dtype=tf.float32, name=None), name='expanded_conv_1/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_1/depthwise/BatchNorm'") (None, 33, 33, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 26 re_lu_4 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 72), dtype=tf.float32, name=None), name='re_lu_4/Relu:0', description="created by layer 're_lu_4'") (None, 33, 33, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 27 expanded_conv_1/project KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 24), dtype=tf.float32, name=None), name='expanded_conv_1/project/Conv2D:0', description="created by layer 'expanded_conv_1/project'") (None, 33, 33, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 28 expanded_conv_1/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 24), dtype=tf.float32, name=None), name='expanded_conv_1/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_1/project/BatchNorm'") (None, 33, 33, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 29 expanded_conv_2/expand KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 88), dtype=tf.float32, name=None), name='expanded_conv_2/expand/Conv2D:0', description="created by layer 'expanded_conv_2/expand'") (None, 33, 33, 88)
(RayTrainWorker pid=673, ip=172.31.210.32) 30 expanded_conv_2/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 88), dtype=tf.float32, name=None), name='expanded_conv_2/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_2/expand/BatchNorm'") (None, 33, 33, 88)
(RayTrainWorker pid=673, ip=172.31.210.32) 31 re_lu_5 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 88), dtype=tf.float32, name=None), name='re_lu_5/Relu:0', description="created by layer 're_lu_5'") (None, 33, 33, 88)
(RayTrainWorker pid=673, ip=172.31.210.32) 32 expanded_conv_2/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 88), dtype=tf.float32, name=None), name='expanded_conv_2/depthwise/depthwise:0', description="created by layer 'expanded_conv_2/depthwise'") (None, 33, 33, 88)
(RayTrainWorker pid=673, ip=172.31.210.32) 33 expanded_conv_2/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 88), dtype=tf.float32, name=None), name='expanded_conv_2/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_2/depthwise/BatchNorm'") (None, 33, 33, 88)
(RayTrainWorker pid=673, ip=172.31.210.32) 34 re_lu_6 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 88), dtype=tf.float32, name=None), name='re_lu_6/Relu:0', description="created by layer 're_lu_6'") (None, 33, 33, 88)
(RayTrainWorker pid=673, ip=172.31.210.32) 35 expanded_conv_2/project KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 24), dtype=tf.float32, name=None), name='expanded_conv_2/project/Conv2D:0', description="created by layer 'expanded_conv_2/project'") (None, 33, 33, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 36 expanded_conv_2/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 24), dtype=tf.float32, name=None), name='expanded_conv_2/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_2/project/BatchNorm'") (None, 33, 33, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 37 expanded_conv_2/Add KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 24), dtype=tf.float32, name=None), name='expanded_conv_2/Add/add:0', description="created by layer 'expanded_conv_2/Add'") (None, 33, 33, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 38 expanded_conv_3/expand KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 96), dtype=tf.float32, name=None), name='expanded_conv_3/expand/Conv2D:0', description="created by layer 'expanded_conv_3/expand'") (None, 33, 33, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 39 expanded_conv_3/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 96), dtype=tf.float32, name=None), name='expanded_conv_3/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_3/expand/BatchNorm'") (None, 33, 33, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 40 tf.__operators__.add_2 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 96), dtype=tf.float32, name=None), name='tf.__operators__.add_2/AddV2:0', description="created by layer 'tf.__operators__.add_2'") (None, 33, 33, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 41 re_lu_7 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 96), dtype=tf.float32, name=None), name='re_lu_7/Relu6:0', description="created by layer 're_lu_7'") (None, 33, 33, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 42 tf.math.multiply_2 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 96), dtype=tf.float32, name=None), name='tf.math.multiply_2/Mul:0', description="created by layer 'tf.math.multiply_2'") (None, 33, 33, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 43 multiply_1 KerasTensor(type_spec=TensorSpec(shape=(None, 33, 33, 96), dtype=tf.float32, name=None), name='multiply_1/mul:0', description="created by layer 'multiply_1'") (None, 33, 33, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 44 expanded_conv_3/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='expanded_conv_3/depthwise/depthwise:0', description="created by layer 'expanded_conv_3/depthwise'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 45 expanded_conv_3/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='expanded_conv_3/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_3/depthwise/BatchNorm'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 46 tf.__operators__.add_3 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='tf.__operators__.add_3/AddV2:0', description="created by layer 'tf.__operators__.add_3'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 47 re_lu_8 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='re_lu_8/Relu6:0', description="created by layer 're_lu_8'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 48 tf.math.multiply_3 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='tf.math.multiply_3/Mul:0', description="created by layer 'tf.math.multiply_3'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 49 multiply_2 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='multiply_2/mul:0', description="created by layer 'multiply_2'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 50 expanded_conv_3/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 96), dtype=tf.float32, name=None), name='expanded_conv_3/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_3/squeeze_excite/AvgPool'") (None, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 51 reshape_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 96), dtype=tf.float32, name=None), name='reshape_1/Reshape:0', description="created by layer 'reshape_1'") (None, 1, 1, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 52 expanded_conv_3/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 24), dtype=tf.float32, name=None), name='expanded_conv_3/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_3/squeeze_excite/Conv'") (None, 1, 1, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 53 expanded_conv_3/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 24), dtype=tf.float32, name=None), name='expanded_conv_3/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_3/squeeze_excite/Relu'") (None, 1, 1, 24)
(RayTrainWorker pid=673, ip=172.31.210.32) 54 expanded_conv_3/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 96), dtype=tf.float32, name=None), name='expanded_conv_3/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_3/squeeze_excite/Conv_1'") (None, 1, 1, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 55 tf.__operators__.add_4 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 96), dtype=tf.float32, name=None), name='tf.__operators__.add_4/AddV2:0', description="created by layer 'tf.__operators__.add_4'") (None, 1, 1, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 56 re_lu_9 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 96), dtype=tf.float32, name=None), name='re_lu_9/Relu6:0', description="created by layer 're_lu_9'") (None, 1, 1, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 57 tf.math.multiply_4 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 96), dtype=tf.float32, name=None), name='tf.math.multiply_4/Mul:0', description="created by layer 'tf.math.multiply_4'") (None, 1, 1, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 58 expanded_conv_3/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 96), dtype=tf.float32, name=None), name='expanded_conv_3/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_3/squeeze_excite/Mul'") (None, 17, 17, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 59 expanded_conv_3/project KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_3/project/Conv2D:0', description="created by layer 'expanded_conv_3/project'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 60 expanded_conv_3/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_3/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_3/project/BatchNorm'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 61 expanded_conv_4/expand KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_4/expand/Conv2D:0', description="created by layer 'expanded_conv_4/expand'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 62 expanded_conv_4/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_4/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_4/expand/BatchNorm'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 63 tf.__operators__.add_5 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.__operators__.add_5/AddV2:0', description="created by layer 'tf.__operators__.add_5'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 64 re_lu_10 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='re_lu_10/Relu6:0', description="created by layer 're_lu_10'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 65 tf.math.multiply_5 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.math.multiply_5/Mul:0', description="created by layer 'tf.math.multiply_5'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 66 multiply_3 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='multiply_3/mul:0', description="created by layer 'multiply_3'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 67 expanded_conv_4/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_4/depthwise/depthwise:0', description="created by layer 'expanded_conv_4/depthwise'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 68 expanded_conv_4/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_4/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_4/depthwise/BatchNorm'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 69 tf.__operators__.add_6 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.__operators__.add_6/AddV2:0', description="created by layer 'tf.__operators__.add_6'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 70 re_lu_11 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='re_lu_11/Relu6:0', description="created by layer 're_lu_11'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 71 tf.math.multiply_6 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.math.multiply_6/Mul:0', description="created by layer 'tf.math.multiply_6'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 72 multiply_4 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='multiply_4/mul:0', description="created by layer 'multiply_4'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 73 expanded_conv_4/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 240), dtype=tf.float32, name=None), name='expanded_conv_4/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_4/squeeze_excite/AvgPool'") (None, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 74 reshape_2 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='reshape_2/Reshape:0', description="created by layer 'reshape_2'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 75 expanded_conv_4/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 64), dtype=tf.float32, name=None), name='expanded_conv_4/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_4/squeeze_excite/Conv'") (None, 1, 1, 64)
(RayTrainWorker pid=673, ip=172.31.210.32) 76 expanded_conv_4/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 64), dtype=tf.float32, name=None), name='expanded_conv_4/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_4/squeeze_excite/Relu'") (None, 1, 1, 64)
(RayTrainWorker pid=673, ip=172.31.210.32) 77 expanded_conv_4/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='expanded_conv_4/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_4/squeeze_excite/Conv_1'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 78 tf.__operators__.add_7 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='tf.__operators__.add_7/AddV2:0', description="created by layer 'tf.__operators__.add_7'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 79 re_lu_12 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='re_lu_12/Relu6:0', description="created by layer 're_lu_12'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 80 tf.math.multiply_7 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='tf.math.multiply_7/Mul:0', description="created by layer 'tf.math.multiply_7'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 81 expanded_conv_4/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_4/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_4/squeeze_excite/Mul'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 82 expanded_conv_4/project KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_4/project/Conv2D:0', description="created by layer 'expanded_conv_4/project'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 83 expanded_conv_4/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_4/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_4/project/BatchNorm'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 84 expanded_conv_4/Add KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_4/Add/add:0', description="created by layer 'expanded_conv_4/Add'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 85 expanded_conv_5/expand KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_5/expand/Conv2D:0', description="created by layer 'expanded_conv_5/expand'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 86 expanded_conv_5/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_5/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_5/expand/BatchNorm'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 87 tf.__operators__.add_8 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.__operators__.add_8/AddV2:0', description="created by layer 'tf.__operators__.add_8'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 88 re_lu_13 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='re_lu_13/Relu6:0', description="created by layer 're_lu_13'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 89 tf.math.multiply_8 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.math.multiply_8/Mul:0', description="created by layer 'tf.math.multiply_8'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 90 multiply_5 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='multiply_5/mul:0', description="created by layer 'multiply_5'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 91 expanded_conv_5/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_5/depthwise/depthwise:0', description="created by layer 'expanded_conv_5/depthwise'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 92 expanded_conv_5/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_5/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_5/depthwise/BatchNorm'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 93 tf.__operators__.add_9 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.__operators__.add_9/AddV2:0', description="created by layer 'tf.__operators__.add_9'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 94 re_lu_14 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='re_lu_14/Relu6:0', description="created by layer 're_lu_14'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 95 tf.math.multiply_9 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='tf.math.multiply_9/Mul:0', description="created by layer 'tf.math.multiply_9'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 96 multiply_6 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='multiply_6/mul:0', description="created by layer 'multiply_6'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 97 expanded_conv_5/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 240), dtype=tf.float32, name=None), name='expanded_conv_5/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_5/squeeze_excite/AvgPool'") (None, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 98 reshape_3 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='reshape_3/Reshape:0', description="created by layer 'reshape_3'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 99 expanded_conv_5/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 64), dtype=tf.float32, name=None), name='expanded_conv_5/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_5/squeeze_excite/Conv'") (None, 1, 1, 64)
(RayTrainWorker pid=673, ip=172.31.210.32) 100 expanded_conv_5/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 64), dtype=tf.float32, name=None), name='expanded_conv_5/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_5/squeeze_excite/Relu'") (None, 1, 1, 64)
(RayTrainWorker pid=673, ip=172.31.210.32) 101 expanded_conv_5/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='expanded_conv_5/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_5/squeeze_excite/Conv_1'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 102 tf.__operators__.add_10 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='tf.__operators__.add_10/AddV2:0', description="created by layer 'tf.__operators__.add_10'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 103 re_lu_15 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='re_lu_15/Relu6:0', description="created by layer 're_lu_15'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 104 tf.math.multiply_10 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 240), dtype=tf.float32, name=None), name='tf.math.multiply_10/Mul:0', description="created by layer 'tf.math.multiply_10'") (None, 1, 1, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 105 expanded_conv_5/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 240), dtype=tf.float32, name=None), name='expanded_conv_5/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_5/squeeze_excite/Mul'") (None, 17, 17, 240)
(RayTrainWorker pid=673, ip=172.31.210.32) 106 expanded_conv_5/project KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_5/project/Conv2D:0', description="created by layer 'expanded_conv_5/project'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 107 expanded_conv_5/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_5/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_5/project/BatchNorm'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 108 expanded_conv_5/Add KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 40), dtype=tf.float32, name=None), name='expanded_conv_5/Add/add:0', description="created by layer 'expanded_conv_5/Add'") (None, 17, 17, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 109 expanded_conv_6/expand KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='expanded_conv_6/expand/Conv2D:0', description="created by layer 'expanded_conv_6/expand'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 110 expanded_conv_6/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='expanded_conv_6/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_6/expand/BatchNorm'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 111 tf.__operators__.add_11 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='tf.__operators__.add_11/AddV2:0', description="created by layer 'tf.__operators__.add_11'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 112 re_lu_16 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='re_lu_16/Relu6:0', description="created by layer 're_lu_16'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 113 tf.math.multiply_11 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='tf.math.multiply_11/Mul:0', description="created by layer 'tf.math.multiply_11'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 114 multiply_7 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='multiply_7/mul:0', description="created by layer 'multiply_7'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 115 expanded_conv_6/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='expanded_conv_6/depthwise/depthwise:0', description="created by layer 'expanded_conv_6/depthwise'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 116 expanded_conv_6/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='expanded_conv_6/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_6/depthwise/BatchNorm'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 117 tf.__operators__.add_12 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='tf.__operators__.add_12/AddV2:0', description="created by layer 'tf.__operators__.add_12'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 118 re_lu_17 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='re_lu_17/Relu6:0', description="created by layer 're_lu_17'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 119 tf.math.multiply_12 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='tf.math.multiply_12/Mul:0', description="created by layer 'tf.math.multiply_12'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 120 multiply_8 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='multiply_8/mul:0', description="created by layer 'multiply_8'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 121 expanded_conv_6/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 120), dtype=tf.float32, name=None), name='expanded_conv_6/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_6/squeeze_excite/AvgPool'") (None, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 122 reshape_4 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 120), dtype=tf.float32, name=None), name='reshape_4/Reshape:0', description="created by layer 'reshape_4'") (None, 1, 1, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 123 expanded_conv_6/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 32), dtype=tf.float32, name=None), name='expanded_conv_6/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_6/squeeze_excite/Conv'") (None, 1, 1, 32)
(RayTrainWorker pid=673, ip=172.31.210.32) 124 expanded_conv_6/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 32), dtype=tf.float32, name=None), name='expanded_conv_6/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_6/squeeze_excite/Relu'") (None, 1, 1, 32)
(RayTrainWorker pid=673, ip=172.31.210.32) 125 expanded_conv_6/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 120), dtype=tf.float32, name=None), name='expanded_conv_6/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_6/squeeze_excite/Conv_1'") (None, 1, 1, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 126 tf.__operators__.add_13 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 120), dtype=tf.float32, name=None), name='tf.__operators__.add_13/AddV2:0', description="created by layer 'tf.__operators__.add_13'") (None, 1, 1, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 127 re_lu_18 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 120), dtype=tf.float32, name=None), name='re_lu_18/Relu6:0', description="created by layer 're_lu_18'") (None, 1, 1, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 128 tf.math.multiply_13 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 120), dtype=tf.float32, name=None), name='tf.math.multiply_13/Mul:0', description="created by layer 'tf.math.multiply_13'") (None, 1, 1, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 129 expanded_conv_6/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 120), dtype=tf.float32, name=None), name='expanded_conv_6/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_6/squeeze_excite/Mul'") (None, 17, 17, 120)
(RayTrainWorker pid=673, ip=172.31.210.32) 130 expanded_conv_6/project KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 48), dtype=tf.float32, name=None), name='expanded_conv_6/project/Conv2D:0', description="created by layer 'expanded_conv_6/project'") (None, 17, 17, 48)
(RayTrainWorker pid=673, ip=172.31.210.32) 131 expanded_conv_6/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 48), dtype=tf.float32, name=None), name='expanded_conv_6/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_6/project/BatchNorm'") (None, 17, 17, 48)
(RayTrainWorker pid=673, ip=172.31.210.32) 132 expanded_conv_7/expand KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='expanded_conv_7/expand/Conv2D:0', description="created by layer 'expanded_conv_7/expand'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 133 expanded_conv_7/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='expanded_conv_7/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_7/expand/BatchNorm'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 134 tf.__operators__.add_14 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='tf.__operators__.add_14/AddV2:0', description="created by layer 'tf.__operators__.add_14'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 135 re_lu_19 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='re_lu_19/Relu6:0', description="created by layer 're_lu_19'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 136 tf.math.multiply_14 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='tf.math.multiply_14/Mul:0', description="created by layer 'tf.math.multiply_14'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 137 multiply_9 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='multiply_9/mul:0', description="created by layer 'multiply_9'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 138 expanded_conv_7/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='expanded_conv_7/depthwise/depthwise:0', description="created by layer 'expanded_conv_7/depthwise'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 139 expanded_conv_7/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='expanded_conv_7/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_7/depthwise/BatchNorm'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 140 tf.__operators__.add_15 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='tf.__operators__.add_15/AddV2:0', description="created by layer 'tf.__operators__.add_15'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 141 re_lu_20 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='re_lu_20/Relu6:0', description="created by layer 're_lu_20'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 142 tf.math.multiply_15 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='tf.math.multiply_15/Mul:0', description="created by layer 'tf.math.multiply_15'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 143 multiply_10 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='multiply_10/mul:0', description="created by layer 'multiply_10'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 144 expanded_conv_7/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 144), dtype=tf.float32, name=None), name='expanded_conv_7/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_7/squeeze_excite/AvgPool'") (None, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 145 reshape_5 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='reshape_5/Reshape:0', description="created by layer 'reshape_5'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 146 expanded_conv_7/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 40), dtype=tf.float32, name=None), name='expanded_conv_7/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_7/squeeze_excite/Conv'") (None, 1, 1, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 147 expanded_conv_7/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 40), dtype=tf.float32, name=None), name='expanded_conv_7/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_7/squeeze_excite/Relu'") (None, 1, 1, 40)
(RayTrainWorker pid=673, ip=172.31.210.32) 148 expanded_conv_7/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='expanded_conv_7/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_7/squeeze_excite/Conv_1'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 149 tf.__operators__.add_16 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='tf.__operators__.add_16/AddV2:0', description="created by layer 'tf.__operators__.add_16'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 150 re_lu_21 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='re_lu_21/Relu6:0', description="created by layer 're_lu_21'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 151 tf.math.multiply_16 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='tf.math.multiply_16/Mul:0', description="created by layer 'tf.math.multiply_16'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 152 expanded_conv_7/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 144), dtype=tf.float32, name=None), name='expanded_conv_7/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_7/squeeze_excite/Mul'") (None, 17, 17, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 153 expanded_conv_7/project KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 48), dtype=tf.float32, name=None), name='expanded_conv_7/project/Conv2D:0', description="created by layer 'expanded_conv_7/project'") (None, 17, 17, 48)
(RayTrainWorker pid=673, ip=172.31.210.32) 154 expanded_conv_7/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 48), dtype=tf.float32, name=None), name='expanded_conv_7/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_7/project/BatchNorm'") (None, 17, 17, 48)
(RayTrainWorker pid=673, ip=172.31.210.32) 155 expanded_conv_7/Add KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 48), dtype=tf.float32, name=None), name='expanded_conv_7/Add/add:0', description="created by layer 'expanded_conv_7/Add'") (None, 17, 17, 48)
(RayTrainWorker pid=673, ip=172.31.210.32) 156 expanded_conv_8/expand KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 288), dtype=tf.float32, name=None), name='expanded_conv_8/expand/Conv2D:0', description="created by layer 'expanded_conv_8/expand'") (None, 17, 17, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 157 expanded_conv_8/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 288), dtype=tf.float32, name=None), name='expanded_conv_8/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_8/expand/BatchNorm'") (None, 17, 17, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 158 tf.__operators__.add_17 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 288), dtype=tf.float32, name=None), name='tf.__operators__.add_17/AddV2:0', description="created by layer 'tf.__operators__.add_17'") (None, 17, 17, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 159 re_lu_22 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 288), dtype=tf.float32, name=None), name='re_lu_22/Relu6:0', description="created by layer 're_lu_22'") (None, 17, 17, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 160 tf.math.multiply_17 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 288), dtype=tf.float32, name=None), name='tf.math.multiply_17/Mul:0', description="created by layer 'tf.math.multiply_17'") (None, 17, 17, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 161 multiply_11 KerasTensor(type_spec=TensorSpec(shape=(None, 17, 17, 288), dtype=tf.float32, name=None), name='multiply_11/mul:0', description="created by layer 'multiply_11'") (None, 17, 17, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 162 expanded_conv_8/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='expanded_conv_8/depthwise/depthwise:0', description="created by layer 'expanded_conv_8/depthwise'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 163 expanded_conv_8/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='expanded_conv_8/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_8/depthwise/BatchNorm'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 164 tf.__operators__.add_18 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='tf.__operators__.add_18/AddV2:0', description="created by layer 'tf.__operators__.add_18'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 165 re_lu_23 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='re_lu_23/Relu6:0', description="created by layer 're_lu_23'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 166 tf.math.multiply_18 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='tf.math.multiply_18/Mul:0', description="created by layer 'tf.math.multiply_18'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 167 multiply_12 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='multiply_12/mul:0', description="created by layer 'multiply_12'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 168 expanded_conv_8/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 288), dtype=tf.float32, name=None), name='expanded_conv_8/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_8/squeeze_excite/AvgPool'") (None, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 169 reshape_6 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 288), dtype=tf.float32, name=None), name='reshape_6/Reshape:0', description="created by layer 'reshape_6'") (None, 1, 1, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 170 expanded_conv_8/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 72), dtype=tf.float32, name=None), name='expanded_conv_8/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_8/squeeze_excite/Conv'") (None, 1, 1, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 171 expanded_conv_8/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 72), dtype=tf.float32, name=None), name='expanded_conv_8/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_8/squeeze_excite/Relu'") (None, 1, 1, 72)
(RayTrainWorker pid=673, ip=172.31.210.32) 172 expanded_conv_8/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 288), dtype=tf.float32, name=None), name='expanded_conv_8/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_8/squeeze_excite/Conv_1'") (None, 1, 1, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 173 tf.__operators__.add_19 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 288), dtype=tf.float32, name=None), name='tf.__operators__.add_19/AddV2:0', description="created by layer 'tf.__operators__.add_19'") (None, 1, 1, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 174 re_lu_24 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 288), dtype=tf.float32, name=None), name='re_lu_24/Relu6:0', description="created by layer 're_lu_24'") (None, 1, 1, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 175 tf.math.multiply_19 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 288), dtype=tf.float32, name=None), name='tf.math.multiply_19/Mul:0', description="created by layer 'tf.math.multiply_19'") (None, 1, 1, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 176 expanded_conv_8/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 288), dtype=tf.float32, name=None), name='expanded_conv_8/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_8/squeeze_excite/Mul'") (None, 9, 9, 288)
(RayTrainWorker pid=673, ip=172.31.210.32) 177 expanded_conv_8/project KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_8/project/Conv2D:0', description="created by layer 'expanded_conv_8/project'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 178 expanded_conv_8/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_8/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_8/project/BatchNorm'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 179 expanded_conv_9/expand KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_9/expand/Conv2D:0', description="created by layer 'expanded_conv_9/expand'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 180 expanded_conv_9/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_9/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_9/expand/BatchNorm'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 181 tf.__operators__.add_20 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_20/AddV2:0', description="created by layer 'tf.__operators__.add_20'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 182 re_lu_25 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='re_lu_25/Relu6:0', description="created by layer 're_lu_25'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 183 tf.math.multiply_20 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.math.multiply_20/Mul:0', description="created by layer 'tf.math.multiply_20'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 184 multiply_13 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='multiply_13/mul:0', description="created by layer 'multiply_13'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 185 expanded_conv_9/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_9/depthwise/depthwise:0', description="created by layer 'expanded_conv_9/depthwise'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 186 expanded_conv_9/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_9/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_9/depthwise/BatchNorm'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 187 tf.__operators__.add_21 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_21/AddV2:0', description="created by layer 'tf.__operators__.add_21'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 188 re_lu_26 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='re_lu_26/Relu6:0', description="created by layer 're_lu_26'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 189 tf.math.multiply_21 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.math.multiply_21/Mul:0', description="created by layer 'tf.math.multiply_21'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 190 multiply_14 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='multiply_14/mul:0', description="created by layer 'multiply_14'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 191 expanded_conv_9/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 576), dtype=tf.float32, name=None), name='expanded_conv_9/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_9/squeeze_excite/AvgPool'") (None, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 192 reshape_7 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='reshape_7/Reshape:0', description="created by layer 'reshape_7'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 193 expanded_conv_9/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='expanded_conv_9/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_9/squeeze_excite/Conv'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 194 expanded_conv_9/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='expanded_conv_9/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_9/squeeze_excite/Relu'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 195 expanded_conv_9/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='expanded_conv_9/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_9/squeeze_excite/Conv_1'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 196 tf.__operators__.add_22 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_22/AddV2:0', description="created by layer 'tf.__operators__.add_22'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 197 re_lu_27 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='re_lu_27/Relu6:0', description="created by layer 're_lu_27'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 198 tf.math.multiply_22 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='tf.math.multiply_22/Mul:0', description="created by layer 'tf.math.multiply_22'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 199 expanded_conv_9/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_9/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_9/squeeze_excite/Mul'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 200 expanded_conv_9/project KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_9/project/Conv2D:0', description="created by layer 'expanded_conv_9/project'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 201 expanded_conv_9/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_9/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_9/project/BatchNorm'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 202 expanded_conv_9/Add KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_9/Add/add:0', description="created by layer 'expanded_conv_9/Add'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 203 expanded_conv_10/expand KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_10/expand/Conv2D:0', description="created by layer 'expanded_conv_10/expand'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 204 expanded_conv_10/expand/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_10/expand/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_10/expand/BatchNorm'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 205 tf.__operators__.add_23 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_23/AddV2:0', description="created by layer 'tf.__operators__.add_23'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 206 re_lu_28 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='re_lu_28/Relu6:0', description="created by layer 're_lu_28'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 207 tf.math.multiply_23 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.math.multiply_23/Mul:0', description="created by layer 'tf.math.multiply_23'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 208 multiply_15 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='multiply_15/mul:0', description="created by layer 'multiply_15'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 209 expanded_conv_10/depthwise KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_10/depthwise/depthwise:0', description="created by layer 'expanded_conv_10/depthwise'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 210 expanded_conv_10/depthwise/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_10/depthwise/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_10/depthwise/BatchNorm'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 211 tf.__operators__.add_24 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_24/AddV2:0', description="created by layer 'tf.__operators__.add_24'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 212 re_lu_29 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='re_lu_29/Relu6:0', description="created by layer 're_lu_29'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 213 tf.math.multiply_24 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.math.multiply_24/Mul:0', description="created by layer 'tf.math.multiply_24'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 214 multiply_16 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='multiply_16/mul:0', description="created by layer 'multiply_16'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 215 expanded_conv_10/squeeze_excite/AvgPool KerasTensor(type_spec=TensorSpec(shape=(None, 576), dtype=tf.float32, name=None), name='expanded_conv_10/squeeze_excite/AvgPool/Mean:0', description="created by layer 'expanded_conv_10/squeeze_excite/AvgPool'") (None, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 216 reshape_8 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='reshape_8/Reshape:0', description="created by layer 'reshape_8'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 217 expanded_conv_10/squeeze_excite/Conv KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='expanded_conv_10/squeeze_excite/Conv/BiasAdd:0', description="created by layer 'expanded_conv_10/squeeze_excite/Conv'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 218 expanded_conv_10/squeeze_excite/Relu KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 144), dtype=tf.float32, name=None), name='expanded_conv_10/squeeze_excite/Relu/Relu:0', description="created by layer 'expanded_conv_10/squeeze_excite/Relu'") (None, 1, 1, 144)
(RayTrainWorker pid=673, ip=172.31.210.32) 219 expanded_conv_10/squeeze_excite/Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='expanded_conv_10/squeeze_excite/Conv_1/BiasAdd:0', description="created by layer 'expanded_conv_10/squeeze_excite/Conv_1'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 220 tf.__operators__.add_25 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_25/AddV2:0', description="created by layer 'tf.__operators__.add_25'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 221 re_lu_30 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='re_lu_30/Relu6:0', description="created by layer 're_lu_30'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 222 tf.math.multiply_25 KerasTensor(type_spec=TensorSpec(shape=(None, 1, 1, 576), dtype=tf.float32, name=None), name='tf.math.multiply_25/Mul:0', description="created by layer 'tf.math.multiply_25'") (None, 1, 1, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 223 expanded_conv_10/squeeze_excite/Mul KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='expanded_conv_10/squeeze_excite/Mul/mul:0', description="created by layer 'expanded_conv_10/squeeze_excite/Mul'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 224 expanded_conv_10/project KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_10/project/Conv2D:0', description="created by layer 'expanded_conv_10/project'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 225 expanded_conv_10/project/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_10/project/BatchNorm/FusedBatchNormV3:0', description="created by layer 'expanded_conv_10/project/BatchNorm'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 226 expanded_conv_10/Add KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 96), dtype=tf.float32, name=None), name='expanded_conv_10/Add/add:0', description="created by layer 'expanded_conv_10/Add'") (None, 9, 9, 96)
(RayTrainWorker pid=673, ip=172.31.210.32) 227 Conv_1 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='Conv_1/Conv2D:0', description="created by layer 'Conv_1'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 228 Conv_1/BatchNorm KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='Conv_1/BatchNorm/FusedBatchNormV3:0', description="created by layer 'Conv_1/BatchNorm'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 229 tf.__operators__.add_26 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.__operators__.add_26/AddV2:0', description="created by layer 'tf.__operators__.add_26'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 230 re_lu_31 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='re_lu_31/Relu6:0', description="created by layer 're_lu_31'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 231 tf.math.multiply_26 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='tf.math.multiply_26/Mul:0', description="created by layer 'tf.math.multiply_26'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 232 multiply_17 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 576), dtype=tf.float32, name=None), name='multiply_17/mul:0', description="created by layer 'multiply_17'") (None, 9, 9, 576)
(RayTrainWorker pid=673, ip=172.31.210.32) 233 Conv_2 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 1024), dtype=tf.float32, name=None), name='Conv_2/BiasAdd:0', description="created by layer 'Conv_2'") (None, 9, 9, 1024)
(RayTrainWorker pid=673, ip=172.31.210.32) 234 tf.__operators__.add_27 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 1024), dtype=tf.float32, name=None), name='tf.__operators__.add_27/AddV2:0', description="created by layer 'tf.__operators__.add_27'") (None, 9, 9, 1024)
(RayTrainWorker pid=673, ip=172.31.210.32) 235 re_lu_32 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 1024), dtype=tf.float32, name=None), name='re_lu_32/Relu6:0', description="created by layer 're_lu_32'") (None, 9, 9, 1024)
(RayTrainWorker pid=673, ip=172.31.210.32) 236 tf.math.multiply_27 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 1024), dtype=tf.float32, name=None), name='tf.math.multiply_27/Mul:0', description="created by layer 'tf.math.multiply_27'") (None, 9, 9, 1024)
(RayTrainWorker pid=673, ip=172.31.210.32) 237 multiply_18 KerasTensor(type_spec=TensorSpec(shape=(None, 9, 9, 1024), dtype=tf.float32, name=None), name='multiply_18/mul:0', description="created by layer 'multiply_18'") (None, 9, 9, 1024)
(RayTrainWorker pid=673, ip=172.31.210.32) Model: "learn_to_pay_att_model"
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) Layer (type)                 Output Shape              Param #   
(RayTrainWorker pid=673, ip=172.31.210.32) =================================================================
(RayTrainWorker pid=673, ip=172.31.210.32) input_2 (InputLayer)         [(None, 260, 260, 3)]     0         
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) MobilenetV3small (Functional (None, 9, 9, 1024)        1529968   
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) batch_normalization (BatchNo (None, 9, 9, 1024)        4096      
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) conv2d (Conv2D)              (None, 9, 9, 256)         262400    
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) self_att_layer (SelfAttLayer (None, 9, 9, 256)         147841    
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) global_average_pooling2d (Gl (None, 256)               0         
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) dropout (Dropout)            (None, 256)               0         
(RayTrainWorker pid=673, ip=172.31.210.32) =================================================================
(RayTrainWorker pid=673, ip=172.31.210.32) Total params: 1,944,305
(RayTrainWorker pid=673, ip=172.31.210.32) Trainable params: 1,930,017
(RayTrainWorker pid=673, ip=172.31.210.32) Non-trainable params: 14,288
(RayTrainWorker pid=673, ip=172.31.210.32) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) Model: "learn_to_pay_att_model"
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) Layer (type)                 Output Shape              Param #   
(RayTrainWorker pid=743, ip=172.31.221.11) =================================================================
(RayTrainWorker pid=743, ip=172.31.221.11) input_2 (InputLayer)         [(None, 260, 260, 3)]     0         
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) MobilenetV3small (Functional (None, 9, 9, 1024)        1529968   
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) batch_normalization (BatchNo (None, 9, 9, 1024)        4096      
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) conv2d (Conv2D)              (None, 9, 9, 256)         262400    
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) self_att_layer (SelfAttLayer (None, 9, 9, 256)         147841    
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) global_average_pooling2d (Gl (None, 256)               0         
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=743, ip=172.31.221.11) dropout (Dropout)            (None, 256)               0         
(RayTrainWorker pid=743, ip=172.31.221.11) =================================================================
(RayTrainWorker pid=743, ip=172.31.221.11) Total params: 1,944,305
(RayTrainWorker pid=743, ip=172.31.221.11) Trainable params: 1,930,017
(RayTrainWorker pid=743, ip=172.31.221.11) Non-trainable params: 14,288
(RayTrainWorker pid=743, ip=172.31.221.11) _________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) Model: "mv_learn_to_pay_att_model"
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) Layer (type)                    Output Shape         Param #     Connected to                     
(RayTrainWorker pid=673, ip=172.31.210.32) ==================================================================================================
(RayTrainWorker pid=673, ip=172.31.210.32) input_1 (InputLayer)            [(None, 260, 260, 15 0                                            
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) lambda (Lambda)                 (None, 260, 260, 3)  0           input_1[0][0]                    
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) lambda_1 (Lambda)               (None, 260, 260, 3)  0           input_1[0][0]                    
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) lambda_2 (Lambda)               (None, 260, 260, 3)  0           input_1[0][0]                    
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) lambda_3 (Lambda)               (None, 260, 260, 3)  0           input_1[0][0]                    
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) lambda_4 (Lambda)               (None, 260, 260, 3)  0           input_1[0][0]                    
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) learn_to_pay_att_model (Functio (None, 256)          1944305     lambda[0][0]                     
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  lambda_1[0][0]                   
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  lambda_2[0][0]                   
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  lambda_3[0][0]                   
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  lambda_4[0][0]                   
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) features (Concatenate)          (None, 1280)         0           learn_to_pay_att_model[0][0]     
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  learn_to_pay_att_model[1][0]     
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  learn_to_pay_att_model[2][0]     
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  learn_to_pay_att_model[3][0]     
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  learn_to_pay_att_model[4][0]     
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) global_att_score_1_dense_0 (Den (None, 256)          327936      features[0][0]                   
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) features_dense_1 (Dense)        (None, 256)          327936      features[0][0]                   
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) global_att_score_1_activation_0 (None, 256)          0           global_att_score_1_dense_0[0][0] 
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) features_activation_1 (Activati (None, 256)          0           features_dense_1[0][0]           
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) global_att_score_1_bn_0 (BatchN (None, 256)          1024        global_att_score_1_activation_0[0
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) features_bn_1 (BatchNormalizati (None, 256)          1024        features_activation_1[0][0]      
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) global_att_score_1_dropout_0 (D (None, 256)          0           global_att_score_1_bn_0[0][0]    
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) features_dropout_1 (Dropout)    (None, 256)          0           features_bn_1[0][0]              
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) activation (Activation)         (None, 256)          0           global_att_score_1_dropout_0[0][0
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) final_feature (Multiply)        (None, 256)          0           features_dropout_1[0][0]         
(RayTrainWorker pid=673, ip=172.31.210.32)                                                                  activation[0][0]                 
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) categorical_out (Dense)         (None, 12)           3072        final_feature[0][0]              
(RayTrainWorker pid=673, ip=172.31.210.32) ==================================================================================================
(RayTrainWorker pid=673, ip=172.31.210.32) Total params: 2,605,297
(RayTrainWorker pid=673, ip=172.31.210.32) Trainable params: 2,589,985
(RayTrainWorker pid=673, ip=172.31.210.32) Non-trainable params: 15,312
(RayTrainWorker pid=673, ip=172.31.210.32) __________________________________________________________________________________________________
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:34:03.594987: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:34:03.582498: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
(RayTrainWorker pid=743, ip=172.31.221.11) 2022-11-30 09:34:03.600760: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2499995000 Hz
(RayTrainWorker pid=673, ip=172.31.210.32) 2022-11-30 09:34:03.616089: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2499995000 Hz
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 1/30
      1/Unknown - 51s 51s/step - loss: 4.7714 - categorical_accuracy: 0.0312 - categorical_crossentropy: 3.1332
      2/Unknown - 52s 1s/step - loss: 4.8127 - categorical_accuracy: 0.0234 - categorical_crossentropy: 3.1746 
      3/Unknown - 53s 977ms/step - loss: 4.8327 - categorical_accuracy: 0.0226 - categorical_crossentropy: 3.1948
      4/Unknown - 54s 956ms/step - loss: 4.8404 - categorical_accuracy: 0.0306 - categorical_crossentropy: 3.2026
      5/Unknown - 55s 946ms/step - loss: 4.8407 - categorical_accuracy: 0.0382 - categorical_crossentropy: 3.2030
      6/Unknown - 56s 937ms/step - loss: 4.8384 - categorical_accuracy: 0.0431 - categorical_crossentropy: 3.2008
      7/Unknown - 56s 870ms/step - loss: 4.8327 - categorical_accuracy: 0.0472 - categorical_crossentropy: 3.1952
      8/Unknown - 57s 814ms/step - loss: 4.8255 - categorical_accuracy: 0.0501 - categorical_crossentropy: 3.1882
8/8 [==============================] - 63s 2s/step - loss: 4.8199 - categorical_accuracy: 0.0524 - categorical_crossentropy: 3.1826 - val_loss: 4.1551 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.5208
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 2/30
1/8 [==>...........................] - ETA: 7s - loss: 4.1660 - categorical_accuracy: 0.1875 - categorical_crossentropy: 2.5317
2/8 [======>.......................] - ETA: 5s - loss: 4.2833 - categorical_accuracy: 0.1484 - categorical_crossentropy: 2.6491
3/8 [==========>...................] - ETA: 4s - loss: 4.3606 - categorical_accuracy: 0.1302 - categorical_crossentropy: 2.7266
4/8 [==============>...............] - ETA: 3s - loss: 4.4195 - categorical_accuracy: 0.1191 - categorical_crossentropy: 2.7856
5/8 [=================>............] - ETA: 2s - loss: 4.4576 - categorical_accuracy: 0.1116 - categorical_crossentropy: 2.8238
6/8 [=====================>........] - ETA: 1s - loss: 4.4865 - categorical_accuracy: 0.1051 - categorical_crossentropy: 2.8528
7/8 [=========================>....] - ETA: 0s - loss: 4.5075 - categorical_accuracy: 0.1010 - categorical_crossentropy: 2.8739
8/8 [==============================] - ETA: 0s - loss: 4.5225 - categorical_accuracy: 0.0978 - categorical_crossentropy: 2.8890
8/8 [==============================] - 7s 877ms/step - loss: 4.5341 - categorical_accuracy: 0.0953 - categorical_crossentropy: 2.9007 - val_loss: 4.0979 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.4674
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 3/30
1/8 [==>...........................] - ETA: 7s - loss: 4.4518 - categorical_accuracy: 0.2188 - categorical_crossentropy: 2.8213
2/8 [======>.......................] - ETA: 5s - loss: 4.4446 - categorical_accuracy: 0.1875 - categorical_crossentropy: 2.8143
3/8 [==========>...................] - ETA: 4s - loss: 4.4249 - categorical_accuracy: 0.1736 - categorical_crossentropy: 2.7946
4/8 [==============>...............] - ETA: 3s - loss: 4.4234 - categorical_accuracy: 0.1673 - categorical_crossentropy: 2.7932
5/8 [=================>............] - ETA: 2s - loss: 4.4102 - categorical_accuracy: 0.1639 - categorical_crossentropy: 2.7801
6/8 [=====================>........] - ETA: 1s - loss: 4.4155 - categorical_accuracy: 0.1600 - categorical_crossentropy: 2.7855
7/8 [=========================>....] - ETA: 0s - loss: 4.4206 - categorical_accuracy: 0.1562 - categorical_crossentropy: 2.7908
8/8 [==============================] - ETA: 0s - loss: 4.4228 - categorical_accuracy: 0.1527 - categorical_crossentropy: 2.7931
8/8 [==============================] - 7s 860ms/step - loss: 4.4246 - categorical_accuracy: 0.1500 - categorical_crossentropy: 2.7949 - val_loss: 4.0346 - val_categorical_accuracy: 0.1562 - val_categorical_crossentropy: 2.4077
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 4/30
1/8 [==>...........................] - ETA: 7s - loss: 4.2816 - categorical_accuracy: 0.1562 - categorical_crossentropy: 2.6547
2/8 [======>.......................] - ETA: 5s - loss: 4.2985 - categorical_accuracy: 0.1328 - categorical_crossentropy: 2.6718
3/8 [==========>...................] - ETA: 4s - loss: 4.2578 - categorical_accuracy: 0.1441 - categorical_crossentropy: 2.6312
4/8 [==============>...............] - ETA: 3s - loss: 4.2363 - categorical_accuracy: 0.1452 - categorical_crossentropy: 2.6098
5/8 [=================>............] - ETA: 2s - loss: 4.2227 - categorical_accuracy: 0.1486 - categorical_crossentropy: 2.5964
6/8 [=====================>........] - ETA: 1s - loss: 4.2235 - categorical_accuracy: 0.1490 - categorical_crossentropy: 2.5972
7/8 [=========================>....] - ETA: 0s - loss: 4.2256 - categorical_accuracy: 0.1495 - categorical_crossentropy: 2.5995
8/8 [==============================] - ETA: 0s - loss: 4.2270 - categorical_accuracy: 0.1496 - categorical_crossentropy: 2.6010
8/8 [==============================] - 7s 856ms/step - loss: 4.2281 - categorical_accuracy: 0.1497 - categorical_crossentropy: 2.6021 - val_loss: 4.0204 - val_categorical_accuracy: 0.1719 - val_categorical_crossentropy: 2.3973
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 5/30
1/8 [==>...........................] - ETA: 7s - loss: 4.1784 - categorical_accuracy: 0.2500 - categorical_crossentropy: 2.5552
2/8 [======>.......................] - ETA: 5s - loss: 4.1507 - categorical_accuracy: 0.2578 - categorical_crossentropy: 2.5276
3/8 [==========>...................] - ETA: 4s - loss: 4.1596 - categorical_accuracy: 0.2448 - categorical_crossentropy: 2.5366
4/8 [==============>...............] - ETA: 3s - loss: 4.1690 - categorical_accuracy: 0.2324 - categorical_crossentropy: 2.5461
5/8 [=================>............] - ETA: 2s - loss: 4.1627 - categorical_accuracy: 0.2272 - categorical_crossentropy: 2.5400
6/8 [=====================>........] - ETA: 1s - loss: 4.1642 - categorical_accuracy: 0.2214 - categorical_crossentropy: 2.5416
7/8 [=========================>....] - ETA: 0s - loss: 4.1683 - categorical_accuracy: 0.2150 - categorical_crossentropy: 2.5458
8/8 [==============================] - ETA: 0s - loss: 4.1701 - categorical_accuracy: 0.2113 - categorical_crossentropy: 2.5477
8/8 [==============================] - 7s 858ms/step - loss: 4.1715 - categorical_accuracy: 0.2085 - categorical_crossentropy: 2.5492 - val_loss: 4.0300 - val_categorical_accuracy: 0.1719 - val_categorical_crossentropy: 2.4105
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 6/30
1/8 [==>...........................] - ETA: 6s - loss: 3.7546 - categorical_accuracy: 0.3125 - categorical_crossentropy: 2.1351
2/8 [======>.......................] - ETA: 5s - loss: 3.8927 - categorical_accuracy: 0.2812 - categorical_crossentropy: 2.2733
3/8 [==========>...................] - ETA: 4s - loss: 3.9820 - categorical_accuracy: 0.2569 - categorical_crossentropy: 2.3627
4/8 [==============>...............] - ETA: 3s - loss: 4.0147 - categorical_accuracy: 0.2474 - categorical_crossentropy: 2.3955
5/8 [=================>............] - ETA: 2s - loss: 4.0332 - categorical_accuracy: 0.2417 - categorical_crossentropy: 2.4141
6/8 [=====================>........] - ETA: 1s - loss: 4.0536 - categorical_accuracy: 0.2361 - categorical_crossentropy: 2.4347
7/8 [=========================>....] - ETA: 0s - loss: 4.0728 - categorical_accuracy: 0.2303 - categorical_crossentropy: 2.4539
8/8 [==============================] - ETA: 0s - loss: 4.0860 - categorical_accuracy: 0.2269 - categorical_crossentropy: 2.4673
8/8 [==============================] - 7s 858ms/step - loss: 4.0964 - categorical_accuracy: 0.2243 - categorical_crossentropy: 2.4777 - val_loss: 4.0482 - val_categorical_accuracy: 0.0938 - val_categorical_crossentropy: 2.4323
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 7/30
1/8 [==>...........................] - ETA: 7s - loss: 3.8801 - categorical_accuracy: 0.3125 - categorical_crossentropy: 2.2642
2/8 [======>.......................] - ETA: 5s - loss: 3.9645 - categorical_accuracy: 0.2656 - categorical_crossentropy: 2.3487
3/8 [==========>...................] - ETA: 4s - loss: 3.9770 - categorical_accuracy: 0.2569 - categorical_crossentropy: 2.3614
4/8 [==============>...............] - ETA: 3s - loss: 3.9652 - categorical_accuracy: 0.2533 - categorical_crossentropy: 2.3496
5/8 [=================>............] - ETA: 2s - loss: 3.9465 - categorical_accuracy: 0.2576 - categorical_crossentropy: 2.3311
6/8 [=====================>........] - ETA: 1s - loss: 3.9458 - categorical_accuracy: 0.2546 - categorical_crossentropy: 2.3305
7/8 [=========================>....] - ETA: 0s - loss: 3.9481 - categorical_accuracy: 0.2522 - categorical_crossentropy: 2.3329
8/8 [==============================] - ETA: 0s - loss: 3.9493 - categorical_accuracy: 0.2511 - categorical_crossentropy: 2.3342
8/8 [==============================] - 7s 857ms/step - loss: 3.9503 - categorical_accuracy: 0.2503 - categorical_crossentropy: 2.3353 - val_loss: 4.0865 - val_categorical_accuracy: 0.0312 - val_categorical_crossentropy: 2.4742
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 8/30
1/8 [==>...........................] - ETA: 6s - loss: 3.8929 - categorical_accuracy: 0.1562 - categorical_crossentropy: 2.2807
2/8 [======>.......................] - ETA: 5s - loss: 3.9798 - categorical_accuracy: 0.1328 - categorical_crossentropy: 2.3677
3/8 [==========>...................] - ETA: 4s - loss: 3.9971 - categorical_accuracy: 0.1198 - categorical_crossentropy: 2.3851
4/8 [==============>...............] - ETA: 3s - loss: 3.9992 - categorical_accuracy: 0.1191 - categorical_crossentropy: 2.3873
5/8 [=================>............] - ETA: 2s - loss: 3.9892 - categorical_accuracy: 0.1266 - categorical_crossentropy: 2.3774
6/8 [=====================>........] - ETA: 1s - loss: 4.0038 - categorical_accuracy: 0.1298 - categorical_crossentropy: 2.3922
7/8 [=========================>....] - ETA: 0s - loss: 4.0090 - categorical_accuracy: 0.1323 - categorical_crossentropy: 2.3974
8/8 [==============================] - ETA: 0s - loss: 4.0097 - categorical_accuracy: 0.1351 - categorical_crossentropy: 2.3982
8/8 [==============================] - 7s 858ms/step - loss: 4.0103 - categorical_accuracy: 0.1373 - categorical_crossentropy: 2.3989 - val_loss: 4.1403 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.5317
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 9/30
1/8 [==>...........................] - ETA: 6s - loss: 3.9726 - categorical_accuracy: 0.2812 - categorical_crossentropy: 2.3640
2/8 [======>.......................] - ETA: 5s - loss: 3.9797 - categorical_accuracy: 0.2500 - categorical_crossentropy: 2.3712
3/8 [==========>...................] - ETA: 4s - loss: 3.9919 - categorical_accuracy: 0.2431 - categorical_crossentropy: 2.3835
4/8 [==============>...............] - ETA: 3s - loss: 3.9829 - categorical_accuracy: 0.2448 - categorical_crossentropy: 2.3746
5/8 [=================>............] - ETA: 2s - loss: 3.9658 - categorical_accuracy: 0.2483 - categorical_crossentropy: 2.3576
6/8 [=====================>........] - ETA: 1s - loss: 3.9726 - categorical_accuracy: 0.2469 - categorical_crossentropy: 2.3645
7/8 [=========================>....] - ETA: 0s - loss: 3.9785 - categorical_accuracy: 0.2456 - categorical_crossentropy: 2.3706
8/8 [==============================] - ETA: 0s - loss: 3.9803 - categorical_accuracy: 0.2442 - categorical_crossentropy: 2.3725
8/8 [==============================] - 7s 863ms/step - loss: 3.9817 - categorical_accuracy: 0.2432 - categorical_crossentropy: 2.3739 - val_loss: 4.2128 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6078
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 10/30
1/8 [==>...........................] - ETA: 7s - loss: 3.6238 - categorical_accuracy: 0.3750 - categorical_crossentropy: 2.0188
2/8 [======>.......................] - ETA: 5s - loss: 3.7020 - categorical_accuracy: 0.3438 - categorical_crossentropy: 2.0971
3/8 [==========>...................] - ETA: 4s - loss: 3.7701 - categorical_accuracy: 0.3194 - categorical_crossentropy: 2.1653
4/8 [==============>...............] - ETA: 3s - loss: 3.8004 - categorical_accuracy: 0.3001 - categorical_crossentropy: 2.1957
5/8 [=================>............] - ETA: 2s - loss: 3.8212 - categorical_accuracy: 0.2876 - categorical_crossentropy: 2.2166
6/8 [=====================>........] - ETA: 1s - loss: 3.8521 - categorical_accuracy: 0.2735 - categorical_crossentropy: 2.2476
7/8 [=========================>....] - ETA: 0s - loss: 3.8738 - categorical_accuracy: 0.2630 - categorical_crossentropy: 2.2694
8/8 [==============================] - ETA: 0s - loss: 3.8891 - categorical_accuracy: 0.2561 - categorical_crossentropy: 2.2848
8/8 [==============================] - 7s 853ms/step - loss: 3.9010 - categorical_accuracy: 0.2508 - categorical_crossentropy: 2.2968 - val_loss: 4.2647 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6633
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 11/30
1/8 [==>...........................] - ETA: 6s - loss: 3.9716 - categorical_accuracy: 0.2188 - categorical_crossentropy: 2.3701
2/8 [======>.......................] - ETA: 5s - loss: 3.9580 - categorical_accuracy: 0.2422 - categorical_crossentropy: 2.3566
3/8 [==========>...................] - ETA: 4s - loss: 3.9387 - categorical_accuracy: 0.2483 - categorical_crossentropy: 2.3374
4/8 [==============>...............] - ETA: 3s - loss: 3.9151 - categorical_accuracy: 0.2546 - categorical_crossentropy: 2.3139
5/8 [=================>............] - ETA: 2s - loss: 3.8882 - categorical_accuracy: 0.2611 - categorical_crossentropy: 2.2871
6/8 [=====================>........] - ETA: 1s - loss: 3.8862 - categorical_accuracy: 0.2602 - categorical_crossentropy: 2.2853
7/8 [=========================>....] - ETA: 0s - loss: 3.8857 - categorical_accuracy: 0.2590 - categorical_crossentropy: 2.2849
8/8 [==============================] - ETA: 0s - loss: 3.8815 - categorical_accuracy: 0.2593 - categorical_crossentropy: 2.2807
8/8 [==============================] - 7s 863ms/step - loss: 3.8782 - categorical_accuracy: 0.2595 - categorical_crossentropy: 2.2775 - val_loss: 4.3095 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7116
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 12/30
1/8 [==>...........................] - ETA: 6s - loss: 3.5778 - categorical_accuracy: 0.2812 - categorical_crossentropy: 1.9799
2/8 [======>.......................] - ETA: 5s - loss: 3.6907 - categorical_accuracy: 0.2891 - categorical_crossentropy: 2.0929
3/8 [==========>...................] - ETA: 4s - loss: 3.7266 - categorical_accuracy: 0.2969 - categorical_crossentropy: 2.1288
4/8 [==============>...............] - ETA: 3s - loss: 3.7415 - categorical_accuracy: 0.3027 - categorical_crossentropy: 2.1438
5/8 [=================>............] - ETA: 2s - loss: 3.7431 - categorical_accuracy: 0.3047 - categorical_crossentropy: 2.1455
6/8 [=====================>........] - ETA: 1s - loss: 3.7530 - categorical_accuracy: 0.2990 - categorical_crossentropy: 2.1556
7/8 [=========================>....] - ETA: 0s - loss: 3.7607 - categorical_accuracy: 0.2958 - categorical_crossentropy: 2.1634
8/8 [==============================] - ETA: 0s - loss: 3.7653 - categorical_accuracy: 0.2920 - categorical_crossentropy: 2.1681
8/8 [==============================] - 7s 866ms/step - loss: 3.7690 - categorical_accuracy: 0.2890 - categorical_crossentropy: 2.1718 - val_loss: 4.3249 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7304
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 13/30
1/8 [==>...........................] - ETA: 7s - loss: 3.6540 - categorical_accuracy: 0.1875 - categorical_crossentropy: 2.0595
2/8 [======>.......................] - ETA: 5s - loss: 3.6439 - categorical_accuracy: 0.2344 - categorical_crossentropy: 2.0495
3/8 [==========>...................] - ETA: 4s - loss: 3.6473 - categorical_accuracy: 0.2604 - categorical_crossentropy: 2.0530
4/8 [==============>...............] - ETA: 3s - loss: 3.6387 - categorical_accuracy: 0.2754 - categorical_crossentropy: 2.0446
5/8 [=================>............] - ETA: 2s - loss: 3.6219 - categorical_accuracy: 0.2891 - categorical_crossentropy: 2.0279
6/8 [=====================>........] - ETA: 1s - loss: 3.6314 - categorical_accuracy: 0.2921 - categorical_crossentropy: 2.0375
7/8 [=========================>....] - ETA: 0s - loss: 3.6385 - categorical_accuracy: 0.2932 - categorical_crossentropy: 2.0447
8/8 [==============================] - ETA: 0s - loss: 3.6408 - categorical_accuracy: 0.2942 - categorical_crossentropy: 2.0470
8/8 [==============================] - 7s 850ms/step - loss: 3.6425 - categorical_accuracy: 0.2949 - categorical_crossentropy: 2.0489 - val_loss: 4.3199 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7289
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 14/30
1/8 [==>...........................] - ETA: 7s - loss: 3.5054 - categorical_accuracy: 0.4688 - categorical_crossentropy: 1.9145
2/8 [======>.......................] - ETA: 5s - loss: 3.4576 - categorical_accuracy: 0.4609 - categorical_crossentropy: 1.8668
3/8 [==========>...................] - ETA: 4s - loss: 3.4553 - categorical_accuracy: 0.4531 - categorical_crossentropy: 1.8646
4/8 [==============>...............] - ETA: 3s - loss: 3.4584 - categorical_accuracy: 0.4434 - categorical_crossentropy: 1.8678
5/8 [=================>............] - ETA: 2s - loss: 3.4620 - categorical_accuracy: 0.4347 - categorical_crossentropy: 1.8715
6/8 [=====================>........]21.11) 
(RayTrainWorker pid=743, ip=172.31.221.11)  - ETA: 1s - loss: 3.4871 - categorical_accuracy: 0.4230 - categorical_crossentropy: 1.8967
7/8 [=========================>....] - ETA: 0s - loss: 3.5082 - categorical_accuracy: 0.4136 - categorical_crossentropy: 1.9178
8/8 [==============================] - ETA: 0s - loss: 3.5208 - categorical_accuracy: 0.4067 - categorical_crossentropy: 1.9306
8/8 [==============================] - 7s 861ms/step - loss: 3.5306 - categorical_accuracy: 0.4013 - categorical_crossentropy: 1.9405 - val_loss: 4.3159 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7284
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 15/30
1/8 [==>...........................] - ETA: 6s - loss: 3.6090 - categorical_accuracy: 0.3750 - categorical_crossentropy: 2.0215
2/8 [======>.......................] - ETA: 5s - loss: 3.6069 - categorical_accuracy: 0.3828 - categorical_crossentropy: 2.0195
3/8 [==========>...................] - ETA: 4s - loss: 3.5994 - categorical_accuracy: 0.3941 - categorical_crossentropy: 2.0121
4/8 [==============>...............] - ETA: 3s - loss: 3.5827 - categorical_accuracy: 0.3913 - categorical_crossentropy: 1.9955
5/8 [=================>............] - ETA: 2s - loss: 3.5671 - categorical_accuracy: 0.3880 - categorical_crossentropy: 1.9799
6/8 [=====================>........] - ETA: 1s - loss: 3.5724 - categorical_accuracy: 0.3798 - categorical_crossentropy: 1.9854
7/8 [=========================>....] - ETA: 0s - loss: 3.5736 - categorical_accuracy: 0.3765 - categorical_crossentropy: 1.9866
8/8 [==============================] - ETA: 0s - loss: 3.5711 - categorical_accuracy: 0.3765 - categorical_crossentropy: 1.9843
8/8 [==============================] - 7s 869ms/step - loss: 3.5693 - categorical_accuracy: 0.3764 - categorical_crossentropy: 1.9825 - val_loss: 4.3026 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7185
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 16/30
1/8 [==>...........................] - ETA: 7s - loss: 3.2878 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.7036
2/8 [======>.......................] - ETA: 5s - loss: 3.2636 - categorical_accuracy: 0.4141 - categorical_crossentropy: 1.6796
3/8 [==========>...................] - ETA: 4s - loss: 3.3091 - categorical_accuracy: 0.4010 - categorical_crossentropy: 1.7251
4/8 [==============>...............] - ETA: 3s - loss: 3.3294 - categorical_accuracy: 0.3984 - categorical_crossentropy: 1.7456
5/8 [=================>............] - ETA: 2s - loss: 3.3368 - categorical_accuracy: 0.4025 - categorical_crossentropy: 1.7530
6/8 [=====================>........] - ETA: 1s - loss: 3.3661 - categorical_accuracy: 0.3979 - categorical_crossentropy: 1.7824
7/8 [=========================>....] - ETA: 0s - loss: 3.3885 - categorical_accuracy: 0.3962 - categorical_crossentropy: 1.8050
8/8 [==============================] - ETA: 0s - loss: 3.4019 - categorical_accuracy: 0.3959 - categorical_crossentropy: 1.8185
8/8 [==============================] - 7s 852ms/step - loss: 3.4124 - categorical_accuracy: 0.3956 - categorical_crossentropy: 1.8290 - val_loss: 4.2446 - val_categorical_accuracy: 0.0312 - val_categorical_crossentropy: 2.6638
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 17/30
1/8 [==>...........................] - ETA: 6s - loss: 3.3952 - categorical_accuracy: 0.4688 - categorical_crossentropy: 1.8144
2/8 [======>.......................] - ETA: 5s - loss: 3.4330 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.8523
3/8 [==========>...................] - ETA: 4s - loss: 3.4446 - categorical_accuracy: 0.3958 - categorical_crossentropy: 1.8640
4/8 [==============>...............] - ETA: 3s - loss: 3.4400 - categorical_accuracy: 0.3984 - categorical_crossentropy: 1.8595
5/8 [=================>............] - ETA: 2s - loss: 3.4274 - categorical_accuracy: 0.4025 - categorical_crossentropy: 1.8470
6/8 [=====================>........] - ETA: 1s - loss: 3.4379 - categorical_accuracy: 0.4005 - categorical_crossentropy: 1.8576
7/8 [=========================>....] - ETA: 0s - loss: 3.4426 - categorical_accuracy: 0.4004 - categorical_crossentropy: 1.8624
8/8 [==============================] - ETA: 0s - loss: 3.4416 - categorical_accuracy: 0.4024 - categorical_crossentropy: 1.8615
8/8 [==============================] - 7s 864ms/step - loss: 3.4408 - categorical_accuracy: 0.4039 - categorical_crossentropy: 1.8608 - val_loss: 4.2219 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6444
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 18/30
1/8 [==>...........................] - ETA: 6s - loss: 3.4565 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.8790
2/8 [======>.......................] - ETA: 5s - loss: 3.4241 - categorical_accuracy: 0.4141 - categorical_crossentropy: 1.8467
3/8 [==========>...................] - ETA: 4s - loss: 3.4173 - categorical_accuracy: 0.4219 - categorical_crossentropy: 1.8400
4/8 [==============>...............] - ETA: 3s - loss: 3.4109 - categorical_accuracy: 0.4277 - categorical_crossentropy: 1.8337
5/8 [=================>............] - ETA: 2s - loss: 3.4072 - categorical_accuracy: 0.4334 - categorical_crossentropy: 1.8302
6/8 [=====================>........] - ETA: 1s - loss: 3.4271 - categorical_accuracy: 0.4289 - categorical_crossentropy: 1.8501
7/8 [=========================>....] - ETA: 0s - loss: 3.4425 - categorical_accuracy: 0.4255 - categorical_crossentropy: 1.8656
8/8 [==============================] - ETA: 0s - loss: 3.4529 - categorical_accuracy: 0.4215 - categorical_crossentropy: 1.8761
8/8 [==============================] - 7s 870ms/step - loss: 3.4610 - categorical_accuracy: 0.4184 - categorical_crossentropy: 1.8843 - val_loss: 4.2265 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6523
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 19/30
1/8 [==>...........................] - ETA: 7s - loss: 3.4328 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.8586
2/8 [======>.......................] - ETA: 5s - loss: 3.4676 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.8934
3/8 [==========>...................] - ETA: 4s - loss: 3.4757 - categorical_accuracy: 0.4097 - categorical_crossentropy: 1.9017
4/8 [==============>...............] - ETA: 3s - loss: 3.4389 - categorical_accuracy: 0.4225 - categorical_crossentropy: 1.8650
5/8 [=================>............] - ETA: 2s - loss: 3.4083 - categorical_accuracy: 0.4318 - categorical_crossentropy: 1.8345
6/8 [=====================>........] - ETA: 1s - loss: 3.4089 - categorical_accuracy: 0.4301 - categorical_crossentropy: 1.8352
7/8 [=========================>....] - ETA: 0s - loss: 3.4109 - categorical_accuracy: 0.4285 - categorical_crossentropy: 1.8373
8/8 [==============================] - ETA: 0s - loss: 3.4116 - categorical_accuracy: 0.4270 - categorical_crossentropy: 1.8380
8/8 [==============================] - 7s 847ms/step - loss: 3.4121 - categorical_accuracy: 0.4257 - categorical_crossentropy: 1.8386 - val_loss: 4.2254 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6544
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 20/30
1/8 [==>...........................] - ETA: 6s - loss: 3.2538 - categorical_accuracy: 0.5312 - categorical_crossentropy: 1.6828
2/8 [======>.......................] - ETA: 5s - loss: 3.2288 - categorical_accuracy: 0.5234 - categorical_crossentropy: 1.6580
3/8 [==========>...................] - ETA: 4s - loss: 3.2146 - categorical_accuracy: 0.5122 - categorical_crossentropy: 1.6439
4/8 [==============>...............] - ETA: 3s - loss: 3.1928 - categorical_accuracy: 0.5111 - categorical_crossentropy: 1.6221
5/8 [=================>............] - ETA: 2s - loss: 3.1745 - categorical_accuracy: 0.5101 - categorical_crossentropy: 1.6039
6/8 [=====================>........] - ETA: 1s - loss: 3.1978 - categorical_accuracy: 0.4980 - categorical_crossentropy: 1.6274
7/8 [=========================>....] - ETA: 0s - loss: 3.2161 - categorical_accuracy: 0.4894 - categorical_crossentropy: 1.6458
8/8 [==============================] - ETA: 0s - loss: 3.2309 - categorical_accuracy: 0.4830 - categorical_crossentropy: 1.6606
8/8 [==============================] - 7s 855ms/step - loss: 3.2424 - categorical_accuracy: 0.4780 - categorical_crossentropy: 1.6722 - val_loss: 4.2056 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6378
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 21/30
1/8 [==>...........................] - ETA: 6s - loss: 3.1923 - categorical_accuracy: 0.4375 - categorical_crossentropy: 1.6245
2/8 [======>.......................] - ETA: 5s - loss: 3.1676 - categorical_accuracy: 0.4375 - categorical_crossentropy: 1.6000
3/8 [==========>...................] - ETA: 4s - loss: 3.1559 - categorical_accuracy: 0.4340 - categorical_crossentropy: 1.5884
4/8 [==============>...............] - ETA: 3s - loss: 3.1464 - categorical_accuracy: 0.4447 - categorical_crossentropy: 1.5789
5/8 [=================>............] - ETA: 2s - loss: 3.1451 - categorical_accuracy: 0.4507 - categorical_crossentropy: 1.5777
6/8 [=====================>........] - ETA: 1s - loss: 3.1641 - categorical_accuracy: 0.4477 - categorical_crossentropy: 1.5969
7/8 [=========================>....] - ETA: 0s - loss: 3.1802 - categorical_accuracy: 0.4456 - categorical_crossentropy: 1.6130
8/8 [==============================] - ETA: 0s - loss: 3.1872 - categorical_accuracy: 0.4463 - categorical_crossentropy: 1.6201
8/8 [==============================] - 7s 860ms/step - loss: 3.1927 - categorical_accuracy: 0.4469 - categorical_crossentropy: 1.6256 - val_loss: 4.2166 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6520
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 22/30
1/8 [==>...........................] - ETA: 6s - loss: 3.0761 - categorical_accuracy: 0.5938 - categorical_crossentropy: 1.5115
2/8 [======>.......................] - ETA: 5s - loss: 3.1159 - categorical_accuracy: 0.5547 - categorical_crossentropy: 1.5514
3/8 [==========>...................] - ETA: 4s - loss: 3.1484 - categorical_accuracy: 0.5295 - categorical_crossentropy: 1.5840
4/8 [==============>...............] - ETA: 3s - loss: 3.1642 - categorical_accuracy: 0.5046 - categorical_crossentropy: 1.5999
5/8 [=================>............] - ETA: 2s - loss: 3.1657 - categorical_accuracy: 0.4961 - categorical_crossentropy: 1.6015
6/8 [=====================>........] - ETA: 1s - loss: 3.1862 - categorical_accuracy: 0.4820 - categorical_crossentropy: 1.6221
7/8 [=========================>....] - ETA: 0s - loss: 3.2043 - categorical_accuracy: 0.4703 - categorical_crossentropy: 1.6403
8/8 [==============================] - ETA: 0s - loss: 3.2144 - categorical_accuracy: 0.4646 - categorical_crossentropy: 1.6505
8/8 [==============================] - 7s 854ms/step - loss: 3.2223 - categorical_accuracy: 0.4602 - categorical_crossentropy: 1.6584 - val_loss: 4.3091 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7476
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 23/30
1/8 [==>...........................] - ETA: 6s - loss: 3.0716 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.5101
2/8 [======>.......................] - ETA: 5s - loss: 3.0934 - categorical_accuracy: 0.4062 - categorical_crossentropy: 1.5320
3/8 [==========>...................] - ETA: 4s - loss: 3.1002 - categorical_accuracy: 0.4236 - categorical_crossentropy: 1.5389
4/8 [==============>...............] - ETA: 3s - loss: 3.0891 - categorical_accuracy: 0.4388 - categorical_crossentropy: 1.5279
5/8 [=================>............] - ETA: 2s - loss: 3.0798 - categorical_accuracy: 0.4510 - categorical_crossentropy: 1.5188
6/8 [=====================>........] - ETA: 1s - loss: 3.0991 - categorical_accuracy: 0.4523 - categorical_crossentropy: 1.5381
7/8 [=========================>....] - ETA: 0s - loss: 3.1153 - categorical_accuracy: 0.4523 - categorical_crossentropy: 1.5544
8/8 [==============================] - ETA: 0s - loss: 3.1240 - categorical_accuracy: 0.4538 - categorical_crossentropy: 1.5632
8/8 [==============================] - 7s 842ms/step - loss: 3.1308 - categorical_accuracy: 0.4550 - categorical_crossentropy: 1.5700 - val_loss: 4.3431 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7848
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 24/30
1/8 [==>...........................] - ETA: 6s - loss: 3.0536 - categorical_accuracy: 0.4688 - categorical_crossentropy: 1.4952
2/8 [======>.......................] - ETA: 5s - loss: 3.1131 - categorical_accuracy: 0.4531 - categorical_crossentropy: 1.5548
3/8 [==========>...................] - ETA: 4s - loss: 3.1604 - categorical_accuracy: 0.4653 - categorical_crossentropy: 1.6023
4/8 [==============>...............] - ETA: 3s - loss: 3.1390 - categorical_accuracy: 0.4954 - categorical_crossentropy: 1.5809
5/8 [=================>............] - ETA: 2s - loss: 3.1216 - categorical_accuracy: 0.5139 - categorical_crossentropy: 1.5636
6/8 [=====================>........] - ETA: 1s - loss: 3.1352 - categorical_accuracy: 0.5168 - categorical_crossentropy: 1.5773
7/8 [=========================>....] - ETA: 0s - loss: 3.1452 - categorical_accuracy: 0.5184 - categorical_crossentropy: 1.5874
8/8 [==============================] - ETA: 0s - loss: 3.1494 - categorical_accuracy: 0.5217 - categorical_crossentropy: 1.5917
8/8 [==============================] - 7s 846ms/step - loss: 3.1527 - categorical_accuracy: 0.5242 - categorical_crossentropy: 1.5950 - val_loss: 4.3377 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7824
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 25/30
(scheduler +6m9s) Resized to 88 CPUs, 2 GPUs.
1/8 [==>...........................] - ETA: 6s - loss: 2.9034 - categorical_accuracy: 0.5938 - categorical_crossentropy: 1.3481
2/8 [======>.......................] - ETA: 5s - loss: 2.9254 - categorical_accuracy: 0.6016 - categorical_crossentropy: 1.3702
3/8 [==========>...................] - ETA: 4s - loss: 2.9431 - categorical_accuracy: 0.6059 - categorical_crossentropy: 1.3880
4/8 [==============>...............] - ETA: 3s - loss: 2.9335 - categorical_accuracy: 0.6146 - categorical_crossentropy: 1.3784
5/8 [=================>............] - ETA: 2s - loss: 2.9216 - categorical_accuracy: 0.6204 - categorical_crossentropy: 1.3667
6/8 [=====================>........] - ETA: 1s - loss: 2.9386 - categorical_accuracy: 0.6142 - categorical_crossentropy: 1.3838
7/8 [=========================>....] - ETA: 0s - loss: 2.9539 - categorical_accuracy: 0.6081 - categorical_crossentropy: 1.3991
8/8 [==============================] - ETA: 0s - loss: 2.9641 - categorical_accuracy: 0.6046 - categorical_crossentropy: 1.4094
8/8 [==============================] - 7s 856ms/step - loss: 2.9720 - categorical_accuracy: 0.6018 - categorical_crossentropy: 1.4174 - val_loss: 4.2670 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.7148
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 26/30
1/8 [==>...........................] - ETA: 6s - loss: 2.8851 - categorical_accuracy: 0.6250 - categorical_crossentropy: 1.3328
2/8 [======>.......................] - ETA: 5s - loss: 2.9992 - categorical_accuracy: 0.5703 - categorical_crossentropy: 1.4470
3/8 [==========>...................] - ETA: 4s - loss: 3.0233 - categorical_accuracy: 0.5642 - categorical_crossentropy: 1.4712
4/8 [==============>...............] - ETA: 3s - loss: 3.0109 - categorical_accuracy: 0.5697 - categorical_crossentropy: 1.4589
5/8 [=================>............] - ETA: 2s - loss: 2.9961 - categorical_accuracy: 0.5720 - categorical_crossentropy: 1.4442
6/8 [=====================>........] - ETA: 1s - loss: 3.0071 - categorical_accuracy: 0.5669 - categorical_crossentropy: 1.4553
7/8 [=========================>....] - ETA: 0s - loss: 3.0127 - categorical_accuracy: 0.5628 - categorical_crossentropy: 1.4609
8/8 [==============================] - ETA: 0s - loss: 3.0136 - categorical_accuracy: 0.5616 - categorical_crossentropy: 1.4620
8/8 [==============================] - 7s 849ms/step - loss: 3.0144 - categorical_accuracy: 0.5607 - categorical_crossentropy: 1.4628 - val_loss: 4.1781 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.6288
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 27/30
1/8 [==>...........................] - ETA: 7s - loss: 2.9844 - categorical_accuracy: 0.4688 - categorical_crossentropy: 1.4351
2/8 [======>.......................] - ETA: 5s - loss: 3.0400 - categorical_accuracy: 0.4375 - categorical_crossentropy: 1.4908
3/8 [==========>...................] - ETA: 4s - loss: 3.0263 - categorical_accuracy: 0.4583 - categorical_crossentropy: 1.4772
4/8 [==============>...............] - ETA: 3s - loss: 2.9956 - categorical_accuracy: 0.4805 - categorical_crossentropy: 1.4466
5/8 [=================>............] - ETA: 2s - loss: 2.9835 - categorical_accuracy: 0.4969 - categorical_crossentropy: 1.4346
6/8 [=====================>........] - ETA: 1s - loss: 2.9943 - categorical_accuracy: 0.4991 - categorical_crossentropy: 1.4455
7/8 [=========================>....] - ETA: 0s - loss: 3.0016 - categorical_accuracy: 0.4999 - categorical_crossentropy: 1.4528
8/8 [==============================] - ETA: 0s - loss: 3.0052 - categorical_accuracy: 0.5005 - categorical_crossentropy: 1.4565
8/8 [==============================] - 7s 845ms/step - loss: 3.0080 - categorical_accuracy: 0.5009 - categorical_crossentropy: 1.4593 - val_loss: 4.1587 - val_categorical_accuracy: 0.0156 - val_categorical_crossentropy: 2.6124
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 28/30
1/8 [==>...........................] - ETA: 6s - loss: 2.8694 - categorical_accuracy: 0.6250 - categorical_crossentropy: 1.3230
2/8 [======>.......................] - ETA: 5s - loss: 2.8742 - categorical_accuracy: 0.6094 - categorical_crossentropy: 1.3280
3/8 [==========>...................] - ETA: 4s - loss: 2.9004 - categorical_accuracy: 0.5938 - categorical_crossentropy: 1.3542
4/8 [==============>...............] - ETA: 3s - loss: 2.8917 - categorical_accuracy: 0.5996 - categorical_crossentropy: 1.3456
5/8 [=================>............] - ETA: 2s - loss: 2.8828 - categorical_accuracy: 0.6047 - categorical_crossentropy: 1.3368
6/8 [=====================>........] - ETA: 1s - loss: 2.9072 - categorical_accuracy: 0.5959 - categorical_crossentropy: 1.3614
7/8 [=========================>....] - ETA: 0s - loss: 2.9223 - categorical_accuracy: 0.5917 - categorical_crossentropy: 1.3764
8/8 [==============================] - ETA: 0s - loss: 2.9306 - categorical_accuracy: 0.5902 - categorical_crossentropy: 1.3849
8/8 [==============================] - 7s 851ms/step - loss: 2.9372 - categorical_accuracy: 0.5891 - categorical_crossentropy: 1.3915 - val_loss: 4.1412 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.5977
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 29/30
1/8 [==>...........................] - ETA: 6s - loss: 2.6348 - categorical_accuracy: 0.6562 - categorical_crossentropy: 1.0914
2/8 [======>.......................] - ETA: 5s - loss: 2.7159 - categorical_accuracy: 0.6328 - categorical_crossentropy: 1.1726
3/8 [==========>...................] - ETA: 4s - loss: 2.7737 - categorical_accuracy: 0.6128 - categorical_crossentropy: 1.2304
4/8 [==============>...............] - ETA: 3s - loss: 2.7831 - categorical_accuracy: 0.6159 - categorical_crossentropy: 1.2400
5/8 [=================>............] - ETA: 2s - loss: 2.7783 - categorical_accuracy: 0.6190 - categorical_crossentropy: 1.2352
6/8 [=====================>........] - ETA: 1s - loss: 2.7993 - categorical_accuracy: 0.6130 - categorical_crossentropy: 1.2564
7/8 [=========================>....] - ETA: 0s - loss: 2.8162 - categorical_accuracy: 0.6084 - categorical_crossentropy: 1.2733
8/8 [==============================] - ETA: 0s - loss: 2.8254 - categorical_accuracy: 0.6065 - categorical_crossentropy: 1.2826
8/8 [==============================] - 7s 851ms/step - loss: 2.8325 - categorical_accuracy: 0.6050 - categorical_crossentropy: 1.2897 - val_loss: 4.1094 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.5688
(RayTrainWorker pid=743, ip=172.31.221.11) Epoch 30/30
1/8 [==>...........................] - ETA: 6s - loss: 2.6296 - categorical_accuracy: 0.7188 - categorical_crossentropy: 1.0890
2/8 [======>.......................] - ETA: 5s - loss: 2.6631 - categorical_accuracy: 0.6875 - categorical_crossentropy: 1.1227
3/8 [==========>...................] - ETA: 4s - loss: 2.6926 - categorical_accuracy: 0.6736 - categorical_crossentropy: 1.1523
4/8 [==============>...............] - ETA: 3s - loss: 2.6963 - categorical_accuracy: 0.6732 - categorical_crossentropy: 1.1560
5/8 [=================>............] - ETA: 2s - loss: 2.6945 - categorical_accuracy: 0.6735 - categorical_crossentropy: 1.1543
6/8 [=====================>........] - ETA: 1s - loss: 2.7220 - categorical_accuracy: 0.6611 - categorical_crossentropy: 1.1819
7/8 [=========================>....] - ETA: 0s - loss: 2.7376 - categorical_accuracy: 0.6551 - categorical_crossentropy: 1.1976
8/8 [==============================] - ETA: 0s - loss: 2.7475 - categorical_accuracy: 0.6506 - categorical_crossentropy: 1.2076
8/8 [==============================] - 7s 846ms/step - loss: 2.7552 - categorical_accuracy: 0.6472 - categorical_crossentropy: 1.2153 - val_loss: 4.1148 - val_categorical_accuracy: 0.0000e+00 - val_categorical_crossentropy: 2.5771
2022-11-30 09:38:34,971 ERROR checkpoint_manager.py:327 -- Result dict has no key: training_iteration. checkpoint_score_attr must be set to a key in the result dict. Valid keys are: ['trial_id', 'experiment_id', 'date', 'timestamp', 'pid', 'hostname', 'node_ip', 'done']
2022-11-30 09:38:34,971 WARNING util.py:244 -- The `process_trial_save` operation took 1.753 s, which may be a performance bottleneck.
2022-11-30 09:38:34,971 WARNING trial_runner.py:1058 -- Consider turning off forced head-worker trial checkpoint syncs by setting sync_on_checkpoint=False. Note that this may result in faulty trial restoration if a failure occurs while the checkpoint is being synced from the worker to the head node.
Trial TensorflowTrainer_ee867_00000 completed. Last result: