Skip to content

Instantly share code, notes, and snippets.

@hughjazzman

hughjazzman/output.txt Secret

Created July 21, 2021 07:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hughjazzman/c0c318563673745b8066ce8183bcb779 to your computer and use it in GitHub Desktop.
Save hughjazzman/c0c318563673745b8066ce8183bcb779 to your computer and use it in GitHub Desktop.
Download ZIP
ST3D
Raw
output.txt
+ NGPUS=8
+ PY_ARGS='--cfg_file cfgs/da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d.yaml'
+ python -m torch.distributed.launch --nproc_per_node=8 train.py --launcher pytorch --cfg_file cfgs/da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d.yaml
[2021-07-21 15:04:55,821 train.py 85 INFO] **********************Start logging**********************
[2021-07-21 15:04:55,822 train.py 87 INFO] CUDA_VISIBLE_DEVICES=ALL
[2021-07-21 15:04:55,822 train.py 91 INFO] total_batch_size: 16
[2021-07-21 15:04:55,822 train.py 93 INFO] cfg_file cfgs/da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d.yaml
[2021-07-21 15:04:55,822 train.py 93 INFO] batch_size 2
[2021-07-21 15:04:55,822 train.py 93 INFO] epochs 30
[2021-07-21 15:04:55,822 train.py 93 INFO] workers 4
[2021-07-21 15:04:55,822 train.py 93 INFO] extra_tag default
[2021-07-21 15:04:55,822 train.py 93 INFO] ckpt None
[2021-07-21 15:04:55,822 train.py 93 INFO] pretrained_model None
[2021-07-21 15:04:55,823 train.py 93 INFO] launcher pytorch
[2021-07-21 15:04:55,823 train.py 93 INFO] tcp_port 18888
[2021-07-21 15:04:55,823 train.py 93 INFO] sync_bn False
[2021-07-21 15:04:55,823 train.py 93 INFO] fix_random_seed False
[2021-07-21 15:04:55,823 train.py 93 INFO] ckpt_save_interval 1
[2021-07-21 15:04:55,823 train.py 93 INFO] local_rank 0
[2021-07-21 15:04:55,823 train.py 93 INFO] max_ckpt_save_num 30
[2021-07-21 15:04:55,823 train.py 93 INFO] merge_all_iters_to_one_epoch False
[2021-07-21 15:04:55,823 train.py 93 INFO] set_cfgs None
[2021-07-21 15:04:55,823 train.py 93 INFO] max_waiting_mins 0
[2021-07-21 15:04:55,823 train.py 93 INFO] start_epoch 0
[2021-07-21 15:04:55,824 train.py 93 INFO] save_to_file False
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.ROOT_DIR: /home/user5/open-mmlab/ST3D
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.LOCAL_RANK: 0
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.CLASS_NAMES: ['Vehicle']
[2021-07-21 15:04:55,824 config.py 9 INFO]
cfg.DATA_CONFIG = edict()
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.DATA_CONFIG.DATASET: WaymoDataset
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.DATA_CONFIG.DATA_PATH: ../data/waymo
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.DATA_CONFIG.PROCESSED_DATA_TAG: waymo_processed_data
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
[2021-07-21 15:04:55,824 config.py 12 INFO] cfg.DATA_CONFIG.MIN_POINTS_OF_GT: 1
[2021-07-21 15:04:55,825 config.py 9 INFO]
cfg.DATA_CONFIG.DATA_SPLIT = edict()
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.DATA_SPLIT.train: train
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.DATA_SPLIT.test: val
[2021-07-21 15:04:55,825 config.py 9 INFO]
cfg.DATA_CONFIG.SAMPLED_INTERVAL = edict()
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.SAMPLED_INTERVAL.train: 2
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.SAMPLED_INTERVAL.test: 5
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.INFO_WITH_FAKELIDAR: False
[2021-07-21 15:04:55,825 config.py 9 INFO]
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['random_object_rotation', 'random_object_scaling', 'normalize_object_size']
[2021-07-21 15:04:55,825 config.py 12 INFO] cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'random_object_scaling', 'SCALE_UNIFORM_NOISE': [0.75, 0.95]}, {'NAME': 'normalize_object_size', 'SIZE_RES': [-0.91, -0.49, -0.26]}, {'NAME': 'random_object_rotation', 'ROT_PROB': 1.0, 'ROT_UNIFORM_NOISE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
[2021-07-21 15:04:55,826 config.py 9 INFO]
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z']
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity', 'elongation']
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': True}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 80000, 'test': 90000}}]
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/da_waymo_dataset.yaml
[2021-07-21 15:04:55,826 config.py 9 INFO]
cfg.DATA_CONFIG_TAR = edict()
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATASET: KittiDataset
[2021-07-21 15:04:55,826 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATA_PATH: ../data/kitti
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
[2021-07-21 15:04:55,827 config.py 9 INFO]
cfg.DATA_CONFIG_TAR.DATA_SPLIT = edict()
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATA_SPLIT.train: train
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATA_SPLIT.test: val
[2021-07-21 15:04:55,827 config.py 9 INFO]
cfg.DATA_CONFIG_TAR.INFO_PATH = edict()
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.INFO_PATH.train: ['kitti_infos_train.pkl']
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.INFO_PATH.test: ['kitti_infos_val.pkl']
[2021-07-21 15:04:55,827 config.py 9 INFO]
cfg.DATA_CONFIG_TAR.DATA_AUGMENTOR = edict()
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
[2021-07-21 15:04:55,827 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'random_object_scaling', 'SCALE_UNIFORM_NOISE': [0.95, 1.05]}, {'NAME': 'random_object_rotation', 'ROT_PROB': 0.8, 'ROT_UNIFORM_NOISE': [-0.38539816, 0.38539816]}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.97, 1.03]}]
[2021-07-21 15:04:55,827 config.py 9 INFO]
cfg.DATA_CONFIG_TAR.POINT_FEATURE_ENCODING = edict()
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z']
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 80000, 'test': 90000}}]
[2021-07-21 15:04:55,828 config.py 9 INFO]
cfg.DATA_CONFIG_TAR.TEST = edict()
[2021-07-21 15:04:55,828 config.py 9 INFO]
cfg.DATA_CONFIG_TAR.TEST.BOX_FILTER = edict()
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.TEST.BOX_FILTER.USE_IMAGE_AREA_FILTER: True
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.TEST.BOX_FILTER.FOV_FILTER: True
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR.TEST.BOX_FILTER.LIMIT_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
[2021-07-21 15:04:55,828 config.py 12 INFO] cfg.DATA_CONFIG_TAR._BASE_CONFIG_: cfgs/dataset_configs/da_kitti_dataset.yaml
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.DATA_CONFIG_TAR.TARGET: True
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.DATA_CONFIG_TAR.FOV_POINTS_ONLY: False
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.DATA_CONFIG_TAR.USE_PSEUDO_LABEL: True
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.DATA_CONFIG_TAR.CLASS_NAMES: ['Car']
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.DATA_CONFIG_TAR.SHIFT_COOR: [0.0, 0.0, 1.6]
[2021-07-21 15:04:55,829 config.py 9 INFO]
cfg.MODEL = edict()
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.MODEL.NAME: PVRCNN
[2021-07-21 15:04:55,829 config.py 9 INFO]
cfg.MODEL.VFE = edict()
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.MODEL.VFE.NAME: MeanVFE
[2021-07-21 15:04:55,829 config.py 9 INFO]
cfg.MODEL.BACKBONE_3D = edict()
[2021-07-21 15:04:55,829 config.py 12 INFO] cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x
[2021-07-21 15:04:55,830 config.py 9 INFO]
cfg.MODEL.MAP_TO_BEV = edict()
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
[2021-07-21 15:04:55,830 config.py 9 INFO]
cfg.MODEL.BACKBONE_2D = edict()
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
[2021-07-21 15:04:55,830 config.py 12 INFO] cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
[2021-07-21 15:04:55,831 config.py 9 INFO]
cfg.MODEL.DENSE_HEAD = edict()
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Vehicle', 'anchor_sizes': [[4.2, 2.0, 1.6]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.55, 'unmatched_threshold': 0.4}]
[2021-07-21 15:04:55,831 config.py 9 INFO]
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
[2021-07-21 15:04:55,831 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
[2021-07-21 15:04:55,832 config.py 9 INFO]
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
[2021-07-21 15:04:55,832 config.py 9 INFO]
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
[2021-07-21 15:04:55,832 config.py 12 INFO] cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
[2021-07-21 15:04:55,832 config.py 9 INFO]
cfg.MODEL.PFE = edict()
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.NAME: VoxelSetAbstraction
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.POINT_SOURCE: raw_points
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.NUM_KEYPOINTS: 4096
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.NUM_OUTPUT_FEATURES: 128
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.SAMPLE_METHOD: FPS
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.FEATURES_SOURCE: ['bev', 'x_conv3', 'x_conv4', 'raw_points']
[2021-07-21 15:04:55,833 config.py 9 INFO]
cfg.MODEL.PFE.SA_LAYER = edict()
[2021-07-21 15:04:55,833 config.py 9 INFO]
cfg.MODEL.PFE.SA_LAYER.raw_points = edict()
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.raw_points.MLPS: [[16, 16], [16, 16]]
[2021-07-21 15:04:55,833 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.raw_points.POOL_RADIUS: [0.4, 0.8]
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.raw_points.NSAMPLE: [16, 16]
[2021-07-21 15:04:55,834 config.py 9 INFO]
cfg.MODEL.PFE.SA_LAYER.x_conv1 = edict()
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv1.DOWNSAMPLE_FACTOR: 1
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv1.MLPS: [[16, 16], [16, 16]]
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv1.POOL_RADIUS: [0.4, 0.8]
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv1.NSAMPLE: [16, 16]
[2021-07-21 15:04:55,834 config.py 9 INFO]
cfg.MODEL.PFE.SA_LAYER.x_conv2 = edict()
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv2.DOWNSAMPLE_FACTOR: 2
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv2.MLPS: [[32, 32], [32, 32]]
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv2.POOL_RADIUS: [0.8, 1.2]
[2021-07-21 15:04:55,834 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv2.NSAMPLE: [16, 32]
[2021-07-21 15:04:55,835 config.py 9 INFO]
cfg.MODEL.PFE.SA_LAYER.x_conv3 = edict()
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv3.DOWNSAMPLE_FACTOR: 4
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv3.MLPS: [[64, 64], [64, 64]]
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv3.POOL_RADIUS: [1.2, 2.4]
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv3.NSAMPLE: [16, 32]
[2021-07-21 15:04:55,835 config.py 9 INFO]
cfg.MODEL.PFE.SA_LAYER.x_conv4 = edict()
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv4.DOWNSAMPLE_FACTOR: 8
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv4.MLPS: [[64, 64], [64, 64]]
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv4.POOL_RADIUS: [2.4, 4.8]
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.PFE.SA_LAYER.x_conv4.NSAMPLE: [16, 32]
[2021-07-21 15:04:55,835 config.py 9 INFO]
cfg.MODEL.POINT_HEAD = edict()
[2021-07-21 15:04:55,835 config.py 12 INFO] cfg.MODEL.POINT_HEAD.NAME: PointHeadSimple
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.POINT_HEAD.CLS_FC: [256, 256]
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.POINT_HEAD.CLASS_AGNOSTIC: True
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.POINT_HEAD.USE_POINT_FEATURES_BEFORE_FUSION: True
[2021-07-21 15:04:55,836 config.py 9 INFO]
cfg.MODEL.POINT_HEAD.TARGET_CONFIG = edict()
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.POINT_HEAD.TARGET_CONFIG.GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
[2021-07-21 15:04:55,836 config.py 9 INFO]
cfg.MODEL.POINT_HEAD.LOSS_CONFIG = edict()
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_REG: smooth-l1
[2021-07-21 15:04:55,836 config.py 9 INFO]
cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_cls_weight: 1.0
[2021-07-21 15:04:55,836 config.py 9 INFO]
cfg.MODEL.ROI_HEAD = edict()
[2021-07-21 15:04:55,836 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NAME: PVRCNNHead
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.CLASS_AGNOSTIC: True
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.SHARED_FC: [256, 256]
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.CLS_FC: [256, 256]
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.REG_FC: [256, 256]
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.DP_RATIO: 0.3
[2021-07-21 15:04:55,837 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.NMS_CONFIG = edict()
[2021-07-21 15:04:55,837 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN = edict()
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_TYPE: nms_gpu
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.MULTI_CLASSES_NMS: False
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_PRE_MAXSIZE: 9000
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_POST_MAXSIZE: 512
[2021-07-21 15:04:55,837 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_THRESH: 0.8
[2021-07-21 15:04:55,838 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST = edict()
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_TYPE: nms_gpu
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.MULTI_CLASSES_NMS: False
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_PRE_MAXSIZE: 1024
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_POST_MAXSIZE: 100
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_THRESH: 0.7
[2021-07-21 15:04:55,838 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.ROI_GRID_POOL = edict()
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.GRID_SIZE: 6
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.MLPS: [[64, 64], [64, 64]]
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.POOL_RADIUS: [0.8, 1.6]
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.NSAMPLE: [16, 16]
[2021-07-21 15:04:55,838 config.py 12 INFO] cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.POOL_METHOD: max_pool
[2021-07-21 15:04:55,839 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.TARGET_CONFIG = edict()
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.BOX_CODER: ResidualCoder
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.ROI_PER_IMAGE: 128
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.FG_RATIO: 0.5
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.SAMPLE_ROI_BY_EACH_CLASS: True
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_SCORE_TYPE: raw_roi_iou
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_FG_THRESH: 0.75
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH: 0.25
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH_LO: 0.1
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.HARD_BG_RATIO: 0.8
[2021-07-21 15:04:55,839 config.py 12 INFO] cfg.MODEL.ROI_HEAD.TARGET_CONFIG.REG_FG_THRESH: 0.55
[2021-07-21 15:04:55,839 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.LOSS_CONFIG = edict()
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CLS_LOSS: BinaryCrossEntropy
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.REG_LOSS: smooth-l1
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CORNER_LOSS_REGULARIZATION: True
[2021-07-21 15:04:55,840 config.py 9 INFO]
cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_cls_weight: 1.0
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_reg_weight: 1.0
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_corner_weight: 1.0
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
[2021-07-21 15:04:55,840 config.py 9 INFO]
cfg.MODEL.POST_PROCESSING = edict()
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
[2021-07-21 15:04:55,840 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
[2021-07-21 15:04:55,841 config.py 9 INFO]
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.1
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
[2021-07-21 15:04:55,841 config.py 9 INFO]
cfg.OPTIMIZATION = edict()
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.OPTIMIZATION.NUM_EPOCHS: 30
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
[2021-07-21 15:04:55,841 config.py 12 INFO] cfg.OPTIMIZATION.LR: 0.0015
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.WEIGHT_DECAY: 0.001
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.MOMENTUM: 0.9
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.PCT_START: 0.4
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.DIV_FACTOR: 10
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.LR_DECAY: 0.1
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.LR_CLIP: 1e-07
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.LR_WARMUP: False
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.WARMUP_EPOCH: 1
[2021-07-21 15:04:55,842 config.py 12 INFO] cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
[2021-07-21 15:04:55,843 config.py 9 INFO]
cfg.SELF_TRAIN = edict()
[2021-07-21 15:04:55,843 config.py 9 INFO]
cfg.SELF_TRAIN.SRC = edict()
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.SRC.USE_DATA: False
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.SRC.USE_GRAD: False
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.SRC.LOSS_WEIGHT: 1.0
[2021-07-21 15:04:55,843 config.py 9 INFO]
cfg.SELF_TRAIN.TAR = edict()
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.TAR.USE_DATA: True
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.TAR.LOSS_WEIGHT: 1.0
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.SCORE_THRESH: [0.6]
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.UPDATE_PSEUDO_LABEL: [0]
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.UPDATE_PSEUDO_LABEL_INTERVAL: 4
[2021-07-21 15:04:55,843 config.py 12 INFO] cfg.SELF_TRAIN.INIT_PS: None
[2021-07-21 15:04:55,844 config.py 9 INFO]
cfg.SELF_TRAIN.PROG_AUG = edict()
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.PROG_AUG.ENABLED: True
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.PROG_AUG.UPDATE_AUG: [5, 10, 20, 25]
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.PROG_AUG.SCALE: 1.2
[2021-07-21 15:04:55,844 config.py 9 INFO]
cfg.SELF_TRAIN.MEMORY_ENSEMBLE = edict()
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.ENABLED: False
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NAME: consistency_ensemble
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.IOU_THRESH: 0.1
[2021-07-21 15:04:55,844 config.py 9 INFO]
cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NMS_CONFIG = edict()
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NMS_CONFIG.NMS_TYPE: nms_gpu
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NMS_CONFIG.MULTI_CLASSES_NMS: False
[2021-07-21 15:04:55,844 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NMS_CONFIG.NMS_PRE_MAXSIZE: 512
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NMS_CONFIG.NMS_POST_MAXSIZE: 100
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.NMS_CONFIG.NMS_THRESH: 0.1
[2021-07-21 15:04:55,845 config.py 9 INFO]
cfg.SELF_TRAIN.MEMORY_ENSEMBLE.MEMORY_VOTING = edict()
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.MEMORY_VOTING.ENABLED: False
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.MEMORY_VOTING.IGNORE_THRESH: 2
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.SELF_TRAIN.MEMORY_ENSEMBLE.MEMORY_VOTING.RM_THRESH: 3
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.TAG: pvrcnn_st3d
[2021-07-21 15:04:55,845 config.py 12 INFO] cfg.EXP_GROUP_PATH: da-waymo-kitti_models/pvrcnn_st3d
[2021-07-21 15:04:55,999 waymo_dataset.py 44 INFO] Loading Waymo dataset
[2021-07-21 15:05:03,819 waymo_dataset.py 60 INFO] Total skipped info 0
[2021-07-21 15:05:03,819 waymo_dataset.py 61 INFO] Total samples for Waymo dataset: 158081
[2021-07-21 15:05:03,839 waymo_dataset.py 68 INFO] Total sampled samples for Waymo dataset: 79041
[2021-07-21 15:05:04,447 kitti_dataset.py 34 INFO] Loading KITTI dataset
[2021-07-21 15:05:04,683 kitti_dataset.py 48 INFO] Total samples for KITTI dataset: 3712
[2021-07-21 15:05:09,018 train.py 149 INFO] DistributedDataParallel(
(module): PVRCNN(
(vfe): MeanVFE()
(backbone_3d): VoxelBackBone8x(
(conv_input): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv1): SparseSequential(
(0): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv2): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv3): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv4): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv_out): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(map_to_bev_module): HeightCompression()
(pfe): VoxelSetAbstraction(
(SA_layers): ModuleList(
(0): StackSAModuleMSG(
(groupers): ModuleList(
(0): QueryAndGroup()
(1): QueryAndGroup()
)
(mlps): ModuleList(
(0): Sequential(
(0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
(1): Sequential(
(0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
)
)
(1): StackSAModuleMSG(
(groupers): ModuleList(
(0): QueryAndGroup()
(1): QueryAndGroup()
)
(mlps): ModuleList(
(0): Sequential(
(0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
(1): Sequential(
(0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
)
)
)
(SA_rawpoints): StackSAModuleMSG(
(groupers): ModuleList(
(0): QueryAndGroup()
(1): QueryAndGroup()
)
(mlps): ModuleList(
(0): Sequential(
(0): Conv2d(3, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
(1): Sequential(
(0): Conv2d(3, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
)
)
(vsa_point_feature_fusion): Sequential(
(0): Linear(in_features=544, out_features=128, bias=False)
(1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(backbone_2d): BaseBEVBackbone(
(blocks): ModuleList(
(0): Sequential(
(0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
(1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False)
(2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(3): ReLU()
(4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(9): ReLU()
(10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(12): ReLU()
(13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(15): ReLU()
(16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(18): ReLU()
)
(1): Sequential(
(0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
(1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False)
(2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(3): ReLU()
(4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(9): ReLU()
(10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(12): ReLU()
(13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(15): ReLU()
(16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(18): ReLU()
)
)
(deblocks): ModuleList(
(0): Sequential(
(0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): Sequential(
(0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
)
(dense_head): AnchorHeadSingle(
(cls_loss_func): SigmoidFocalClassificationLoss()
(reg_loss_func): WeightedSmoothL1Loss()
(dir_loss_func): WeightedCrossEntropyLoss()
(conv_cls): Conv2d(512, 2, kernel_size=(1, 1), stride=(1, 1))
(conv_box): Conv2d(512, 14, kernel_size=(1, 1), stride=(1, 1))
(conv_dir_cls): Conv2d(512, 4, kernel_size=(1, 1), stride=(1, 1))
)
(point_head): PointHeadSimple(
(cls_loss_func): SigmoidFocalClassificationLoss()
(cls_layers): Sequential(
(0): Linear(in_features=544, out_features=256, bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Linear(in_features=256, out_features=256, bias=False)
(4): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
(6): Linear(in_features=256, out_features=1, bias=True)
)
)
(roi_head): PVRCNNHead(
(proposal_target_layer): ProposalTargetLayer()
(reg_loss_func): WeightedSmoothL1Loss()
(roi_grid_pool_layer): StackSAModuleMSG(
(groupers): ModuleList(
(0): QueryAndGroup()
(1): QueryAndGroup()
)
(mlps): ModuleList(
(0): Sequential(
(0): Conv2d(131, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
(1): Sequential(
(0): Conv2d(131, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU()
)
)
)
(shared_fc_layer): Sequential(
(0): Conv1d(27648, 256, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3)
(4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU()
)
(cls_layers): Sequential(
(0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3)
(4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv1d(256, 1, kernel_size=(1,), stride=(1,))
)
(reg_layers): Sequential(
(0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3)
(4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv1d(256, 7, kernel_size=(1,), stride=(1,))
)
)
)
)
[2021-07-21 15:05:09,022 train.py 168 INFO] **********************Start training da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d(default)**********************
generate_ps_e0: 100%|████████████████████| 232/232 [03:14<00:00, 1.19it/s, pos_ps_box=0.000(0.000), ign_ps_box=15.000(14.899)]
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002080
epochs: 0%| | 0/30 [04:05<?, ?it/s]
Traceback (most recent call last): | 0/232 [00:00<?, ?it/s]
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002394
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002829
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
pid, sts = os.waitpid(self.pid, flag)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 63, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 5123) is killed by signal: Terminated.
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 001773
Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/distributed/launch.py", line 235, in <module>
main()
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/distributed/launch.py", line 231, in main
cmd=process.args)
subprocess.CalledProcessError: Command '['/home/user5/anaconda3/envs/st3d7/bin/python', '-u', 'train.py', '--local_rank=0', '--launcher', 'pytorch', '--cfg_file', 'cfgs/da-waymo-kitti_models/pvrcnn_st3d/pvrcnn_st3d.yaml']' returned non-zero exit status 1.
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 003342
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002466
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 002501
Traceback (most recent call last):
File "train.py", line 199, in <module>
main()
File "train.py", line 191, in main
ema_model=None
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 157, in train_model_st
dataloader_iter=dataloader_iter, ema_model=ema_model
File "/home/user5/open-mmlab/ST3D/tools/train_utils/train_st_utils.py", line 42, in train_one_epoch_st
target_batch = next(dataloader_iter)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/anaconda3/envs/st3d7/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/kitti/kitti_dataset.py", line 413, in __getitem__
self.fill_pseudo_labels(input_dict)
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/datasets/dataset.py", line 146, in fill_pseudo_labels
gt_boxes = self_training_utils.load_ps_label(input_dict['frame_id'])
File "/home/user5/open-mmlab/ST3D/tools/../pcdet/utils/self_training_utils.py", line 221, in load_ps_label
raise ValueError('Cannot find pseudo label for frame: %s' % frame_id)
ValueError: Cannot find pseudo label for frame: 003752
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment