diff --git "a/grounding_mini_rgbd_dat/job.log" "b/grounding_mini_rgbd_dat/job.log" new file mode 100644--- /dev/null +++ "b/grounding_mini_rgbd_dat/job.log" @@ -0,0 +1,6748 @@ +/usr/local/lib/python3.10/dist-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated +and will be removed in future. Use torchrun. +Note that --use-env is set by default in torchrun. +If your script expects `--local-rank` argument to be set, please +change it to read from `os.environ['LOCAL_RANK']` instead. See +https://pytorch.org/docs/stable/distributed.html#launch-utility for +further instructions + + warnings.warn( +[2025-02-11 14:47:41,819] torch.distributed.run: [WARNING] +[2025-02-11 14:47:41,819] torch.distributed.run: [WARNING] ***************************************** +[2025-02-11 14:47:41,819] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +[2025-02-11 14:47:41,819] torch.distributed.run: [WARNING] ***************************************** +[2025-02-11 14:47:48,206] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2025-02-11 14:47:48,207] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2025-02-11 14:47:48,207] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2025-02-11 14:47:48,207] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2025-02-11 14:47:48,207] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) +[2025-02-11 14:47:48,207] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) +df: /home/users/xuewu.lin/.triton/autotunedf: /home/users/xuewu.lin/.triton/autotunedf: /home/users/xuewu.lin/.triton/autotunedf: /home/users/xuewu.lin/.triton/autotunedf: /home/users/xuewu.lin/.triton/autotunedf: /home/users/xuewu.lin/.triton/autotune: No such file or directory +: No such file or directory +: No such file or directory +: No such file or directory +: No such file or directory +: No such file or directory + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + +[2025-02-11 14:47:48,525] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible +[2025-02-11 14:47:49,020] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-dev package with apt + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1 + [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py:56: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. + warnings.warn( +hobot-job-19878746-task-0:322:322 [0] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:322:322 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:322:322 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:322:322 [0] NCCL INFO cudaDriverVersion 12040 +NCCL version 2.18.5+cuda11.8 +hobot-job-19878746-task-0:323:323 [1] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:327:327 [5] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:326:326 [4] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:325:325 [3] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:329:329 [7] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:324:324 [2] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:328:328 [6] NCCL INFO cudaDriverVersion 12040 +hobot-job-19878746-task-0:328:328 [6] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:328:328 [6] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:328:328 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:329:329 [7] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:329:329 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:329:329 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:324:324 [2] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:324:324 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:324:324 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:323:323 [1] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:323:323 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:323:323 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:327:327 [5] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:327:327 [5] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:327:327 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:325:325 [3] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:325:325 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:325:325 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:326:326 [4] NCCL INFO Bootstrap : Using eth0:172.16.72.75<0> +hobot-job-19878746-task-0:326:326 [4] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory +hobot-job-19878746-task-0:326:326 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation +hobot-job-19878746-task-0:322:840 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:322:840 [0] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Using network Socket +hobot-job-19878746-task-0:329:843 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:329:843 [7] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Using network Socket +hobot-job-19878746-task-0:328:842 [6] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Using network Socket +hobot-job-19878746-task-0:324:844 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:324:844 [2] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Using network Socket +hobot-job-19878746-task-0:323:845 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:325:847 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Using network Socket +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:325:847 [3] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Using network Socket +hobot-job-19878746-task-0:326:848 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 0. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Failed to open libibverbs.so[.1] +hobot-job-19878746-task-0:326:848 [4] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Using network Socket +hobot-job-19878746-task-0:327:846 [5] NCCL INFO NET/Socket : Using [0]eth0:172.16.72.75<0> +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Using network Socket +hobot-job-19878746-task-0:322:840 [0] NCCL INFO comm 0x55a4ad08dc90 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 89000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:323:845 [1] NCCL INFO comm 0x559ae59024f0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8a000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:324:844 [2] NCCL INFO comm 0x5629ff2f9ba0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId d1000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:327:846 [5] NCCL INFO comm 0x55ed7a65ae20 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 18a000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:326:848 [4] NCCL INFO comm 0x55883859d9a0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 189000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:328:842 [6] NCCL INFO comm 0x557b74105780 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 1d1000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:325:847 [3] NCCL INFO comm 0x5608c9eab210 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId d2000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:329:843 [7] NCCL INFO comm 0x563b868a8d30 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 1d2000 commId 0x27de19bd0615f1b1 - Init START +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff00,00000000,0fffff00 +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff00,00000000,0fffff00 +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,00000000,ffffffff,00000000 +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,00000000,ffffffff,00000000 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,00000000,ffffffff,00000000 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff00,00000000,0fffff00 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff00,00000000,0fffff00 +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,00000000,ffffffff,00000000 +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Channel 00/02 : 0 1 2 3 4 5 6 7 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Channel 01/02 : 0 1 2 3 4 5 6 7 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P Chunksize set to 131072 +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Channel 00 : 4[4] -> 5[5] via SHM/direct/direct +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Channel 00 : 1[1] -> 2[2] via SHM/direct/direct +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Channel 00 : 6[6] -> 7[7] via SHM/direct/direct +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Channel 01 : 1[1] -> 2[2] via SHM/direct/direct +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Channel 00 : 2[2] -> 3[3] via SHM/direct/direct +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Channel 00 : 7[7] -> 0[0] via SHM/direct/direct +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Channel 01 : 4[4] -> 5[5] via SHM/direct/direct +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Channel 00 : 5[5] -> 6[6] via SHM/direct/direct +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Channel 01 : 7[7] -> 0[0] via SHM/direct/direct +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Channel 00 : 3[3] -> 4[4] via SHM/direct/direct +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Channel 01 : 6[6] -> 7[7] via SHM/direct/direct +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Channel 01 : 2[2] -> 3[3] via SHM/direct/direct +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Channel 01 : 5[5] -> 6[6] via SHM/direct/direct +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Channel 01 : 3[3] -> 4[4] via SHM/direct/direct +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Connected all rings +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Connected all rings +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Connected all rings +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Connected all rings +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Connected all rings +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Channel 00 : 7[7] -> 6[6] via SHM/direct/direct +hobot-job-19878746-task-0:329:843 [7] NCCL INFO P2P is disabled between connected GPUs 7 and 6. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Connected all rings +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Connected all rings +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Connected all rings +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Could not enable P2P between dev 7(=1d2000) and dev 6(=1d1000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Channel 01 : 7[7] -> 6[6] via SHM/direct/direct +hobot-job-19878746-task-0:322:840 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Could not enable P2P between dev 0(=89000) and dev 1(=8a000) +hobot-job-19878746-task-0:324:844 [2] NCCL INFO P2P is disabled between connected GPUs 2 and 3. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Could not enable P2P between dev 2(=d1000) and dev 3(=d2000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO P2P is disabled between connected GPUs 6 and 7. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct +hobot-job-19878746-task-0:323:845 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Could not enable P2P between dev 1(=8a000) and dev 0(=89000) +hobot-job-19878746-task-0:326:848 [4] NCCL INFO P2P is disabled between connected GPUs 4 and 5. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Could not enable P2P between dev 6(=1d1000) and dev 7(=1d2000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Could not enable P2P between dev 4(=189000) and dev 5(=18a000) +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:322:840 [0] NCCL INFO Connected all trees +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Channel 00 : 5[5] -> 4[4] via SHM/direct/direct +hobot-job-19878746-task-0:327:846 [5] NCCL INFO P2P is disabled between connected GPUs 5 and 4. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:322:840 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:322:840 [0] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Channel 00 : 2[2] -> 1[1] via SHM/direct/direct +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Could not enable P2P between dev 5(=18a000) and dev 4(=189000) +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Channel 01 : 2[2] -> 1[1] via SHM/direct/direct +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Channel 01 : 5[5] -> 4[4] via SHM/direct/direct +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Channel 00 : 6[6] -> 5[5] via SHM/direct/direct +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Channel 00 : 3[3] -> 2[2] via SHM/direct/direct +hobot-job-19878746-task-0:325:847 [3] NCCL INFO P2P is disabled between connected GPUs 3 and 2. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1. +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Channel 00 : 4[4] -> 3[3] via SHM/direct/direct +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Channel 01 : 6[6] -> 5[5] via SHM/direct/direct +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Could not enable P2P between dev 3(=d2000) and dev 2(=d1000) +hobot-job-19878746-task-0:323:845 [1] NCCL INFO Connected all trees +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Channel 01 : 4[4] -> 3[3] via SHM/direct/direct +hobot-job-19878746-task-0:329:843 [7] NCCL INFO Connected all trees +hobot-job-19878746-task-0:328:842 [6] NCCL INFO Connected all trees +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Channel 01 : 3[3] -> 2[2] via SHM/direct/direct +hobot-job-19878746-task-0:323:845 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:323:845 [1] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:329:843 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:329:843 [7] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:328:842 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:328:842 [6] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:327:846 [5] NCCL INFO Connected all trees +hobot-job-19878746-task-0:326:848 [4] NCCL INFO Connected all trees +hobot-job-19878746-task-0:324:844 [2] NCCL INFO Connected all trees +hobot-job-19878746-task-0:325:847 [3] NCCL INFO Connected all trees +hobot-job-19878746-task-0:327:846 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:327:846 [5] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:326:848 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:326:848 [4] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:324:844 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:325:847 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 +hobot-job-19878746-task-0:324:844 [2] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:325:847 [3] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer +hobot-job-19878746-task-0:323:845 [1] NCCL INFO comm 0x559ae59024f0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 8a000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:327:846 [5] NCCL INFO comm 0x55ed7a65ae20 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 18a000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:325:847 [3] NCCL INFO comm 0x5608c9eab210 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId d2000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:322:840 [0] NCCL INFO comm 0x55a4ad08dc90 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 89000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:326:848 [4] NCCL INFO comm 0x55883859d9a0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 189000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:328:842 [6] NCCL INFO comm 0x557b74105780 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId 1d1000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:329:843 [7] NCCL INFO comm 0x563b868a8d30 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId 1d2000 commId 0x27de19bd0615f1b1 - Init COMPLETE +hobot-job-19878746-task-0:324:844 [2] NCCL INFO comm 0x5629ff2f9ba0 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId d1000 commId 0x27de19bd0615f1b1 - Init COMPLETE +02/11 14:47:50 - mmengine - WARNING - Failed to search registry with scope "bip3d" in the "log_processor" registry tree. As a workaround, the current "log_processor" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "bip3d" is a correct scope, or whether the registry is initialized. +02/11 14:47:51 - mmengine - INFO - +------------------------------------------------------------ +System environment: + sys.platform: linux + Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] + CUDA available: True + MUSA available: False + numpy_random_seed: 0 + GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 4090 + CUDA_HOME: /usr/local/cuda + NVCC: Cuda compilation tools, release 11.8, V11.8.89 + GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 + PyTorch: 2.1.0+cu118 + PyTorch compiling details: PyTorch built with: + - GCC 9.3 + - C++ Version: 201703 + - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications + - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4) + - OpenMP 201511 (a.k.a. OpenMP 4.5) + - LAPACK is enabled (usually provided by MKL) + - NNPACK is enabled + - CPU capability usage: AVX512 + - CUDA Runtime 11.8 + - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_90,code=sm_90 + - CuDNN 8.9.4 + - Built with CuDNN 8.7 + - Magma 2.6.1 + - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, + + TorchVision: 0.16.0+cu118 + OpenCV: 4.10.0 + MMEngine: 0.10.4 + +Runtime environment: + cudnn_benchmark: False + mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} + dist_cfg: {'backend': 'nccl'} + seed: 0 + Distributed launcher: pytorch + Distributed training: True + GPU number: 8 +------------------------------------------------------------ + +02/11 14:47:52 - mmengine - INFO - Config: +backend_args = None +cam_standardization = True +class_names = ( + 'adhesive tape', + 'air conditioner', + 'alarm', + 'album', + 'arch', + 'backpack', + 'bag', + 'balcony', + 'ball', + 'banister', + 'bar', + 'barricade', + 'baseboard', + 'basin', + 'basket', + 'bathtub', + 'beam', + 'beanbag', + 'bed', + 'bench', + 'bicycle', + 'bidet', + 'bin', + 'blackboard', + 'blanket', + 'blinds', + 'board', + 'body loofah', + 'book', + 'boots', + 'bottle', + 'bowl', + 'box', + 'bread', + 'broom', + 'brush', + 'bucket', + 'cabinet', + 'calendar', + 'camera', + 'can', + 'candle', + 'candlestick', + 'cap', + 'car', + 'carpet', + 'cart', + 'case', + 'chair', + 'chandelier', + 'cleanser', + 'clock', + 'clothes', + 'clothes dryer', + 'coat hanger', + 'coffee maker', + 'coil', + 'column', + 'commode', + 'computer', + 'conducting wire', + 'container', + 'control', + 'copier', + 'cosmetics', + 'couch', + 'counter', + 'countertop', + 'crate', + 'crib', + 'cube', + 'cup', + 'curtain', + 'cushion', + 'decoration', + 'desk', + 'detergent', + 'device', + 'dish rack', + 'dishwasher', + 'dispenser', + 'divider', + 'door', + 'door knob', + 'doorframe', + 'doorway', + 'drawer', + 'dress', + 'dresser', + 'drum', + 'duct', + 'dumbbell', + 'dustpan', + 'dvd', + 'eraser', + 'excercise equipment', + 'fan', + 'faucet', + 'fence', + 'file', + 'fire extinguisher', + 'fireplace', + 'flowerpot', + 'flush', + 'folder', + 'food', + 'footstool', + 'frame', + 'fruit', + 'furniture', + 'garage door', + 'garbage', + 'glass', + 'globe', + 'glove', + 'grab bar', + 'grass', + 'guitar', + 'hair dryer', + 'hamper', + 'handle', + 'hanger', + 'hat', + 'headboard', + 'headphones', + 'heater', + 'helmets', + 'holder', + 'hook', + 'humidifier', + 'ironware', + 'jacket', + 'jalousie', + 'jar', + 'kettle', + 'keyboard', + 'kitchen island', + 'kitchenware', + 'knife', + 'label', + 'ladder', + 'lamp', + 'laptop', + 'ledge', + 'letter', + 'light', + 'luggage', + 'machine', + 'magazine', + 'mailbox', + 'map', + 'mask', + 'mat', + 'mattress', + 'menu', + 'microwave', + 'mirror', + 'molding', + 'monitor', + 'mop', + 'mouse', + 'napkins', + 'notebook', + 'ottoman', + 'oven', + 'pack', + 'package', + 'pad', + 'pan', + 'panel', + 'paper', + 'paper cutter', + 'partition', + 'pedestal', + 'pen', + 'person', + 'piano', + 'picture', + 'pillar', + 'pillow', + 'pipe', + 'pitcher', + 'plant', + 'plate', + 'player', + 'plug', + 'plunger', + 'pool', + 'pool table', + 'poster', + 'pot', + 'price tag', + 'printer', + 'projector', + 'purse', + 'rack', + 'radiator', + 'radio', + 'rail', + 'range hood', + 'refrigerator', + 'remote control', + 'ridge', + 'rod', + 'roll', + 'roof', + 'rope', + 'sack', + 'salt', + 'scale', + 'scissors', + 'screen', + 'seasoning', + 'shampoo', + 'sheet', + 'shelf', + 'shirt', + 'shoe', + 'shovel', + 'shower', + 'sign', + 'sink', + 'soap', + 'soap dish', + 'soap dispenser', + 'socket', + 'speaker', + 'sponge', + 'spoon', + 'stairs', + 'stall', + 'stand', + 'stapler', + 'statue', + 'steps', + 'stick', + 'stool', + 'stopcock', + 'stove', + 'structure', + 'sunglasses', + 'support', + 'switch', + 'table', + 'tablet', + 'teapot', + 'telephone', + 'thermostat', + 'tissue', + 'tissue box', + 'toaster', + 'toilet', + 'toilet paper', + 'toiletry', + 'tool', + 'toothbrush', + 'toothpaste', + 'towel', + 'toy', + 'tray', + 'treadmill', + 'trophy', + 'tube', + 'tv', + 'umbrella', + 'urn', + 'utensil', + 'vacuum cleaner', + 'vanity', + 'vase', + 'vent', + 'ventilation', + 'wardrobe', + 'washbasin', + 'washing machine', + 'water cooler', + 'water heater', + 'window', + 'window frame', + 'windowsill', + 'wine', + 'wire', + 'wood', + 'wrap', +) +common_labels = [ + 189, + 164, + 101, + 205, + 273, + 233, + 131, + 180, + 86, + 220, + 67, + 268, + 224, + 270, + 53, + 203, + 237, + 226, + 10, + 133, + 248, + 41, + 55, + 16, + 199, + 134, + 99, + 185, + 2, + 20, + 234, + 194, + 253, + 35, + 174, + 8, + 223, + 13, + 91, + 262, + 230, + 121, + 49, + 63, + 119, + 162, + 79, + 168, + 245, + 267, + 122, + 104, + 100, + 1, + 176, + 280, + 140, + 209, + 259, + 143, + 165, + 147, + 117, + 85, + 105, + 95, + 109, + 207, + 68, + 175, + 106, + 60, + 4, + 46, + 171, + 204, + 111, + 211, + 108, + 120, + 157, + 222, + 17, + 264, + 151, + 98, + 38, + 261, + 123, + 78, + 118, + 127, + 240, + 124, +] +custom_hooks = [ + dict(after_iter=False, type='EmptyCacheHook'), +] +data_root = 'data' +dataset_type = 'EmbodiedScanDetGroundingDataset' +default_hooks = dict( + checkpoint=dict(interval=1, max_keep_ckpts=3, type='CheckpointHook'), + logger=dict(interval=25, type='LoggerHook'), + param_scheduler=dict(type='ParamSchedulerHook'), + sampler_seed=dict(type='DistSamplerSeedHook'), + timer=dict(type='IterTimerHook')) +default_scope = 'bip3d' +depth = True +depth_loss = True +env_cfg = dict( + cudnn_benchmark=False, + dist_cfg=dict(backend='nccl'), + mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) +find_unused_parameters = False +head_labels = [ + 48, + 177, + 82, + 179, + 37, + 243, + 28, + 277, + 32, + 84, + 215, + 145, + 182, + 170, + 22, + 72, + 30, + 141, + 65, + 257, + 221, + 225, + 52, + 75, + 231, + 158, + 236, + 156, + 47, + 74, + 6, + 18, + 71, + 242, + 217, + 251, + 66, + 263, + 5, + 45, + 14, + 73, + 278, + 198, + 24, + 23, + 196, + 252, + 19, + 135, + 26, + 229, + 183, + 200, + 107, + 272, + 246, + 269, + 125, + 59, + 279, + 15, + 163, + 258, + 57, + 195, + 51, + 88, + 97, + 58, + 102, + 36, + 137, + 31, + 80, + 160, + 155, + 61, + 238, + 96, + 190, + 25, + 219, + 152, + 142, + 201, + 274, + 249, + 178, + 192, +] +if_cluster = True +image_wh = ( + 512, + 512, +) +launcher = 'pytorch' +load_from = 'http://svcspawner.bcloud.hobot.cc/user/homespace/xuewu.lin/plat_gpu/bip3d_det_withdepth_img18-20250209-112800.789859/output/work_dirs/epoch_24.pth' +log_level = 'INFO' +log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50) +lr = 0.0002 +max_depth = 10 +max_epochs = 2 +metainfo = dict( + box_type_3d='euler-depth', + classes=( + 'adhesive tape', + 'air conditioner', + 'alarm', + 'album', + 'arch', + 'backpack', + 'bag', + 'balcony', + 'ball', + 'banister', + 'bar', + 'barricade', + 'baseboard', + 'basin', + 'basket', + 'bathtub', + 'beam', + 'beanbag', + 'bed', + 'bench', + 'bicycle', + 'bidet', + 'bin', + 'blackboard', + 'blanket', + 'blinds', + 'board', + 'body loofah', + 'book', + 'boots', + 'bottle', + 'bowl', + 'box', + 'bread', + 'broom', + 'brush', + 'bucket', + 'cabinet', + 'calendar', + 'camera', + 'can', + 'candle', + 'candlestick', + 'cap', + 'car', + 'carpet', + 'cart', + 'case', + 'chair', + 'chandelier', + 'cleanser', + 'clock', + 'clothes', + 'clothes dryer', + 'coat hanger', + 'coffee maker', + 'coil', + 'column', + 'commode', + 'computer', + 'conducting wire', + 'container', + 'control', + 'copier', + 'cosmetics', + 'couch', + 'counter', + 'countertop', + 'crate', + 'crib', + 'cube', + 'cup', + 'curtain', + 'cushion', + 'decoration', + 'desk', + 'detergent', + 'device', + 'dish rack', + 'dishwasher', + 'dispenser', + 'divider', + 'door', + 'door knob', + 'doorframe', + 'doorway', + 'drawer', + 'dress', + 'dresser', + 'drum', + 'duct', + 'dumbbell', + 'dustpan', + 'dvd', + 'eraser', + 'excercise equipment', + 'fan', + 'faucet', + 'fence', + 'file', + 'fire extinguisher', + 'fireplace', + 'flowerpot', + 'flush', + 'folder', + 'food', + 'footstool', + 'frame', + 'fruit', + 'furniture', + 'garage door', + 'garbage', + 'glass', + 'globe', + 'glove', + 'grab bar', + 'grass', + 'guitar', + 'hair dryer', + 'hamper', + 'handle', + 'hanger', + 'hat', + 'headboard', + 'headphones', + 'heater', + 'helmets', + 'holder', + 'hook', + 'humidifier', + 'ironware', + 'jacket', + 'jalousie', + 'jar', + 'kettle', + 'keyboard', + 'kitchen island', + 'kitchenware', + 'knife', + 'label', + 'ladder', + 'lamp', + 'laptop', + 'ledge', + 'letter', + 'light', + 'luggage', + 'machine', + 'magazine', + 'mailbox', + 'map', + 'mask', + 'mat', + 'mattress', + 'menu', + 'microwave', + 'mirror', + 'molding', + 'monitor', + 'mop', + 'mouse', + 'napkins', + 'notebook', + 'ottoman', + 'oven', + 'pack', + 'package', + 'pad', + 'pan', + 'panel', + 'paper', + 'paper cutter', + 'partition', + 'pedestal', + 'pen', + 'person', + 'piano', + 'picture', + 'pillar', + 'pillow', + 'pipe', + 'pitcher', + 'plant', + 'plate', + 'player', + 'plug', + 'plunger', + 'pool', + 'pool table', + 'poster', + 'pot', + 'price tag', + 'printer', + 'projector', + 'purse', + 'rack', + 'radiator', + 'radio', + 'rail', + 'range hood', + 'refrigerator', + 'remote control', + 'ridge', + 'rod', + 'roll', + 'roof', + 'rope', + 'sack', + 'salt', + 'scale', + 'scissors', + 'screen', + 'seasoning', + 'shampoo', + 'sheet', + 'shelf', + 'shirt', + 'shoe', + 'shovel', + 'shower', + 'sign', + 'sink', + 'soap', + 'soap dish', + 'soap dispenser', + 'socket', + 'speaker', + 'sponge', + 'spoon', + 'stairs', + 'stall', + 'stand', + 'stapler', + 'statue', + 'steps', + 'stick', + 'stool', + 'stopcock', + 'stove', + 'structure', + 'sunglasses', + 'support', + 'switch', + 'table', + 'tablet', + 'teapot', + 'telephone', + 'thermostat', + 'tissue', + 'tissue box', + 'toaster', + 'toilet', + 'toilet paper', + 'toiletry', + 'tool', + 'toothbrush', + 'toothpaste', + 'towel', + 'toy', + 'tray', + 'treadmill', + 'trophy', + 'tube', + 'tv', + 'umbrella', + 'urn', + 'utensil', + 'vacuum cleaner', + 'vanity', + 'vase', + 'vent', + 'ventilation', + 'wardrobe', + 'washbasin', + 'washing machine', + 'water cooler', + 'water heater', + 'window', + 'window frame', + 'windowsill', + 'wine', + 'wire', + 'wood', + 'wrap', + ), + classes_split=( + [ + 48, + 177, + 82, + 179, + 37, + 243, + 28, + 277, + 32, + 84, + 215, + 145, + 182, + 170, + 22, + 72, + 30, + 141, + 65, + 257, + 221, + 225, + 52, + 75, + 231, + 158, + 236, + 156, + 47, + 74, + 6, + 18, + 71, + 242, + 217, + 251, + 66, + 263, + 5, + 45, + 14, + 73, + 278, + 198, + 24, + 23, + 196, + 252, + 19, + 135, + 26, + 229, + 183, + 200, + 107, + 272, + 246, + 269, + 125, + 59, + 279, + 15, + 163, + 258, + 57, + 195, + 51, + 88, + 97, + 58, + 102, + 36, + 137, + 31, + 80, + 160, + 155, + 61, + 238, + 96, + 190, + 25, + 219, + 152, + 142, + 201, + 274, + 249, + 178, + 192, + ], + [ + 189, + 164, + 101, + 205, + 273, + 233, + 131, + 180, + 86, + 220, + 67, + 268, + 224, + 270, + 53, + 203, + 237, + 226, + 10, + 133, + 248, + 41, + 55, + 16, + 199, + 134, + 99, + 185, + 2, + 20, + 234, + 194, + 253, + 35, + 174, + 8, + 223, + 13, + 91, + 262, + 230, + 121, + 49, + 63, + 119, + 162, + 79, + 168, + 245, + 267, + 122, + 104, + 100, + 1, + 176, + 280, + 140, + 209, + 259, + 143, + 165, + 147, + 117, + 85, + 105, + 95, + 109, + 207, + 68, + 175, + 106, + 60, + 4, + 46, + 171, + 204, + 111, + 211, + 108, + 120, + 157, + 222, + 17, + 264, + 151, + 98, + 38, + 261, + 123, + 78, + 118, + 127, + 240, + 124, + ], + [ + 76, + 149, + 173, + 250, + 275, + 255, + 34, + 77, + 266, + 283, + 112, + 115, + 186, + 136, + 256, + 40, + 254, + 172, + 9, + 212, + 213, + 181, + 154, + 94, + 191, + 193, + 3, + 130, + 146, + 70, + 128, + 167, + 126, + 81, + 7, + 11, + 148, + 228, + 239, + 247, + 21, + 42, + 89, + 153, + 161, + 244, + 110, + 0, + 29, + 114, + 132, + 159, + 218, + 232, + 260, + 56, + 92, + 116, + 282, + 33, + 113, + 138, + 12, + 188, + 44, + 150, + 197, + 271, + 169, + 206, + 90, + 235, + 103, + 281, + 184, + 208, + 216, + 202, + 214, + 241, + 129, + 210, + 276, + 64, + 27, + 87, + 139, + 227, + 187, + 62, + 43, + 50, + 69, + 93, + 144, + 166, + 265, + 54, + 83, + 39, + ], + )) +min_depth = 0.25 +model = dict( + backbone=dict( + attn_drop_rate=0.0, + convert_weights=False, + depths=[ + 2, + 2, + 6, + 2, + ], + drop_path_rate=0.2, + drop_rate=0.0, + embed_dims=96, + mlp_ratio=4, + num_heads=[ + 3, + 6, + 12, + 24, + ], + out_indices=( + 1, + 2, + 3, + ), + patch_norm=True, + qk_scale=None, + qkv_bias=True, + type='mmdet.SwinTransformer', + window_size=7, + with_cp=True), + backbone_3d=dict( + base_channels=4, + depth=34, + in_channels=1, + norm_cfg=dict(requires_grad=True, type='BN'), + norm_eval=True, + num_stages=4, + out_indices=( + 1, + 2, + 3, + ), + style='pytorch', + type='mmdet.ResNet', + with_cp=True), + data_preprocessor=dict( + bgr_to_rgb=True, + mean=[ + 123.675, + 116.28, + 103.53, + ], + pad_size_divisor=32, + std=[ + 58.395, + 57.12, + 57.375, + ], + type='CustomDet3DDataPreprocessor'), + decoder=dict( + anchor_encoder=dict(embed_dims=256, rot_dims=3, type='DoF9BoxEncoder'), + deformable_model=dict( + embed_dims=256, + filter_outlier=True, + kps_generator=dict( + fix_scale=[ + [ + 0, + 0, + 0, + ], + [ + 0.45, + 0, + 0, + ], + [ + -0.45, + 0, + 0, + ], + [ + 0, + 0.45, + 0, + ], + [ + 0, + -0.45, + 0, + ], + [ + 0, + 0, + 0.45, + ], + [ + 0, + 0, + -0.45, + ], + ], + num_learnable_pts=9, + type='SparseBox3DKeyPointsGenerator'), + max_depth=10, + min_depth=0.25, + num_groups=8, + num_levels=4, + type='DeformableFeatureAggregation', + use_camera_embed=True, + use_deformable_func=True, + with_depth=True, + with_value_proj=True), + ffn=dict( + embed_dims=256, + feedforward_channels=2048, + ffn_drop=0.0, + type='FFN'), + graph_model=dict( + batch_first=True, + dropout=0.0, + embed_dims=256, + num_heads=8, + type='MultiheadAttention'), + gt_cls_key='tokens_positive', + gt_reg_key='gt_bboxes_3d', + instance_bank=dict( + anchor='anchor_files/embodiedscan_kmeans_det_cam_log_z-0.2-3.npy', + anchor_in_camera=True, + embed_dims=256, + num_anchor=50, + type='InstanceBank'), + look_forward_twice=True, + loss_cls=dict( + alpha=0.25, + gamma=2.0, + loss_weight=1.0, + type='mmdet.FocalLoss', + use_sigmoid=True), + loss_reg=dict( + loss_weight_cd=0.8, + loss_weight_pcd=0.0, + loss_weight_wd=1.0, + type='DoF9BoxLoss'), + norm_layer=dict(normalized_shape=256, type='LN'), + post_processor=dict(num_output=1000, type='GroundingBox3DPostProcess'), + refine_layer=dict( + cls_bias=True, + embed_dims=256, + output_dim=9, + type='GroundingRefineClsHead'), + sampler=dict( + box_weight=1.0, + cls_weight=1.0, + cost_weight_cd=0.8, + cost_weight_pcd=0.0, + cost_weight_wd=1.0, + embed_dims=256, + num_classes=284, + num_dn=100, + type='Grounding3DTarget', + with_dn_query=True), + text_cross_attn=dict( + batch_first=True, + dropout=0.0, + embed_dims=256, + num_heads=8, + type='MultiheadAttention'), + type='BBox3DDecoder', + with_instance_id=False), + feature_enhancer=dict( + embed_dims=256, + img_attn_block=dict( + ffn_cfg=dict( + embed_dims=256, feedforward_channels=2048, ffn_drop=0.0), + self_attn_cfg=dict( + dropout=0.0, embed_dims=256, im2col_step=64, num_levels=4)), + num_feature_levels=4, + num_layers=6, + positional_encoding=dict( + normalize=True, num_feats=128, offset=0.0, temperature=20), + text_attn_block=dict( + ffn_cfg=dict( + embed_dims=256, feedforward_channels=1024, ffn_drop=0.0), + self_attn_cfg=dict(dropout=0.0, embed_dims=256, num_heads=4)), + text_img_attn_block=dict( + embed_dim=1024, + init_values=0.0001, + l_dim=256, + num_heads=4, + v_dim=256), + type='TextImageDeformable2DEnhancer'), + input_3d='depth_img', + neck=dict( + act_cfg=None, + bias=True, + in_channels=[ + 192, + 384, + 768, + ], + kernel_size=1, + norm_cfg=dict(num_groups=32, type='GN'), + num_outs=4, + out_channels=256, + type='mmdet.ChannelMapper'), + neck_3d=dict( + act_cfg=None, + bias=True, + in_channels=[ + 8, + 16, + 32, + ], + kernel_size=1, + norm_cfg=dict(num_groups=4, type='GN'), + num_outs=4, + out_channels=32, + type='mmdet.ChannelMapper'), + spatial_enhancer=dict( + embed_dims=256, + feature_3d_dim=32, + loss_depth_weight=1.0, + max_depth=10, + min_depth=0.25, + num_depth=64, + num_depth_layers=2, + type='DepthFusionSpatialEnhancer', + with_feature_3d=True), + text_encoder=dict( + add_pooling_layer=False, + max_tokens=768, + name='./ckpt/bert-base-uncased', + pad_to_max=False, + return_tokenized=True, + special_tokens_list=[ + '[CLS]', + '[SEP]', + ], + type='BertModel', + use_checkpoint=True, + use_sub_sentence_represent=True), + type='BIP3D', + use_depth_grid_mask=True) +num_depth = 64 +optim_wrapper = dict( + clip_grad=dict(max_norm=10, norm_type=2), + optimizer=dict(lr=0.0002, type='AdamW', weight_decay=0.0005), + paramwise_cfg=dict( + custom_keys=dict({ + 'absolute_pos_embed': dict(decay_mult=0.0), + 'backbone.': dict(lr_mult=0.1) + })), + type='OptimWrapper') +param_scheduler = [ + dict( + begin=0, by_epoch=False, end=500, start_factor=0.001, type='LinearLR'), + dict( + begin=0, + by_epoch=False, + end=62000, + gamma=0.1, + milestones=[ + 40000, + 55000, + ], + type='MultiStepLR'), +] +randomness = dict(seed=0) +resize = dict( + dst_intrinsic=[ + [ + 432.57943431339237, + 0.0, + 256, + ], + [ + 0.0, + 539.8570854208559, + 256, + ], + [ + 0.0, + 0.0, + 1.0, + ], + ], + dst_wh=( + 512, + 512, + ), + type='CamIntrisicStandardization') +resume = False +rotate_3rscan = True +sep_token = '[SEP]' +swin_base = False +tail_labels = [ + 76, + 149, + 173, + 250, + 275, + 255, + 34, + 77, + 266, + 283, + 112, + 115, + 186, + 136, + 256, + 40, + 254, + 172, + 9, + 212, + 213, + 181, + 154, + 94, + 191, + 193, + 3, + 130, + 146, + 70, + 128, + 167, + 126, + 81, + 7, + 11, + 148, + 228, + 239, + 247, + 21, + 42, + 89, + 153, + 161, + 244, + 110, + 0, + 29, + 114, + 132, + 159, + 218, + 232, + 260, + 56, + 92, + 116, + 282, + 33, + 113, + 138, + 12, + 188, + 44, + 150, + 197, + 271, + 169, + 206, + 90, + 235, + 103, + 281, + 184, + 208, + 216, + 202, + 214, + 241, + 129, + 210, + 276, + 64, + 27, + 87, + 139, + 227, + 187, + 62, + 43, + 50, + 69, + 93, + 144, + 166, + 265, + 54, + 83, + 39, +] +test_cfg = dict(type='TestLoop') +test_dataloader = dict( + batch_size=1, + dataset=dict( + ann_file='embodiedscan/embodiedscan_infos_val.pkl', + box_type_3d='Euler-Depth', + data_root='data', + filter_empty_gt=True, + metainfo=dict( + box_type_3d='euler-depth', + classes=( + 'adhesive tape', + 'air conditioner', + 'alarm', + 'album', + 'arch', + 'backpack', + 'bag', + 'balcony', + 'ball', + 'banister', + 'bar', + 'barricade', + 'baseboard', + 'basin', + 'basket', + 'bathtub', + 'beam', + 'beanbag', + 'bed', + 'bench', + 'bicycle', + 'bidet', + 'bin', + 'blackboard', + 'blanket', + 'blinds', + 'board', + 'body loofah', + 'book', + 'boots', + 'bottle', + 'bowl', + 'box', + 'bread', + 'broom', + 'brush', + 'bucket', + 'cabinet', + 'calendar', + 'camera', + 'can', + 'candle', + 'candlestick', + 'cap', + 'car', + 'carpet', + 'cart', + 'case', + 'chair', + 'chandelier', + 'cleanser', + 'clock', + 'clothes', + 'clothes dryer', + 'coat hanger', + 'coffee maker', + 'coil', + 'column', + 'commode', + 'computer', + 'conducting wire', + 'container', + 'control', + 'copier', + 'cosmetics', + 'couch', + 'counter', + 'countertop', + 'crate', + 'crib', + 'cube', + 'cup', + 'curtain', + 'cushion', + 'decoration', + 'desk', + 'detergent', + 'device', + 'dish rack', + 'dishwasher', + 'dispenser', + 'divider', + 'door', + 'door knob', + 'doorframe', + 'doorway', + 'drawer', + 'dress', + 'dresser', + 'drum', + 'duct', + 'dumbbell', + 'dustpan', + 'dvd', + 'eraser', + 'excercise equipment', + 'fan', + 'faucet', + 'fence', + 'file', + 'fire extinguisher', + 'fireplace', + 'flowerpot', + 'flush', + 'folder', + 'food', + 'footstool', + 'frame', + 'fruit', + 'furniture', + 'garage door', + 'garbage', + 'glass', + 'globe', + 'glove', + 'grab bar', + 'grass', + 'guitar', + 'hair dryer', + 'hamper', + 'handle', + 'hanger', + 'hat', + 'headboard', + 'headphones', + 'heater', + 'helmets', + 'holder', + 'hook', + 'humidifier', + 'ironware', + 'jacket', + 'jalousie', + 'jar', + 'kettle', + 'keyboard', + 'kitchen island', + 'kitchenware', + 'knife', + 'label', + 'ladder', + 'lamp', + 'laptop', + 'ledge', + 'letter', + 'light', + 'luggage', + 'machine', + 'magazine', + 'mailbox', + 'map', + 'mask', + 'mat', + 'mattress', + 'menu', + 'microwave', + 'mirror', + 'molding', + 'monitor', + 'mop', + 'mouse', + 'napkins', + 'notebook', + 'ottoman', + 'oven', + 'pack', + 'package', + 'pad', + 'pan', + 'panel', + 'paper', + 'paper cutter', + 'partition', + 'pedestal', + 'pen', + 'person', + 'piano', + 'picture', + 'pillar', + 'pillow', + 'pipe', + 'pitcher', + 'plant', + 'plate', + 'player', + 'plug', + 'plunger', + 'pool', + 'pool table', + 'poster', + 'pot', + 'price tag', + 'printer', + 'projector', + 'purse', + 'rack', + 'radiator', + 'radio', + 'rail', + 'range hood', + 'refrigerator', + 'remote control', + 'ridge', + 'rod', + 'roll', + 'roof', + 'rope', + 'sack', + 'salt', + 'scale', + 'scissors', + 'screen', + 'seasoning', + 'shampoo', + 'sheet', + 'shelf', + 'shirt', + 'shoe', + 'shovel', + 'shower', + 'sign', + 'sink', + 'soap', + 'soap dish', + 'soap dispenser', + 'socket', + 'speaker', + 'sponge', + 'spoon', + 'stairs', + 'stall', + 'stand', + 'stapler', + 'statue', + 'steps', + 'stick', + 'stool', + 'stopcock', + 'stove', + 'structure', + 'sunglasses', + 'support', + 'switch', + 'table', + 'tablet', + 'teapot', + 'telephone', + 'thermostat', + 'tissue', + 'tissue box', + 'toaster', + 'toilet', + 'toilet paper', + 'toiletry', + 'tool', + 'toothbrush', + 'toothpaste', + 'towel', + 'toy', + 'tray', + 'treadmill', + 'trophy', + 'tube', + 'tv', + 'umbrella', + 'urn', + 'utensil', + 'vacuum cleaner', + 'vanity', + 'vase', + 'vent', + 'ventilation', + 'wardrobe', + 'washbasin', + 'washing machine', + 'water cooler', + 'water heater', + 'window', + 'window frame', + 'windowsill', + 'wine', + 'wire', + 'wood', + 'wrap', + ), + classes_split=( + [ + 48, + 177, + 82, + 179, + 37, + 243, + 28, + 277, + 32, + 84, + 215, + 145, + 182, + 170, + 22, + 72, + 30, + 141, + 65, + 257, + 221, + 225, + 52, + 75, + 231, + 158, + 236, + 156, + 47, + 74, + 6, + 18, + 71, + 242, + 217, + 251, + 66, + 263, + 5, + 45, + 14, + 73, + 278, + 198, + 24, + 23, + 196, + 252, + 19, + 135, + 26, + 229, + 183, + 200, + 107, + 272, + 246, + 269, + 125, + 59, + 279, + 15, + 163, + 258, + 57, + 195, + 51, + 88, + 97, + 58, + 102, + 36, + 137, + 31, + 80, + 160, + 155, + 61, + 238, + 96, + 190, + 25, + 219, + 152, + 142, + 201, + 274, + 249, + 178, + 192, + ], + [ + 189, + 164, + 101, + 205, + 273, + 233, + 131, + 180, + 86, + 220, + 67, + 268, + 224, + 270, + 53, + 203, + 237, + 226, + 10, + 133, + 248, + 41, + 55, + 16, + 199, + 134, + 99, + 185, + 2, + 20, + 234, + 194, + 253, + 35, + 174, + 8, + 223, + 13, + 91, + 262, + 230, + 121, + 49, + 63, + 119, + 162, + 79, + 168, + 245, + 267, + 122, + 104, + 100, + 1, + 176, + 280, + 140, + 209, + 259, + 143, + 165, + 147, + 117, + 85, + 105, + 95, + 109, + 207, + 68, + 175, + 106, + 60, + 4, + 46, + 171, + 204, + 111, + 211, + 108, + 120, + 157, + 222, + 17, + 264, + 151, + 98, + 38, + 261, + 123, + 78, + 118, + 127, + 240, + 124, + ], + [ + 76, + 149, + 173, + 250, + 275, + 255, + 34, + 77, + 266, + 283, + 112, + 115, + 186, + 136, + 256, + 40, + 254, + 172, + 9, + 212, + 213, + 181, + 154, + 94, + 191, + 193, + 3, + 130, + 146, + 70, + 128, + 167, + 126, + 81, + 7, + 11, + 148, + 228, + 239, + 247, + 21, + 42, + 89, + 153, + 161, + 244, + 110, + 0, + 29, + 114, + 132, + 159, + 218, + 232, + 260, + 56, + 92, + 116, + 282, + 33, + 113, + 138, + 12, + 188, + 44, + 150, + 197, + 271, + 169, + 206, + 90, + 235, + 103, + 281, + 184, + 208, + 216, + 202, + 214, + 241, + 129, + 210, + 276, + 64, + 27, + 87, + 139, + 227, + 187, + 62, + 43, + 50, + 69, + 93, + 144, + 166, + 265, + 54, + 83, + 39, + ], + )), + mode='grounding', + pipeline=[ + dict(type='LoadAnnotations3D'), + dict( + max_n_images=50, + n_images=1, + ordered=True, + rotate_3rscan=True, + transforms=[ + dict(backend_args=None, type='LoadImageFromFile'), + dict(backend_args=None, type='LoadDepthFromFile'), + dict( + dst_intrinsic=[ + [ + 432.57943431339237, + 0.0, + 256, + ], + [ + 0.0, + 539.8570854208559, + 256, + ], + [ + 0.0, + 0.0, + 1.0, + ], + ], + dst_wh=( + 512, + 512, + ), + type='CamIntrisicStandardization'), + ], + type='MultiViewPipeline'), + dict( + keys=[ + 'img', + 'depth_img', + 'gt_bboxes_3d', + 'gt_labels_3d', + ], + type='Pack3DDetInputs'), + ], + test_mode=True, + type='EmbodiedScanDetGroundingDataset', + vg_file='embodiedscan/embodiedscan_val_mini_vg.json'), + drop_last=False, + num_workers=4, + persistent_workers=False, + sampler=dict(shuffle=False, type='DefaultSampler')) +test_evaluator = dict(type='GroundingMetric') +test_pipeline = [ + dict(type='LoadAnnotations3D'), + dict( + max_n_images=50, + n_images=1, + ordered=True, + rotate_3rscan=True, + transforms=[ + dict(backend_args=None, type='LoadImageFromFile'), + dict(backend_args=None, type='LoadDepthFromFile'), + dict( + dst_intrinsic=[ + [ + 432.57943431339237, + 0.0, + 256, + ], + [ + 0.0, + 539.8570854208559, + 256, + ], + [ + 0.0, + 0.0, + 1.0, + ], + ], + dst_wh=( + 512, + 512, + ), + type='CamIntrisicStandardization'), + ], + type='MultiViewPipeline'), + dict( + keys=[ + 'img', + 'depth_img', + 'gt_bboxes_3d', + 'gt_labels_3d', + ], + type='Pack3DDetInputs'), +] +train_cfg = dict(max_epochs=2, type='EpochBasedTrainLoop', val_interval=1) +train_dataloader = dict( + batch_size=1, + dataset=dict( + ann_file='embodiedscan/embodiedscan_infos_train.pkl', + box_type_3d='Euler-Depth', + data_root='data', + filter_empty_gt=True, + metainfo=dict( + box_type_3d='euler-depth', + classes=( + 'adhesive tape', + 'air conditioner', + 'alarm', + 'album', + 'arch', + 'backpack', + 'bag', + 'balcony', + 'ball', + 'banister', + 'bar', + 'barricade', + 'baseboard', + 'basin', + 'basket', + 'bathtub', + 'beam', + 'beanbag', + 'bed', + 'bench', + 'bicycle', + 'bidet', + 'bin', + 'blackboard', + 'blanket', + 'blinds', + 'board', + 'body loofah', + 'book', + 'boots', + 'bottle', + 'bowl', + 'box', + 'bread', + 'broom', + 'brush', + 'bucket', + 'cabinet', + 'calendar', + 'camera', + 'can', + 'candle', + 'candlestick', + 'cap', + 'car', + 'carpet', + 'cart', + 'case', + 'chair', + 'chandelier', + 'cleanser', + 'clock', + 'clothes', + 'clothes dryer', + 'coat hanger', + 'coffee maker', + 'coil', + 'column', + 'commode', + 'computer', + 'conducting wire', + 'container', + 'control', + 'copier', + 'cosmetics', + 'couch', + 'counter', + 'countertop', + 'crate', + 'crib', + 'cube', + 'cup', + 'curtain', + 'cushion', + 'decoration', + 'desk', + 'detergent', + 'device', + 'dish rack', + 'dishwasher', + 'dispenser', + 'divider', + 'door', + 'door knob', + 'doorframe', + 'doorway', + 'drawer', + 'dress', + 'dresser', + 'drum', + 'duct', + 'dumbbell', + 'dustpan', + 'dvd', + 'eraser', + 'excercise equipment', + 'fan', + 'faucet', + 'fence', + 'file', + 'fire extinguisher', + 'fireplace', + 'flowerpot', + 'flush', + 'folder', + 'food', + 'footstool', + 'frame', + 'fruit', + 'furniture', + 'garage door', + 'garbage', + 'glass', + 'globe', + 'glove', + 'grab bar', + 'grass', + 'guitar', + 'hair dryer', + 'hamper', + 'handle', + 'hanger', + 'hat', + 'headboard', + 'headphones', + 'heater', + 'helmets', + 'holder', + 'hook', + 'humidifier', + 'ironware', + 'jacket', + 'jalousie', + 'jar', + 'kettle', + 'keyboard', + 'kitchen island', + 'kitchenware', + 'knife', + 'label', + 'ladder', + 'lamp', + 'laptop', + 'ledge', + 'letter', + 'light', + 'luggage', + 'machine', + 'magazine', + 'mailbox', + 'map', + 'mask', + 'mat', + 'mattress', + 'menu', + 'microwave', + 'mirror', + 'molding', + 'monitor', + 'mop', + 'mouse', + 'napkins', + 'notebook', + 'ottoman', + 'oven', + 'pack', + 'package', + 'pad', + 'pan', + 'panel', + 'paper', + 'paper cutter', + 'partition', + 'pedestal', + 'pen', + 'person', + 'piano', + 'picture', + 'pillar', + 'pillow', + 'pipe', + 'pitcher', + 'plant', + 'plate', + 'player', + 'plug', + 'plunger', + 'pool', + 'pool table', + 'poster', + 'pot', + 'price tag', + 'printer', + 'projector', + 'purse', + 'rack', + 'radiator', + 'radio', + 'rail', + 'range hood', + 'refrigerator', + 'remote control', + 'ridge', + 'rod', + 'roll', + 'roof', + 'rope', + 'sack', + 'salt', + 'scale', + 'scissors', + 'screen', + 'seasoning', + 'shampoo', + 'sheet', + 'shelf', + 'shirt', + 'shoe', + 'shovel', + 'shower', + 'sign', + 'sink', + 'soap', + 'soap dish', + 'soap dispenser', + 'socket', + 'speaker', + 'sponge', + 'spoon', + 'stairs', + 'stall', + 'stand', + 'stapler', + 'statue', + 'steps', + 'stick', + 'stool', + 'stopcock', + 'stove', + 'structure', + 'sunglasses', + 'support', + 'switch', + 'table', + 'tablet', + 'teapot', + 'telephone', + 'thermostat', + 'tissue', + 'tissue box', + 'toaster', + 'toilet', + 'toilet paper', + 'toiletry', + 'tool', + 'toothbrush', + 'toothpaste', + 'towel', + 'toy', + 'tray', + 'treadmill', + 'trophy', + 'tube', + 'tv', + 'umbrella', + 'urn', + 'utensil', + 'vacuum cleaner', + 'vanity', + 'vase', + 'vent', + 'ventilation', + 'wardrobe', + 'washbasin', + 'washing machine', + 'water cooler', + 'water heater', + 'window', + 'window frame', + 'windowsill', + 'wine', + 'wire', + 'wood', + 'wrap', + ), + classes_split=( + [ + 48, + 177, + 82, + 179, + 37, + 243, + 28, + 277, + 32, + 84, + 215, + 145, + 182, + 170, + 22, + 72, + 30, + 141, + 65, + 257, + 221, + 225, + 52, + 75, + 231, + 158, + 236, + 156, + 47, + 74, + 6, + 18, + 71, + 242, + 217, + 251, + 66, + 263, + 5, + 45, + 14, + 73, + 278, + 198, + 24, + 23, + 196, + 252, + 19, + 135, + 26, + 229, + 183, + 200, + 107, + 272, + 246, + 269, + 125, + 59, + 279, + 15, + 163, + 258, + 57, + 195, + 51, + 88, + 97, + 58, + 102, + 36, + 137, + 31, + 80, + 160, + 155, + 61, + 238, + 96, + 190, + 25, + 219, + 152, + 142, + 201, + 274, + 249, + 178, + 192, + ], + [ + 189, + 164, + 101, + 205, + 273, + 233, + 131, + 180, + 86, + 220, + 67, + 268, + 224, + 270, + 53, + 203, + 237, + 226, + 10, + 133, + 248, + 41, + 55, + 16, + 199, + 134, + 99, + 185, + 2, + 20, + 234, + 194, + 253, + 35, + 174, + 8, + 223, + 13, + 91, + 262, + 230, + 121, + 49, + 63, + 119, + 162, + 79, + 168, + 245, + 267, + 122, + 104, + 100, + 1, + 176, + 280, + 140, + 209, + 259, + 143, + 165, + 147, + 117, + 85, + 105, + 95, + 109, + 207, + 68, + 175, + 106, + 60, + 4, + 46, + 171, + 204, + 111, + 211, + 108, + 120, + 157, + 222, + 17, + 264, + 151, + 98, + 38, + 261, + 123, + 78, + 118, + 127, + 240, + 124, + ], + [ + 76, + 149, + 173, + 250, + 275, + 255, + 34, + 77, + 266, + 283, + 112, + 115, + 186, + 136, + 256, + 40, + 254, + 172, + 9, + 212, + 213, + 181, + 154, + 94, + 191, + 193, + 3, + 130, + 146, + 70, + 128, + 167, + 126, + 81, + 7, + 11, + 148, + 228, + 239, + 247, + 21, + 42, + 89, + 153, + 161, + 244, + 110, + 0, + 29, + 114, + 132, + 159, + 218, + 232, + 260, + 56, + 92, + 116, + 282, + 33, + 113, + 138, + 12, + 188, + 44, + 150, + 197, + 271, + 169, + 206, + 90, + 235, + 103, + 281, + 184, + 208, + 216, + 202, + 214, + 241, + 129, + 210, + 276, + 64, + 27, + 87, + 139, + 227, + 187, + 62, + 43, + 50, + 69, + 93, + 144, + 166, + 265, + 54, + 83, + 39, + ], + )), + mode='grounding', + num_text=10, + pipeline=[ + dict(type='LoadAnnotations3D'), + dict( + max_n_images=18, + n_images=1, + ordered=True, + rotate_3rscan=True, + transforms=[ + dict(backend_args=None, type='LoadImageFromFile'), + dict(backend_args=None, type='LoadDepthFromFile'), + dict( + dst_intrinsic=[ + [ + 432.57943431339237, + 0.0, + 256, + ], + [ + 0.0, + 539.8570854208559, + 256, + ], + [ + 0.0, + 0.0, + 1.0, + ], + ], + dst_wh=( + 512, + 512, + ), + type='CamIntrisicStandardization'), + dict( + data_aug_conf=dict( + H=512, + W=512, + bot_pct_lim=( + 0.0, + 0.05, + ), + final_dim=( + 512, + 512, + ), + rand_flip=False, + resize_lim=( + 1.0, + 1.0, + ), + rot_lim=( + 0, + 0, + )), + type='ResizeCropFlipImage'), + ], + type='MultiViewPipeline'), + dict( + keys=[ + 'img', + 'depth_img', + 'gt_bboxes_3d', + 'gt_labels_3d', + ], + type='Pack3DDetInputs'), + dict( + max_depth=10, + min_depth=0.25, + num_depth=64, + origin_stride=4, + type='DepthProbLabelGenerator'), + ], + sep_token='[SEP]', + test_mode=False, + type='EmbodiedScanDetGroundingDataset', + vg_file='embodiedscan/embodiedscan_train_mini_vg.json'), + num_workers=4, + persistent_workers=False, + sampler=dict(shuffle=True, type='DefaultSampler')) +train_pipeline = [ + dict(type='LoadAnnotations3D'), + dict( + max_n_images=18, + n_images=1, + ordered=True, + rotate_3rscan=True, + transforms=[ + dict(backend_args=None, type='LoadImageFromFile'), + dict(backend_args=None, type='LoadDepthFromFile'), + dict( + dst_intrinsic=[ + [ + 432.57943431339237, + 0.0, + 256, + ], + [ + 0.0, + 539.8570854208559, + 256, + ], + [ + 0.0, + 0.0, + 1.0, + ], + ], + dst_wh=( + 512, + 512, + ), + type='CamIntrisicStandardization'), + dict( + data_aug_conf=dict( + H=512, + W=512, + bot_pct_lim=( + 0.0, + 0.05, + ), + final_dim=( + 512, + 512, + ), + rand_flip=False, + resize_lim=( + 1.0, + 1.0, + ), + rot_lim=( + 0, + 0, + )), + type='ResizeCropFlipImage'), + ], + type='MultiViewPipeline'), + dict( + keys=[ + 'img', + 'depth_img', + 'gt_bboxes_3d', + 'gt_labels_3d', + ], + type='Pack3DDetInputs'), + dict( + max_depth=10, + min_depth=0.25, + num_depth=64, + origin_stride=4, + type='DepthProbLabelGenerator'), +] +trainval = False +val_cfg = dict(type='ValLoop') +val_dataloader = dict( + batch_size=1, + dataset=dict( + ann_file='embodiedscan/embodiedscan_infos_val.pkl', + box_type_3d='Euler-Depth', + data_root='data', + filter_empty_gt=True, + metainfo=dict( + box_type_3d='euler-depth', + classes=( + 'adhesive tape', + 'air conditioner', + 'alarm', + 'album', + 'arch', + 'backpack', + 'bag', + 'balcony', + 'ball', + 'banister', + 'bar', + 'barricade', + 'baseboard', + 'basin', + 'basket', + 'bathtub', + 'beam', + 'beanbag', + 'bed', + 'bench', + 'bicycle', + 'bidet', + 'bin', + 'blackboard', + 'blanket', + 'blinds', + 'board', + 'body loofah', + 'book', + 'boots', + 'bottle', + 'bowl', + 'box', + 'bread', + 'broom', + 'brush', + 'bucket', + 'cabinet', + 'calendar', + 'camera', + 'can', + 'candle', + 'candlestick', + 'cap', + 'car', + 'carpet', + 'cart', + 'case', + 'chair', + 'chandelier', + 'cleanser', + 'clock', + 'clothes', + 'clothes dryer', + 'coat hanger', + 'coffee maker', + 'coil', + 'column', + 'commode', + 'computer', + 'conducting wire', + 'container', + 'control', + 'copier', + 'cosmetics', + 'couch', + 'counter', + 'countertop', + 'crate', + 'crib', + 'cube', + 'cup', + 'curtain', + 'cushion', + 'decoration', + 'desk', + 'detergent', + 'device', + 'dish rack', + 'dishwasher', + 'dispenser', + 'divider', + 'door', + 'door knob', + 'doorframe', + 'doorway', + 'drawer', + 'dress', + 'dresser', + 'drum', + 'duct', + 'dumbbell', + 'dustpan', + 'dvd', + 'eraser', + 'excercise equipment', + 'fan', + 'faucet', + 'fence', + 'file', + 'fire extinguisher', + 'fireplace', + 'flowerpot', + 'flush', + 'folder', + 'food', + 'footstool', + 'frame', + 'fruit', + 'furniture', + 'garage door', + 'garbage', + 'glass', + 'globe', + 'glove', + 'grab bar', + 'grass', + 'guitar', + 'hair dryer', + 'hamper', + 'handle', + 'hanger', + 'hat', + 'headboard', + 'headphones', + 'heater', + 'helmets', + 'holder', + 'hook', + 'humidifier', + 'ironware', + 'jacket', + 'jalousie', + 'jar', + 'kettle', + 'keyboard', + 'kitchen island', + 'kitchenware', + 'knife', + 'label', + 'ladder', + 'lamp', + 'laptop', + 'ledge', + 'letter', + 'light', + 'luggage', + 'machine', + 'magazine', + 'mailbox', + 'map', + 'mask', + 'mat', + 'mattress', + 'menu', + 'microwave', + 'mirror', + 'molding', + 'monitor', + 'mop', + 'mouse', + 'napkins', + 'notebook', + 'ottoman', + 'oven', + 'pack', + 'package', + 'pad', + 'pan', + 'panel', + 'paper', + 'paper cutter', + 'partition', + 'pedestal', + 'pen', + 'person', + 'piano', + 'picture', + 'pillar', + 'pillow', + 'pipe', + 'pitcher', + 'plant', + 'plate', + 'player', + 'plug', + 'plunger', + 'pool', + 'pool table', + 'poster', + 'pot', + 'price tag', + 'printer', + 'projector', + 'purse', + 'rack', + 'radiator', + 'radio', + 'rail', + 'range hood', + 'refrigerator', + 'remote control', + 'ridge', + 'rod', + 'roll', + 'roof', + 'rope', + 'sack', + 'salt', + 'scale', + 'scissors', + 'screen', + 'seasoning', + 'shampoo', + 'sheet', + 'shelf', + 'shirt', + 'shoe', + 'shovel', + 'shower', + 'sign', + 'sink', + 'soap', + 'soap dish', + 'soap dispenser', + 'socket', + 'speaker', + 'sponge', + 'spoon', + 'stairs', + 'stall', + 'stand', + 'stapler', + 'statue', + 'steps', + 'stick', + 'stool', + 'stopcock', + 'stove', + 'structure', + 'sunglasses', + 'support', + 'switch', + 'table', + 'tablet', + 'teapot', + 'telephone', + 'thermostat', + 'tissue', + 'tissue box', + 'toaster', + 'toilet', + 'toilet paper', + 'toiletry', + 'tool', + 'toothbrush', + 'toothpaste', + 'towel', + 'toy', + 'tray', + 'treadmill', + 'trophy', + 'tube', + 'tv', + 'umbrella', + 'urn', + 'utensil', + 'vacuum cleaner', + 'vanity', + 'vase', + 'vent', + 'ventilation', + 'wardrobe', + 'washbasin', + 'washing machine', + 'water cooler', + 'water heater', + 'window', + 'window frame', + 'windowsill', + 'wine', + 'wire', + 'wood', + 'wrap', + ), + classes_split=( + [ + 48, + 177, + 82, + 179, + 37, + 243, + 28, + 277, + 32, + 84, + 215, + 145, + 182, + 170, + 22, + 72, + 30, + 141, + 65, + 257, + 221, + 225, + 52, + 75, + 231, + 158, + 236, + 156, + 47, + 74, + 6, + 18, + 71, + 242, + 217, + 251, + 66, + 263, + 5, + 45, + 14, + 73, + 278, + 198, + 24, + 23, + 196, + 252, + 19, + 135, + 26, + 229, + 183, + 200, + 107, + 272, + 246, + 269, + 125, + 59, + 279, + 15, + 163, + 258, + 57, + 195, + 51, + 88, + 97, + 58, + 102, + 36, + 137, + 31, + 80, + 160, + 155, + 61, + 238, + 96, + 190, + 25, + 219, + 152, + 142, + 201, + 274, + 249, + 178, + 192, + ], + [ + 189, + 164, + 101, + 205, + 273, + 233, + 131, + 180, + 86, + 220, + 67, + 268, + 224, + 270, + 53, + 203, + 237, + 226, + 10, + 133, + 248, + 41, + 55, + 16, + 199, + 134, + 99, + 185, + 2, + 20, + 234, + 194, + 253, + 35, + 174, + 8, + 223, + 13, + 91, + 262, + 230, + 121, + 49, + 63, + 119, + 162, + 79, + 168, + 245, + 267, + 122, + 104, + 100, + 1, + 176, + 280, + 140, + 209, + 259, + 143, + 165, + 147, + 117, + 85, + 105, + 95, + 109, + 207, + 68, + 175, + 106, + 60, + 4, + 46, + 171, + 204, + 111, + 211, + 108, + 120, + 157, + 222, + 17, + 264, + 151, + 98, + 38, + 261, + 123, + 78, + 118, + 127, + 240, + 124, + ], + [ + 76, + 149, + 173, + 250, + 275, + 255, + 34, + 77, + 266, + 283, + 112, + 115, + 186, + 136, + 256, + 40, + 254, + 172, + 9, + 212, + 213, + 181, + 154, + 94, + 191, + 193, + 3, + 130, + 146, + 70, + 128, + 167, + 126, + 81, + 7, + 11, + 148, + 228, + 239, + 247, + 21, + 42, + 89, + 153, + 161, + 244, + 110, + 0, + 29, + 114, + 132, + 159, + 218, + 232, + 260, + 56, + 92, + 116, + 282, + 33, + 113, + 138, + 12, + 188, + 44, + 150, + 197, + 271, + 169, + 206, + 90, + 235, + 103, + 281, + 184, + 208, + 216, + 202, + 214, + 241, + 129, + 210, + 276, + 64, + 27, + 87, + 139, + 227, + 187, + 62, + 43, + 50, + 69, + 93, + 144, + 166, + 265, + 54, + 83, + 39, + ], + )), + mode='grounding', + pipeline=[ + dict(type='LoadAnnotations3D'), + dict( + max_n_images=50, + n_images=1, + ordered=True, + rotate_3rscan=True, + transforms=[ + dict(backend_args=None, type='LoadImageFromFile'), + dict(backend_args=None, type='LoadDepthFromFile'), + dict( + dst_intrinsic=[ + [ + 432.57943431339237, + 0.0, + 256, + ], + [ + 0.0, + 539.8570854208559, + 256, + ], + [ + 0.0, + 0.0, + 1.0, + ], + ], + dst_wh=( + 512, + 512, + ), + type='CamIntrisicStandardization'), + ], + type='MultiViewPipeline'), + dict( + keys=[ + 'img', + 'depth_img', + 'gt_bboxes_3d', + 'gt_labels_3d', + ], + type='Pack3DDetInputs'), + ], + test_mode=True, + type='EmbodiedScanDetGroundingDataset', + vg_file='embodiedscan/embodiedscan_val_mini_vg.json'), + drop_last=False, + num_workers=4, + persistent_workers=False, + sampler=dict(shuffle=False, type='DefaultSampler')) +val_evaluator = dict(type='GroundingMetric') +vis_backends = [ + dict(save_dir='/job_tboard', type='TensorboardVisBackend'), +] +visualizer = dict( + name='visualizer', + type='Visualizer', + vis_backends=[ + dict(save_dir='/job_tboard', type='TensorboardVisBackend'), + ]) +work_dir = '/job_data/work_dirs' +z_range = [ + -0.2, + 3, +] + +02/11 14:47:52 - mmengine - WARNING - Failed to search registry with scope "bip3d" in the "visualizer" registry tree. As a workaround, the current "visualizer" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "bip3d" is a correct scope, or whether the registry is initialized. +02/11 14:47:52 - mmengine - WARNING - Failed to search registry with scope "bip3d" in the "vis_backend" registry tree. As a workaround, the current "vis_backend" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "bip3d" is a correct scope, or whether the registry is initialized. +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() + return self.fget.__get__(instance, owner)() +02/11 14:48:06 - mmengine - WARNING - Failed to search registry with scope "bip3d" in the "hook" registry tree. As a workaround, the current "hook" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "bip3d" is a correct scope, or whether the registry is initialized. +02/11 14:48:06 - mmengine - INFO - Hooks will be executed in the following order: +before_run: +(VERY_HIGH ) RuntimeInfoHook +(BELOW_NORMAL) LoggerHook + -------------------- +before_train: +(VERY_HIGH ) RuntimeInfoHook +(NORMAL ) IterTimerHook +(VERY_LOW ) CheckpointHook + -------------------- +before_train_epoch: +(VERY_HIGH ) RuntimeInfoHook +(NORMAL ) IterTimerHook +(NORMAL ) DistSamplerSeedHook +(NORMAL ) EmptyCacheHook + -------------------- +before_train_iter: +(VERY_HIGH ) RuntimeInfoHook +(NORMAL ) IterTimerHook + -------------------- +after_train_iter: +(VERY_HIGH ) RuntimeInfoHook +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook +(BELOW_NORMAL) LoggerHook +(LOW ) ParamSchedulerHook +(VERY_LOW ) CheckpointHook + -------------------- +after_train_epoch: +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook +(LOW ) ParamSchedulerHook +(VERY_LOW ) CheckpointHook + -------------------- +before_val: +(VERY_HIGH ) RuntimeInfoHook + -------------------- +before_val_epoch: +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook + -------------------- +before_val_iter: +(NORMAL ) IterTimerHook + -------------------- +after_val_iter: +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook +(BELOW_NORMAL) LoggerHook + -------------------- +after_val_epoch: +(VERY_HIGH ) RuntimeInfoHook +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook +(BELOW_NORMAL) LoggerHook +(LOW ) ParamSchedulerHook +(VERY_LOW ) CheckpointHook + -------------------- +after_val: +(VERY_HIGH ) RuntimeInfoHook + -------------------- +after_train: +(VERY_HIGH ) RuntimeInfoHook +(VERY_LOW ) CheckpointHook + -------------------- +before_test: +(VERY_HIGH ) RuntimeInfoHook + -------------------- +before_test_epoch: +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook + -------------------- +before_test_iter: +(NORMAL ) IterTimerHook + -------------------- +after_test_iter: +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook +(BELOW_NORMAL) LoggerHook + -------------------- +after_test_epoch: +(VERY_HIGH ) RuntimeInfoHook +(NORMAL ) IterTimerHook +(NORMAL ) EmptyCacheHook +(BELOW_NORMAL) LoggerHook + -------------------- +after_test: +(VERY_HIGH ) RuntimeInfoHook + -------------------- +after_run: +(BELOW_NORMAL) LoggerHook + -------------------- +02/11 14:48:07 - mmengine - WARNING - Failed to search registry with scope "bip3d" in the "loop" registry tree. As a workaround, the current "loop" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "bip3d" is a correct scope, or whether the registry is initialized. +02/11 14:48:07 - mmengine - WARNING - euler-depth is not a meta file, simply parsed as meta information + Loading Train dataset: 0%| | 0/3113 [00:00