SyncBatchNorm number of input channels should be non-zero

Package:

torch

50580

Exception Class:

ValueError

Raise code

nput.dim() < 2:
            raise ValueError(
                "expected at least 2D input (got {}D input)".format(input.dim())
            )

    def _check_non_zero_input_channels(self, input):
        if input.size(1) == 0:
            raise ValueError(
                "SyncBatchNorm number of input channels should be non-zero"
            )

    def forward(self, input: Tensor) -> Tensor:
        # currently only GPU input is supported
        if not input.is_cuda:
            raise ValueError("SyncBatchNorm expected input tensor to be on GPU")

        sel

Links to the raise (1)

https://github.com/pytorch/pytorch/blob/e56d3b023818f54553f2dc5d30b6b7aaf6b6a325/torch/nn/modules/batchnorm.py#L675

Ways to fix

SyncBatchNorm is a type of batch normalization used for multi-GPU training. Standard batch normalization only normalizes the data within each device (GPU).

Error code:

from torch import nn
import torch,os
from torch.distributed import init_process_group

item = torch.randn(0,0).cuda()
#Initialize process group
os.environ['MASTER_ADDR'] = '127.168.1.1'
os.environ['MASTER_PORT'] = '8001'
init_process_group("gloo",world_size=1,rank=2)

m = nn.SyncBatchNorm(100)
m.forward(item)

forward method take tensor as input and checks if it has 0 sizes. Here our item has 0 size. So, error raises.

Fix code:

from torch import nn
import torch,os
from torch.distributed import init_process_group

t = torch.randn(2,2).cuda()  #<---- it has 2,2 size
#Initialize process group
os.environ['MASTER_ADDR'] = '127.168.1.1'
os.environ['MASTER_PORT'] = '8001'
init_process_group("gloo",world_size=1,rank=2)

m = nn.SyncBatchNorm(100)
m.forward(t)

Jul 03, 2021

anonim 13.0k

Add a possible fix

Please authorize to post fix