Skip to content

CTCLoss produces NaNs in some situations #14335

@WenmuZhou

Description

@WenmuZhou

🐛 Bug

when I train a cnn-rnn-ctc text recognize model, I meet nan loss after some iters, but it's ok at pytorch 0.4 with warpctc

To Reproduce

Steps to reproduce the behavior:

  1. download the code from https://github.com/WenmuZhou/crnn.pytorch
  2. change the ctc loss from warpctc to nn.CTCloss()
  3. run
  • PyTorch Version (e.g., 1.0): 1.0.0.dev20181115
  • OS (e.g., Linux): ubuntu 16.04
  • How you installed PyTorch (conda, pip, source): pip3
  • Build command you used (if compiling from source):
  • Python version: 3.5.2
  • CUDA/cuDNN version: 8.0/6.0
  • GPU models and configuration: 1080ti
  • Any other relevant information:

Metadata

Metadata

Assignees

No one assigned

    Labels

    high prioritymodule: cudaRelated to torch.cuda, and CUDA support in generalmodule: determinismmodule: nnRelated to torch.nntriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions