Has this system been tested on inputs that have unequal width and height? If I use the provided DataLayer with both mirroring and cropping disabled on portrait inputs, the first convolution layer quickly tends towards (usually within the first 500 iterations) a set of filters with horizontal bands. Sometimes the loss goes to nan (and many of the parameters in the first convolution layer have gone to nan).

I wrote a separate data layer that replaces cropsize with crop_height and crop_width so that I could manually specify a rectangular crop. I've tried setting crop_height == crop_width and so far the issue hasn't shown up, but I haven't spent much time training (but still much longer than the rectangular cases). If I set the aspect ratio to something other than 1, then the bands do appear and their width seems to depend on the aspect ratio of the crop.

I'm using the imagenet architecture applied to a different dataset. The inputs from leveldb are 264x105x3 and the cropped inputs to the network are 220x88x3.

The issue is reproducible with both solver mode 0 and 1.