AFAIK CuDNN always had optimizations for NCHW and that was one of Tensorflow speed issue when they choose to default on NHWC, plus the related issues on writing transformation pipelines.
The optimal memory layout usually depends upon which dimensions are the reduction dimensions (for 2d convnet convolution, the spatial dimension on which there is also reduction is likely less important than the batch or channel dimension). Thus, the optimal memory layout for forward and backward passes usually differs a lot, but transposing between the different layouts on the fly has high cost.
Other alternatives beyond just permuting the dimensions include strip mining/tiling and raising/sinking dimensions, techniques which come from loop nest analysis (and correspond exactly to what one would do with the loops of the code): e.g., translating NCHW -> N(C/4)(4)HW -> N(C/4)HW(4) for vectorization purposes, where we turn a 4 dimensional array into a 5 dimensional array, with the innermost dimension being a set of 4 contiguous channels which is sunk into the loop nest, and is amenable to vectorization.
Since many of these kernels are hand-tuned or generated by library vendors, there is likely not much of a choice available, but there are likely many other more optimal memory layouts out there that would require machine learning-driven compilation or mathematical optimization techniques like polyhedral compilation to explore and discover.
The most promising here being the Halide and Tiramisu compiler. Halide uses machine learning to discover scheduling and Tiramisu uses a polyhedral approach.
Fun fact: Alex Krizhevsky's cuda-convnet was also an early adopter of CHWN tensor layout. Basically, having the batch size N as the major dimension limits you to running batch sizes that are multiples of the warp size (typically 32), but then you also have an easier time of implementing fast kernels for all your neural and tensor ops, including tensor convolutions, without getting nearly as stuck in the weeds of microarchitectural optimizations.
AFAIK CuDNN always had optimizations for NCHW and that was one of Tensorflow speed issue when they choose to default on NHWC, plus the related issues on writing transformation pipelines.
So what does NHWC enables that is new?
Relevant in-depth discussion including CuDNN team lead, Julien Demouth, and Scott Gray who implemented Winograd convolution for Nervana Neon (which interestingly was CHWN so batch last): https://github.com/soumith/convnet-benchmarks/issues/93#issu...