Why benchmark using 32-bit FP on a V100? That means it’s not using tensor cores, which is a shame since they were built for this purpose.
There’s no reason not to benchmark using FP16 here.
If you click around enough you’ll see they benchmarked in 32-bit FP. Glad they have a mixed precision training option but I really think it’s a mistake in 2020 to do work related to efficient inference using 32-but FP.
The problem is that your conclusions aren’t independent of this choice. A different network might be far better in terms of accuracy/speed tradeoffs when evaluated at a lower precision. But there is no reason to use 32-but precision for inference, so this is just a big mistake.
