I don't think you are misremembering; -ffast-math causes __builtin_isnan and __builtin_isinf to unconditionally return zero, for both gcc and clang:

https://godbolt.org/z/1qjxvW3KK

I found this out the hard way several years ago and still have my own implementations of these functions laying about in case I want to use -ffast-math.