You can't fully disable fast-math per-library, moreover a library compiled with fast-math might also introduce inaccuracies in a seemingly unrelated library or application code in the same executable. The reason is that fast-math enables some dynamic initialization of the library that changes the floating point environment in some ways.
Ah, of course, very good point. A header-only library doesn’t have separate compile options. This is a great reason for a float-sensitive library to not be header-only, right?
It's not just about being header-only, lots of libraries which aren't header-only still have code in headers. The library may choose to put certain functions in headers for performance reasons (to let compiler inline them), or, in C++, function templates and class templates generally have to be in headers.
But yeah, it's probably a good idea to not put code which breaks under -ffast-math in headers if possible.
