I'm not on MacOS but find Inkscape pretty snappy in general, even with big projects.
Perhaps it's not the Layers but the effects? Blur in particular is pretty slow. Have you tried reducing the displayed filter quality? This doesn't effect the image itself, only the display in Inkscape. It's in the render tab in Inkscape preferences.
At View -> Display Mode you can also toggle between full display, display without filters and only outlines. If Inkscape is significantly faster without displayed filters, setting down the display quality will definitely help.
The very first item on Affinity Designer's feature list [1] is "Pan and zoom at 60fps". You don't generally accomplish this by accident. They apparently wrote the engine from scratch to use hardware acceleration, and be usable on an iPad with a very small amount of available RAM [2].
There's no reason blur, in particular, must necessarily be slow. Everybody (and their phone) has a GPU that's more than fast enough to do that in real-time. I wonder if the (cross-platform) interfaces that Inkscape are using on macOS aren't hardware accelerated. Or if it's using some syscall which is cheap on some platforms and expensive on others.
I haven't done any testing but I agree that Inkscape feels much slower than Affinity Designer.
I use Affinity Designer and Photo and they are very fast even at huge sizes (I do art). But they don't listen to user feedback or suggestions much if at all. AD/P can use OpenGL Vulcan or Metal (I use Metal) and it's amazingly fast for most operations. I also use Artrage 5 and it's abysmally slow probably not optimized for any platform either, probably not much different from Inkscape (though mostly bitmap drawing).
I'm using plain symbols and pen drawing when testing and CPU shoots up to 70% instantly, moving objects are sluggish and every move makes the CPU spike again.
The more curves I have, the more sluggish it becomes.
I love Affinity Designer! I used to use Inkscape, but it has a horrible user interface (typical FOSS software). Paying $50 to buy Affinity Designer with a decent UI is ok for me.
While on Affinity, I can have hundred layers without noticeable performance issues.