If you're using the mouse, not having focus follows mouse means you have to click twice to activate something in a window without focus, which from my point of view as someone who uses focus-follows-mouse, is pointless and irksome.

When I'm not using the mouse, I change the focus with the keyboard (In my setup, focus follows mouse only when the mouse moves).