Yes I understand all that, but direct detection is definitely not impossible. The PTOLEMY experiment at Princeton is attempting to do just that, although it's very very hard. Might still be decades away.

http://arxiv.org/abs/1307.4738