Would we be able to detect any suspicious activity on our closed source operating system, or would a backdoor implemented specifically for intelligence agencies allow undetectable communication between hosts?
We probably would be able to. But consider that the "backdoor" can be incredibly sneaky. Look up the claimed "Crypto AG" attack, which involved pre-internet crypto devices. The claimed way it worked was to include the key or enough data to allow decryption in the transmitted message in some form.
NSA wouldn't need something to contact their data centres and leak the information, because presumably they can tap some fiber connection or other your data ends up travelling to. All they'd need would be a slow leak of your keys. Even, say, just a few of tweaked data per packet, or tacked on to a file format that is resistant to it, so that they can pick up key information together with the data stream, and they're good.
Or even just intentionally introducing a specific way of generating keys, known to the with sufficient precision to allow them to reduce the key space enough that brute forcing the remaining bits is feasible. For a conceptual example, look up the old Netscape SSL vulnerability (Netscape's SSL used to rely on factors like the host time and pid, that especially on a typical Unix-y system also running services like mail servers that often would include the pid of of the delivery agent in headers etc., was extremely easy to narrow down)
It's difficult to do this in a way that can't/won't sooner or later be discovered, but given the number of likely unintentional mistakes that have been made that still have taken in some cases years to be discovered (publicly at least), it's plausible that there are intentional flaws in at least some software to make it possible for specific parties to be intended to be able to break it.
So you've compiled your entire OS from source with a compiler you verified to be safe? Because if you use a prebuilt OS how do you know it isn't already "root kitted" to not show certain traffic?
That's what's bothering me. How feasible would it be to implement something like that on a massive scale and people not finding out? And how about using your router to deny incoming and outgoing traffic unless it's whitelisted? Is there some way for the router to identify the service that sent out a packet and deny or allow it or at least "scan" the content and make sure nothing malicious is being received or sent?
The OS being inherently untrustworthy forces you to operate under the assumption that it is tracking you and that it does have backdoors. So instead of focusing on weeding out any rootkits or backdoors, you'd be putting the OS under external scrutiny, including filtering all of its network traffic on a separate device.
If, on the other hand, you let yourself get convinced that a stock Debian installation is more likely than not to be "clean", then you just lost your edge and any interested party can waltz in with a truckload of rootkits and you wouldn't even think to care. This whole "open source is more trustworthy" mindset is really quite dangerous.
I don't understand how it's apples an oranges.
So your telling me right now the choice between Windows, OSX and Linux and the choice for security is somehow ambiguous? Both Microsoft and Apple have been directly passing data to the NSA. Where are the data leaks from Linux or other open source software. Open source software is a lot harder to control, not only do you have people with principles in play, your gonna have a tough time gaining control over decentralized systems.
Just remember if someone(like the NSA or any other skilled hacker) wants to get on to your computer they will, with enough time anything can be broken. So whats the solution let's pick Windows and never look back.
(edit) Now, for comparison's sake, you'd have to try and apply that same rigour to MS Windows or MacOS/iOS...