I wonder how they store all that. Surely a side benefit of this could be NSA contributions to CS journals about database techniques.
Also I doubt the veracity of the claim that they collect "nearly everything". Wouldn't they show up on, say, Sandvine's Internet traffic reports? I think it's more likely this claim is made simply to generate FUD in the general population.
I think the era of government being far ahead of commercial tech capability is over. The government mostly outsources now (a problem Snowden identified in terms of information control) or develops in-house with vendors.
The government mostly outsources now (a problem Snowden identified in terms of information control) or develops in-house with vendors.
There are a lot of fingers in that pie. Oracle, for example, has a National Security Group, whose job is to come up with "solutions" and then try to sell them to three-letter-agencies.
Another benefit could be realised in the future if historians and linguists manage to get access to all this data. I imagine that researchers of the social sciences would end up enjoying the same sort of large-scale collaborative projects that particle physicists or genomic bioinformaticians currently have with their huge datasets.
Well, now they have a massive data center in Utah. That's most likely where it's all going today. Standard open source tools are all you really need.
> Wouldn't they show up on, say, Sandvine's Internet traffic reports?
No. If some script kiddie/hacker type installs a packet sniffer and logs all your traffic to your ISP, that won't show up anywhere. Traffic goes somewhere. You're sending packets out. Merely logging packets is entirely passive and undetectable.
Also I doubt the veracity of the claim that they collect "nearly everything". Wouldn't they show up on, say, Sandvine's Internet traffic reports? I think it's more likely this claim is made simply to generate FUD in the general population.