The solution, which they're implementing now (http://www.forbes.com/sites/andygreenberg/2013/06/18/nsa-dir...), is to require two or more administrators present whenever anyone has to build, maintain, or otherwise access potentially sensitive software, computers, information, etc.

They would be required to report on any strange activity they see from the other administrator(s).

Extremely inefficient, but it's far more secure than just allowing any administrator to do what they want with stuff.

This is a great example of Assange's "secrecy tax" degrading the effectiveness of the organization.

So leaks payoff twice -- once in direct transparency, and again by damaging the ability of the organization to act.

http://cryptome.org/0002/ja-conspiracies.pdf

  The more secretive or unjust an organization is, the more leaks induce
  fear and paranoia in its leadership and planning coterie. This must
  result in minimization of efficient internal communications
  mechanisms (an increase in cognitive "secrecy tax") and consequent
  system-wide cognitive decline resulting in decreased ability
  to hold onto power as the environment demands adaption.

It is the good 'ol "No Lone Zone". Nuclear ICBM sites have those. You need our accountabil-a-buddy with you at all times.

Here is an example placard from a Titan II site:

http://www.flickr.com/photos/mattblaze/4182509642/

(By Matt Blaze)

Not a bad concept. But your accountabil-a-buddy has to be at the same level as you technically. One can easily fool or mislead non-technical people. "Q: What are you doing?" "A: Rebooting the flux capacitor"

I've worked (in non-government contexts) under rules that required two people for certain actions.

My experiences with such processes is that unless you can have some sort of technical measure that proves both people are actually paying active attention to what is being done, the second person will often just zone out. Sometimes, if asked even a few hours later, they won't even have a clear recollection of the event taking place, much less what was actually done.

Additionally if you both double up and cut the workforce, chances are the two will be doubling up in another way too, i.e. both working on different tasks and claiming to keep an eye on eachother.

But really, you are going to lay off 90% of the sysadmins and require two different people involved to change a password?

What this shows is if anything how much you need a combination of good monitoring and enough people. And once one account is compromised you have a chance for the sysadmin to be using sock puppets for accountability actions.

Is it? I'd expect a write-only logging system that runs at a low level exporting logging to remote servers that other admins don't have access to. As the article suggests, you layer admins so no single person can truly compromise things.

It should be pretty basic to catch someone that's resetting user's passwords.

Sure, you can remote syslog everything to a remote logging service, but:

1] Do you log every single thing that happens on the machine?

2] If not, did you actually log the proper set of commands that were used to possibly commit a crime?

3] Granted you are not a small startup, do you have a process in place already to data mine, extract and analyze these potentially dangerous commands from log sets that are potentially TB in size?

4] Do you have the personnel in place with the qualifications and time each day/week/month to audit these potentially large results?

5] If you've caught something, aren't you already too late? Do you have some type of magical software in place that will somehow recognize these types of crimes in progress and cut off access in realtime?

Many of these articles totally ignore how these real world systems work (mainly I'm guessing because the authors have never been involved with companies that run 10k-200k servers around the world). It really is more complex than you think. Most companies that aren't startup size are continually playing catch up. Sure they need to put logging in place, auditing, yada yada. That is being balanced with the other day to day pressing tasks also. It always seems to be a game of catch up...

Not intending to be terse, but:

1+2) You could catch a decent portion of this type of behaviour with a tiny fraction of the effort. Logging local activity on individual user desktops is a waste of time, for example. Raising the difficulty of doing things undetected raises the bar significantly, particularly if users don't know exactly what's going to trip an alarm somewhere and get them caught.

3+4) Considering the nature of their work, I'd expect the NSA to be the right people for this with existing processes, suitable personnel and the storage/processing power required. It's their job to intercept and analyse large quantities of network traffic.

5) There was a period of around a year end-to-end in Snowden's case, and was in contact with The Guardian months before leaving the US (http://www.reuters.com/article/2013/08/15/us-usa-security-sn...).

Yes, it's a difficult problem (and impossible to comprehensively solve, doubly so when you don't trust administrator-level users), but when you're dealing with classified information in government, security is more important than in a regular commercial context.

If anything, the lack of decent internal auditing at the NSA is probably intentional.

At last year's AWS Public Sector Summit, I watched a demo of Xceedium's access control appliance [1]. It's deployed as a jump host and acts like a DVR for SSH/RDP. The CAC crowd was very interested.

[1] http://www.xceedium.com/solutions/privileged-identity-manage...

Re #1 – at the rate the NSA is reported to be archiving data, do you _really_ think logging every single keystroke that every employee/contractor with any elevated privileges at all ever types would be even vaguely a problem? TB-sized logfiles and processes to datamine them are surely interview tasks or intern familiarisation exercises at their scale?

Yes. A previous employer of mine was apparently logging a PB of log data a day between all departments, it isn't something that is just tossed up and somehow works in a month.

Do you have infrastructure in place to store all of that? How are you storing all of this content coherently so that you can go back and audit records of systems at a point in time you are concerned about? Do you have an internal Hadoop cluster configured (not trivial) and the hardware to ingest and process these logs? How are you actually logging everything? Do you have dedicated and qualified people to write this software and maintain it? When someone has su -'d to root, how are you logging everything they type coherently? Is your regex smart enough to ingest and contextualize multiline commands as root and add them to your report? What if no actual "bad" commands on your list were typed as root? What if I as a luser copied the "bad" commands to /tmp/kittens.txt, then su -'d to root and ran them there? Is this somehow captured? Are you 100% sure syslog is running 100% of the time on all hosts you are concerned about? Can't I as a sysadmin kill the syslog pid before I commit a crime? Are you using redundant syslog hosts? Are you using TCP and not the default UDP so that syslog doesn't drop packets under load?

Like I said, this isn't as simple as everyone thinks...

Ballpark numbers - an employee typing at 100wpm for 40 hours a week generates not quite 70MB worth of keystrokes in a year - and I'd guess that estimate is probably something like 2 orders of magnitude too high - no-one actually types 100wpm nonstop all day every single day at work. Even ignoring that, I've got enough disk space sitting under my TV to store (uncompressed) every keystroke typed in a year by something like 100,000 such mythical "100wpm typists working at 100% utilisation" employees. 1PB would, to a first approximation store every single keypress it'd be possible for the rumored 4 million "top secret or above clearance" people in the US typing flat out for something like 15 years.

So I don't think your "do you have the infrastructure" argument holds too much water here.

The "do you have internal Hadoop clusters" question falls the same way. Sure, I don't have that connected to my 8TB of external USB drives plugged into my media server - but if I worked at Google or FaceBook or Twitter, I'd fully expect to be able to provision and spin up adequately sized clusters of VMs and storage to effectively consume and run reports on data that size and bigger. And it's surely not just the NSA and Google/Facebook/Twitter routinely dealing with collecting and processing data at that scale - any decent sized telco, any non-trivial web analytics service, any large financial institution, every HFT business, most bio-med businesses, probably every physics and astronomy department at any university – there must be tens of thousands of businesses routinely dealing with that sort of sized data sets.

It's not simple - certainly not simple enough for me to do it on a Mac Mini and a bunch of external hard drives – but I also don't think it's anything like "uncharted waters" territory. (I'm pretty sure I could find the expertise required in my 1st level LinkedIn connections, and have absolutely no doubt I'd be able to manage designing, developing and deploying exactly such a system if someone came to me with a high six or low seven figure budget.)

I think your approximations are focusing solely on keystrokes, whereas the parent specified just "data", which makes me believe that network traffic, application logs, etc are included in this. I can believe the 1PB/day number, it's not that far fetched when you consider the above.

I also agree with the parent. There are so many possible scenarios and things to log that eventually you're playing a "logging" version of whack-a-mole. Even just managing these files (as the parent talks about) is really no trivial task. Honestly, I wouldn't even know how to begin managing a petabyte worth of daily data.

Spying on all of the email isn't simple either. When there's a will and a bottomless font of taxpayer money in the name of national security, there's a way.

First, you're tossing 100% around way too much - security in this environment is a pretty huge sliding scale. But for something as basic as password resets, yes, I'd expect 100% functionality and auditing to catch it.

Second, I don't think anyone thinks it's totally trivial. But, this is the NSA we're discussing. Supposedly, their secrets are so special, the USA will collapse or something if they're compromised. If any organization in the world is set to handle an admin resetting people's passwords, it's the NSA.

Are you really arguing that it's just too hard for the NSA to notice a massive violation of policy?

Sysadmins are supposed to reset passwords...or I'm sure there is some central auth service in place that allows access to hosts one has been determined to have access to.

The people you trust to admin systems are going to easily be able to abuse their power and it is very hard to stop them from doing so, without making their job so cumbersome for it to be near impossible.

I worked at a firm in 1999 as IT manager... the director required our syslog to be piped out to a line printer printing a real-time physical copy of log/messages and logwatch output on our linux machines.

I was required to manually go through each 24-hour output with a highlighter and look for attacks....

This was not a fun part of my job.

I had systems doing this in the late '90s, and I'm 100% certain I was following "current best practice" at the time, rather than having invented anything novel or notable…

(Having sid that, I wonder how many people have patents or patent applications in for that idea right now?)

>If anyone believes that you can totally lock down access to every system on your network from your trusted sysadmins and have 100% audibility and accountability you are unfortunately living in a fantasy land. NSA or not, this really isn't something that is 100% preventable.

A /huge/ part of managing systems is vetting the folks who must be completely trusted. As part of this, you want to reduce the number of people you trust 100%, but yeah, some still exist.

Those people you trust 100%? you must vet them carefully. You should not put some random contractor in that position. Clearly, as Regan would say, "mistakes were made" - and nobody seems to be taking responsibility.

Just keeping the secret data on a secured server that only a few, very highly vetted sysadmins had root access to, that carefully logged all requests for information (and set off a pager somewhere if too many unscheduled requests were made) would have solved the problem, assuming you didn't hire some random body shop to staff /those/ sysadmins.

I mean hell, I get paged sometimes because some fuckwit at one of my upstreams starts bouncing my packets that are traveling from san jose to sacramento through texas. Surely, someone could be woken up if someone starts accessing suspicious amounts of data at once.

This is part of the huge "who you know" factor in the valley; Generally speaking, you hire folks that your current folks know. Not only does this provide some technical validation, it also makes the cost of defection higher. (Of course, there are lots of downsides to that approach, too.)

But no matter how you do your vetting, you /must/ vet folks with root.

And you can (and should) reduce the number of folks with full root. Especially when there is sensitive data on hand. Give your contractors limited tools.

> Just keeping the secret data on a secured server that only a few, very highly vetted sysadmins had root access to, that carefully logged all requests for information (and set off a pager somewhere if too many unscheduled requests were made) would have solved the problem, assuming you didn't hire some random body shop to staff /those/ sysadmins.

The problem with that theory is that you're assuming only some data is secret. Actually, all the data is secret. Even the information about which data is secret, or what are the criteria of secret data, or how secret data should be handled. Everything is secret.

So rather than dealing with a neat pack of documents that you want to keep secret, think of an organisation with 100k+ people where every single bit of data they produce or interact with every single day is top secret.

The idea is similar; First thing is to setup a situation where few people can access the data in an unmonitored way.