You didn't actually respond to his comment. Assuming, for example, you have:

* 99% of applicants are bad * 50% false negative (You look over about one good developer for every good developer you hire) * 1% false positive (one out of a hundred bad devs can snooker you into hiring)

In that scenario, you're twice as likely to hire a bad dev as a good one. And if you halve your false positive rate by increasing your false negative rate by 50%, you're still twice as likely to hire a bad dev, it will just take you twice as much work.

I see what you're saying. I was pointing out the dangers of false negatives, but he responded that he hires 10% of the people he invites for an onsite interview.

Even if you hire 10% of the candidates you on-site interview, that says nothing about your actual false positive or false negative rate. For all I know, he could be weeding out all the top candidates at the pre-interivew stage, and then hiring the best of a mediocre group of people.

It's easy to measure your false positive rate, people you are forced to fire (or wish you could fire if not for corporate bureaucracy).

It's harder to measure your false negative rate. The only way you could measure your false negative rate is to pick a random sample of people who fail your interview, AND HIRE THEM ANYWAY. (However, that could be a lawsuit risk. It would be unfair to the people who hire despite failing the interview. A small business couldn't afford to do it, only some huge corporation could do the experiment.)

Also, I doubt the ability of most businesses to identify the best performers AFTER THEY ARE HIRED and working there for a couple of years.

No, I did not say I hire 10% of candidates I interview :). What I said was that's a fair estimate in industry.

I feel you are truly confused about FP and FN. Whether there are 99% bad developers out there or if you hire 10% of candidates you interview - these both quantities are independent of FN and FP. FN says that you are turning away X good people and it's again independent of FP which ultimately decides how many bad developers you would eventually end up hiring regardless of other 3 quantities I mentioned. See here: http://en.wikipedia.org/wiki/Confusion_matrix

It's not easy to measure FN, FP, TN or TP. Even good people fail due to different reasons like bad manager and bad people may succeed despite of mediocre skills. Looking at who you had to fire or who got promoted doesn't give accurate measurements at all although they may serve as weak proxy. The scenario I described was hypothetical to point out that cost of FP is far more higher than additional cost in hiring due to FN.

If you increase FN and FP stays the same, you are more likely to make a bad hire. Do you understand this?

May be I'm completely missing something here but my understanding is this: FP = (good hires you made) / (all hires you made). Your likelyhood of making bad hire is 1-FP. If you hired 100 people and your FP was 10% then on average you would have 10 bad hires on your team. So FP determines the number of bad hires you would eventually have. FN has nothing to do with it - it only determines how long before you make a good hire, it doesn't influence actual number of bad hires you will make.

I'm using standard terminologies here. There are plenty of textbooks and articles on confusion matrix, precision, recall, RoC etc. Not sure what definitions you are using to arrive at conclusion that FN increases the number of good hires (it only increases effort).

You aren't using statisics correctly.

FP = probability, given a bad candidate, you will hire him

FN = probability, given a good candidate, you will pass

Suppose 100 bad candidates, 10 good candidates

FP=10%, FN=10%

You make 10 bad hires and 9 good hires

FP=10%, FN=20%

You make 10 bad hires and 8 good hires.

So increasing FN lowers your yield.

Your statstic (good hires / total hires) tells you nothing about your actual FP or FN value.

If you don't get it, I'm not wasting time on you anymore. You are very dangerous. You think you know statistics, but you don't.

Your statistic (good hires / total hires), using the jargon from your link, is precision or positive predictive value (PPV).

Using the math from that link, if you decrease FN, then PPV increases.

Sorry, I did mixed up precision in my reply. I just got time to think about this whole debate more carefully and I realize you are actually right if we fix up some of the terminology you have used. The mis-statements and confusion on my part has occurred due to this terminology differences.

First FP and FN are not probabilities. They are just unbounded numbers. This may feel pedantic but in a moment I'll show you why this is critical. Let me draw the confusion matrix first (G = Good candidates, H = Hired candidate etc):

\ H NH \--------- G | TP FN B | FP TN

What you are referring to as probabilities is actually False Positive Rate or FPR and TNR respectively which is defined as follows:

  FPR = FP / (FP + TN) = FP/B
  FNR = FN / (TP + FN) = FN/G
 

Now the quantity you are after is probability that given you did hiring and ended up with good guy which is, nothing but precision:

precision = P(G|H) = TP/H

So how do we get TP to calculate precision if we only knew FPR, FNR, G and B? I did little equation gymnastics using above and got below:

TP = G - GFNR H = TP + FP = TP + FPRB

So now you can plug this in to above equation for precision and find that as you increase FNR, precision goes down while you keep FPR constant. So you are actually correct. Although it might look like unnecessary exercise vs following intuition I think above equation can actually help calculate exact drop in precision and multiply that with cost of FP vs FN to get the operating sweet spot. On my part I need to do some soul searching to figure out why this didn't triggered to me before :).

The following comment by @fsk which I think what you are describing is inaccurate:

if you decrease your false negative rate by more than you decrease your false positive rate, you're actually hiring MORE bad candidates

First false negatives (FN) and false positives (FP) are independent of each other. FP estimates how many bad developers you would end up having regardless of your FN. The FN determines how many good developers you would turn away regardless of your FP. If you are confused about this, well, these numbers are part of appropriately called "Confusion Matrix". I would highly recommand reading up on Wikipedia (http://en.wikipedia.org/wiki/Confusion_matrix) or any textbooks before you jump on commenting and through bayesian equation around because you are certainly not using right terminology. Also both of these are again independent of actual % of bad developers out there (i.e. whether market has 99% bad or 1% doesn't matter, FP solely determines what many bad developers you would end up with).

Next, it might be actually easier for you to think in terms of precision and recall instead of FP/FN. Interviewing process is nothing but classification problem and P/R is standard way to measure its performance. Again Wikipedia is your friend to brush up on that.

A classic situation in classifier performance is referred to as precision recall tradeoff. You can plot that on curve called RoC and choose your operating point. The way you typically do that is by quantifying how much you would get hurt due to loss in precision (~ more FP) compared to increase in recall (~ less FN). You plug the costs in equation and decide your operating point. For companies that can rapidly deal with FP, increasing recall may make sense and other way around. However in most cases there are too many other reasons that I'd listed should typically prevent you from lowering your precision too much.

Hiring rate of 10%? You mean you hire 10% of all people who submit a resume? Or 10% of all people who come in for an on-site interview? Those are two different things.

Why should it take 2 years to fire someone? That sounds like a corporate bureaucracy problem.

10% of total number of full interviews that needed to be conducted. At most publicly listed companies, firing can't be done without accumulation of enough evidence (what HR usually refers to as "paper trail"). This is true even when employment was "at will" mainly because of legal liability (for example, fired employee can claim that he was a victim of XYZ) and bad PR it can generate. I think Facebook is (or was) probably rare exception in aggressive firing and not sure how they managed it. Most companies also require annual review to be in place before firing occurs. Typically hiring manager would avoid firing within first year because that usually looks very bad on them. Most true negatives don't get fired until hiring manager changes or years pass by. At startups things are obviously different. Resources are scarce and true negative probably won't survive beyond 6 month or in worse case beyond a year. But still that's a significant period to cause enough of hemorrhaging for a true negative.