If ssp is hacking the test suite to bypass a failing test
I'm talking about hacking the application to pass the test suite, not hacking the test suite. You can often "fix" a failing test by doing
if (condition that failed)
whack the application state so that the test suite will pass.
without understanding what the actual bug were. And it's not always obvious to yourself that this is what you are doing.
ssp has the problem, not the test suite
Tuning to benchmarks is not some unique character flaw of mine. When you measure some aspect of people's behavior, they will optimize to that measurement. If the measurement is a boolean PASS/FAIL from the test suite, then they will optimize their behavior to get a PASS.
But the actual desired outcome is not PASS, it is "bug free program".
Hack the app, hack the test suite, I meant either equally. Which I think if you read my post is pretty clear that the main dichotomy is between "hack" and "real fix" and where the hack goes hardly matters.
Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. I say that's nonsense. The solution is to use the benchmarks better. Are we programmers or automatons? (Or managers?) If you're going to be that defeatist about programming you're not going to be able to be a successful programmer under any circumstances, the entire field is a minefield of superficially-appealing optimization opportunities!
I'm talking about hacking the application to pass the test suite, not hacking the test suite. You can often "fix" a failing test by doing
without understanding what the actual bug were. And it's not always obvious to yourself that this is what you are doing.ssp has the problem, not the test suite
Tuning to benchmarks is not some unique character flaw of mine. When you measure some aspect of people's behavior, they will optimize to that measurement. If the measurement is a boolean PASS/FAIL from the test suite, then they will optimize their behavior to get a PASS.
But the actual desired outcome is not PASS, it is "bug free program".