Engineers can be held accountable for a failure in a system they said would work, and it happens when something goes wrong with a bridge or building. I do not agree with the point. It is like saying, "A condominium in Florida cannot be held accountable." :-)
We're talking about systems, algorithms, black box models that we don't understand or have the ability to reason about, that are being used to make decisions for us.
Those systems are already proven to have embedded biases.
But now instead of a human we can question, we only have inscrutable machines that no one, not even the people who built them, understand.
If condominiums in Florida were making the decision about who could get a mortgage and live in them, then your analogy would make sense.
But they don't.
There are, on the other hand, computer systems that absolutely do, and it's very possible (I would predict probable) that they will end up effectively re-creating redlining.
Furthermore, going back to your broken analogy, unlike the case with building codes and engineering professional organizations, there are no laws or regulations that could be used to hold accountable those who build or use those systems.
You can inspect the design of a condo but ML models can't be examined, their training data is the only thing you can examine. It's like being able to inspect the raw materials but nothing else for the condo.
And further to your point, if a DA's office keeps arresting people incorrectly, they won't keep that job.
The fact that we as a society are not okay with just arresting the wrong person is how we started talking about 'stop & frisk' and its shortcomings in the public domain. It's how we had BLM summer and many other protests after the killing/arrest/etc. of innocent people.
and that's how we should respond when the wrong person is arrested
who can I call when "the AI got it wrong"? It's worse than google customer service.
The reality here is none of that will happen in all but the most egregious cases.
If either of those scenarios came to pass the police unions stand ready to defend them. At most the cop will get some paid leave and a settlement payout (as well as the victim at great expense to the town/city) and maybe cop will be transferred to a new department while the whole complaint is put under a NDA.
Really? I am not so sure about that. Who'd get punished for misidentifying a face in book, say, of faces?