That's a very good point. IMO the biggest problem with unit tests is that so many people mis-use them, believing that they are a replacement for QA, integration testing, and load testing. You need people to look at your user interfaces to validate their usability, for one thing.
When I was working with Rails, I found the unit test framework to be nearly useless for my application since it relied on a SOAP service (and at least back then, you couldn't hit the service with a unit test), but the Rails integration tests were great.
So I ended up writing a fairly extensive suite of integration tests, but nearly no unit tests at all.
Yup, unit tests are a programmer aid, not a replacement for QA. If the unit tests aren't helping you code faster (by preventing debugging), they're not worth it.
It's probably close to impossible to prevent people from using unit tests as a substitute for QA, though. The concepts are similar enough that people are bound to confuse them, or at least take the confidence they get from passing unit tests as "good enough" to substitute for QA.