I'm not saying it isn't a problem, I just wanted to know if I was correct in thinking that it is only a problem the first time you compile each crate (or rather each version of each crate, I think).
Yes, it is a problem when you do a clean build, or when you have common crates across projects, because cargo doesn't have a concept of build cache.
You can try to workaround it by setting all target directories to same one via target-dir in your .cargo/config file, but there is no guarantee that the crate won't get rebuild.
[0] - I eschew header only libraries, only if there is no technical way around them (e.g. templates).
[1] - I can AOT compile with Excelsior JET, IBM J9, JamaicaVM, PTC, , Oracle Java 9 (Linux x64 for the time being), ...
[2] - I can AOT compile with Mono, NGEN, .NET Native, CoreRT, IL2CPP, ...