Compared to other areas, I'd say "negligible" - but I wasn't involved personally (only joined the team later on), so take it with a grain of salt as it's not impossible that I'm missing something.
The M4 core was already there in the SoC sitting unused, it's not like it was added just for firmware loading ;)
I would have bought a Librem 5 but haven't only beceause of the power issues (which is same reason I didn't get a Pinephone).