if you work a day for me and then I decide not to pay you, you aren't any poorer than you were before so no harm right?
In the absence of a contract (if I just decided to work for a day for you without your prior agreement to pay for the day of work), yes. Doing work does not automatically create an obligation of others to pay you for having done it.
There are two things that can create such an obligation:
* altering something you already had (such as an object or your body)
If I work a day for you, it is a day that I am not able to work for someone else (possibly myself in my own business). So, yes, I have lost a workday. If I am not compensated for that workday, then I am that much poorer.