I'm basically arguing that any such line is arbitrary.

The distinction between science fiction and fantasy is probably a useful one for helping readers find books they like. Going further than that seems like it is going to almost always involve a lot of personal preference.