You actually can add people; open up the member list (the stylized "bust" icon at the top left or the information icon to the right) and there's a button there: "Add people".
Yeah, leaving is my main issue. When I get pulled into something and I decide it's not relevant to me, I can close the conversation, but it'll reopen as soon as anyone writes anything, so muting and having it pollute my conversation list is the only option.
Can added people read previous messages, or are they only starting at the moment they enter?
All that does with a private DM group of < 7 people is create a new DM group, and the conversation is not migrated. So you're effectively starting from scratch, and the new addition has no ability to go back and catch up.
I can understand the reasoning for this, but its also an area where private channels are much better at.
I don't think you can leave though.