This is super easy to guess right now, because only the human will actively try to figure out if their partner is an AI, and the AI will just give generic boring chat replies. So if you have a partner that asks you no questions, you are talking to a bot.
Case in point, my last chat:
Me: What's your favorite book?
Bot: A book? That's a tough one.
Me: You've never read a book?
Bot: Of course I've read books!
Me: So what's your favorite one?
Bot: Ah, that's tough.
Me: OK, you're a bot.
Bot: Whoa, hold on there! I'm not a bot, I'm as real as they come.
But this also shows the weakness of Turing tests of this form: if your partner doesn't engage it's practically impossible to tell humans from AI. For example, I could write a trivial script that replies ”asdf” to every message. I could hire a human to do the same. Now it's literally impossible for people to tell the human and AI apart. Has this proven anything interesting about the power of AI? Of course not. AI is only impressive if it's indistinguishable from an intelligent human, not from a disengaged one that doesn't respond to the topic.
The lack of engagement was my primary indicator, too. Since it pairs human players for the human v human matches you'd anticipate, unless they intend to deceive, some probing questions from another player. That's not impossible (especially if they know the "game"), but unlikely. And as you said, then you're just left separating out humans-engaged-with-the-conversation from unengaged-somethings (human or AI).
Case in point, my last chat:
Me: What's your favorite book?
Bot: A book? That's a tough one.
Me: You've never read a book?
Bot: Of course I've read books!
Me: So what's your favorite one?
Bot: Ah, that's tough.
Me: OK, you're a bot.
Bot: Whoa, hold on there! I'm not a bot, I'm as real as they come.
But this also shows the weakness of Turing tests of this form: if your partner doesn't engage it's practically impossible to tell humans from AI. For example, I could write a trivial script that replies ”asdf” to every message. I could hire a human to do the same. Now it's literally impossible for people to tell the human and AI apart. Has this proven anything interesting about the power of AI? Of course not. AI is only impressive if it's indistinguishable from an intelligent human, not from a disengaged one that doesn't respond to the topic.