In this case it is a shorthand for specialist level knowledge. Thre kind of information you would expect a person pursuing a PhD in a specific field to know but someone not doing that to not know.
For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.”
That’s the kind of “PhD knowledge” which this evaluation does not require.
The question i quoted about the humming bird anatomy? That is the point. That is why this evaluation explicitly decides to not go down that route.
Nobody (well, not me anyway) wants to convince you that it is usefull. That is the kind of question the authors of this evaluation looked at, they also felt what you are feeling, and decided to do something which doesn’t require that kind of deep and specialist knowledge. And that is what they describe in the paper’s title as “PhD knowledge not required”.
For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.”
That’s the kind of “PhD knowledge” which this evaluation does not require.