"1. Tell me about some peer reviewed papers that you published as first author?"
It could be that the author's first criterion is an important predictor. But it seems to me that unless somebody is actually in academia, publishing papers is more akin to a hobby than a professional qualification, especially given the inherent bias against unaffiliated authors.
Edit: On re-reading, the author (hardtke) writes the above when talking specifically about weeding through post-doc applicants. So my quote is out of context and my criticism unfair.
I'd amend this to "tell me about an academic publication that you've been a major contributing author". Depending on the field and circumstances first author doesn't mean the same thing.
I will also say that most people that will excel at data science may not ever be pushing the envelope of statistical methods enough to warrant writing a research paper. Being able to apply and understand the state of the art algorithms is inherently a great skill to have.
The world is a dirty place, and just like there are thousands of applications that just need a developer to implement a CRUD app to expose an API on the web, there are tens or hundreds of domain specific problems to where a 'data scientist' can implement the state of the art data cleaning and machine learning algorithms.
However, being able to understand and apply graduate level textbook statistical methods to a large dataset (bigger than an Excel file) might be boring to some research scientists, it is cool as hell to see what the data is saying.
Yeah, I fail that criterion hard (I've had five rejections, does that count?). Nonetheless, while the author is probably showing his (or her) biases, there's definitely a nugget of truth there. For a more coding focused candidate, an equivalent question would be questions about software you have designed, built and promoted (as that's essentially what the question is asking).
Basically, I want to see if people can finish stuff. For software engineers, tasks tend to be easily defined and of shorter duration. At least at Bright, we have Data Scientists working on multiple simultaneous projects that take a few days to several weeks to complete. Knowing when you are done with a complicated, possibly open-ended, project is very difficult.
For software engineers, tasks tend to be easily defined and of shorter duration.
For crappy ones doing commodity work, this is true. For good software engineers, not so much.
I'm a data scientist by pedigree (before it was called that) who's spent the past few years in "regular old" software engineering (and probably heading back in the DS direction). Trust me that software engineering done right is as subtle and talent-intensive as DS.
The problem is that SWE's are terrible at marketing themselves as a group and generally get too little respect and autonomy to have architectural successes.
It could be that the author's first criterion is an important predictor. But it seems to me that unless somebody is actually in academia, publishing papers is more akin to a hobby than a professional qualification, especially given the inherent bias against unaffiliated authors.
Edit: On re-reading, the author (hardtke) writes the above when talking specifically about weeding through post-doc applicants. So my quote is out of context and my criticism unfair.