Hiring data scientists is difficult. But no more than defining data science.
We are expanding my team and that has given me a chance to revisit my own experience applying and interviewing for data science jobs and also to read posts from other people on both sides of the market. This is not the first time I am involved in a recruitment process, but in the past they were on the data analytics side, whereas now we want to fill a position that will focus on building data pipelines with predictive models. What makes this position different from other data science jobs is that we are a research company with a very heavy focus on statistics and that makes us look for a very especial set of technical skills. However, contrary to what seems to be the norm, we do not hand take-home tests to the candidates. We never do that.
I remember this one interview with a company that took pride in sending a 4-hour long test even before letting you talk to anyone. The other interview in the process? Yes, an onsite day-long test with a mock presentation at the end. I was a bit surprised that the only chance I had to learn something about the position, the culture of the company or the team was during the 30 minute break for lunch. Not even a one-on-one meeting with a manager or some potential colleagues.
Of course that was an extreme case. Not even teams use a take-home and, when they do, it is usually after you have a chance to evaluate whether the company is a good fit. The divide is not between new teams and old teams, or between startups and consolidated companies. Some companies, some companies don’t. However, that experience pushed me to think more carefully about what you can and cannot learn from a take-home and, more importantly, at what cost.
I am sure companies that use take-home tests are very successful in hiring. But I have always found them to be a poor and very burdensome way to elicit technical prowess. In my experience, there were only two cases in which the tests were more than a review of elementary concepts that could have been covered in an informal 20 minute talk. And yet, most companies see no problem in asking applicants to spend 8 hours working on a non-transferable task. Of course, you want that job. Why wouldn’t you make that kind of investment? So you take it, and you take it again for the next company, and for another one. And you soon end up spending a few days working full-time on tests that try to capture the exact same thing.
There is no question that a good test can be valuable. A good test can tell you something about programming skills and coding style, and also about how comfortable the candidate is when dealing with the kind of data problems that will appear in the job. In a new field like data science which requires a non-standard set of skills and when it is not unusual to see lots of candidates from wildly different backgrounds maybe that’s the only way to pick the best person for the job. Good tests are uncommon, though. Most of them feel like the exam of an unexperienced professor who is still struggling to gauge what should be required from the class.
Worst still, some companies don’t provide feedback on them and that can be very frustrating, especially after spending so much time working on something. I can only remember one process that stopped after the take-home and still today I don’t know what I did wrong. I learned nothing after having spent a full day working on something. What is it that I did wrong? What should I improve? Was my test wrong or was there another candidate who performed better? But I would also have liked to know more about those tests that lead to an onsite interview. Would they have done things differently? Was there a better way to handle the problem?
My argument is not that take-homes are useless but rather that they are not always the best tool, and that it is too easy to use them incorrectly. Next time you are designing a recruitment process, think if you can learn what you need using some other way. Also, think if you can commit to the design of the test the same time you are demanding from the candidate. Otherwise, you may be settling for a suboptimal method to select your staff.