Let me tell you a story. A company has a list of coding challenges that are used for interviewing candidates. They use them until someone discovers the challenge got leaked on Glassdoor or Leetcode.com . After that happens it needs to be removed and replaced by a different question. Someone sits down and prepares the next challenge. When ready, people start to use it. After a while the new challenge leaked and the whole process starts again.
Sounds familiar? I think every single company I ever worked for had the same problem.
It’s a game you cannot win. Your questions will leak eventually. So what can you do about it? There is a way you can improve the longevity of your coding challenges while making them more relevant. How? By changing the game. Let me tell you how.
Why do companies not want candidates to know what are the questions they ask in advance? The usual answer is that the companies want to see if candidates can solve problems that are deemed to be on the level of engineers who work at the company. All that on the fly within a limited timeframe. Solving the problem will allow the candidate to proceed to another round. That’s why leaked questions are problematic. If a candidate knows them in advance, the interview does not check anything and candidates move automatically to the next round. So how can we fix the situation if we cannot control leakage of the questions? (sorry but even NDAs are semi-effective)
We can shift our focus from binary “solved \ unsolved” challenges to more open-ended tasks that are invitations for discussion where the real questions are asked. If you are doing a lot of integrations with 3rd party services make it a task for a candidate. Check how they approach it. Talk with them if they ever worked on something like this. Check if they would handle the usual pitfalls. Check how they would test it \ handle errors \ model to make it extensible \ rollout etc. Use the challenge as an introduction to discussion and a task where candidates can show off how good of an engineer they are. There are multiple benefits to such an approach
- You check knowledge that is relevant to the position for which you are hiring
- You check how candidate cooperates and discuss different approaches to solution
- You can see candidate code a solution in a problem space close to real domain
At the end of the day, as an interviewer, we want to know if the candidate will be able to be effective in a role we are hiring for. With such an approach you would not only improve the quality of your interview but also extend the life of your interview questions. Even if someone will post them on Glassdoor it does not matter. For each such an open ended question the space for discussion is huge and for many of those questions there is more than one good answer.
Use the interview question as an invitation for discussion and don’t treat the candidate’s code as the only artifact of the interview. That’s the secret.