Stories about Software


The Best Way to Hire Developers

Editorial Note: I originally wrote this post for the Infragistics blog.  You can find the original here, at their site.  There’s a lot of good stuff over there, so go give it a look.

The other night, I was remembering what might have been my most impressive performance in the interview process.  What makes this performance particularly interesting, however, is not how well I did, but rather how I did well.  And the how left me feeling unsatisfied with myself and with the process.

I was interviewing for a software development position, and this particular organization’s interview process was (1) phone interview, (2) programming exercise, (3) in-person interview. The phone interview went pretty well, and the recruiter had told me that the company was excited about me – a mildly good sign, for whatever it was worth.

However accurate the recruiter’s assessment may or may not have been, the company’s feelings were positive enough to give me the programming exercise.  This all occurred back when I was in grad school, and, at the time of this particular interview, I was in a class called “Advanced Database Design,” in which we explored persistence options beyond the traditional, relational database.  This was a bit of an avant garde class, at the time, because the NoSQL movement had yet to gain a ton of steam.

When they handed me the programming exercise, I had just, in this very class, wrapped up a chapter in which we’d studied using R-Trees to store geographical information.  This unit of study included what they were, how they were used, and a bit of pseudocode to really drive the point home.  As fate would have it, the R-Tree happened to be an extremely elegant solution to the programming exercise for this interview.


The Anatomy of Winning an Interview

This exercise felt like, as they say, shooting fish in a barrel.  I recognized that the thing I had just learned was applicable to my situation, and I, well, applied it.  I ran some automated tests on it for verification purposes and then turned the exercise around pretty quickly.

The company was ecstatic, and I didn’t need to try to read their reaction through the recruiter.  They reached out directly about flying me out for an interview and told me, point blank, that this was by far the best solution they’d seen submitted to the exercise.  I felt as though I were the hero in some really lame Indiana Jones movie, where the ancient gatekeeper had been waiting millennia for someone to come along and solve the programming exercise with an R-Tree.

I wound up passing on the interview, but only because something better came along as I negotiated the details of flying out to meet them.  They were clearly disappointed, and they encouraged me to let them know if I changed my mind.  I liked the company, and this overture made me feel good.

And yet, I came away from the whole experience feeling strangely empty.  As good as it made me feel to have impressed a company so profoundly, there was no denying that this was largely a matter of luck.  Had I done the coding exercise a few weeks earlier, I wouldn’t have known about R-Trees.  Had I done the exercise a few months later, I might have forgotten about R-Trees altogether as an option.  In neither case would I have blown away my interviewers, meaning my ”unprecedented” performance was a happy coincidence rather than a reflection of me being some kind of uber-programmer.

Programmers, Don’t Sweat “Failure”

Have you ever sat for an interview or programming exercise that demanded knowledge you simply didn’t have?  Famously, if you ever have occasion to interview for Google, you’ll discover that their initial phone interview is basically like an oral midterm from your undergrad CS program’s senior-year algorithms class.  O notation. Sorting algorithms. All that fun stuff.  They aren’t alone in that by a long shot, but their interview process is iconic.

To combat being caught off guard, you probably study up before you hit the job market, making sure you can field a wide array of technical questions.  After all, you never know what they’ll throw at you.  But the thing is, you never know what they’ll ask—or what knowledge you or any of the other candidates have recently acquired that will be perfectly timed for this interview.

You may be up against someone like me who just happens to have read a white paper detailing a solution to the exact problem of which the interviewers are so proud.  If that’s what happens, well, you’re just out of luck.

And that’s why you shouldn’t feel bad.  The interview process is actually depressingly random and non-scientific.  “Failing” an interview doesn’t mean that you’ve actually “failed” at anything. It just means that you haven’t had the good fortune to read the material that would have let you pass.

Companies, Focus on the Right Things

Do you have an interview process like this one?  Do you have a process where, all else being equal, you’ll wind up hiring a candidate that just happened to read up on R-Trees?  If you do, I’d suggest some introspection to evaluate whether or not that might be a hiring process smell.

In my experience, interviews aim to understand one or more of the following three things: what does this candidate know, what has this candidate done, and what can this candidate do?  These are ordered both in usefulness (least to most) and difficulty to evaluate (least to most).

What a candidate knows is probably the least useful, since human beings spend their lives learning and what they know can change easily.  But it’s easy to evaluate. Include trivia quizzes and the like as part of your interview process, and you can figure out what they know.

What a candidate has done is a little trickier, but it’s more useful.  I mean, sure, they have resumes and anecdotes, but dissembling is always a possibility.  You can be more sure that they know what an R-Tree is by quizzing them than you can be sure that they’ve implemented R Trees by hearing them make that assertion.  But if they’ve successfully implemented it before, that’s more valuable than them knowing what one is.

But the importance of what they know or what they’ve done pales in comparison to this: “Will this person, confronted with a situation that needs an R-Tree, do the requisite research and translate that into a solution?”  That’s what you really want to know. Those last two things are just proxies for trying to know it.

I certainly don’t have any unprecedented insight into the best way to conduct interviews. Evaluating candidates is hard.  But I can tell you that the closer you get to creating evaluations for what candidates can do, the more likely you are to have successful hires.

  • Minnesota Steve

    I really don’t like the programming questions. Or even questions about things I can look up in a book if I need it. I’ve found the best interviews involved a discussion. Like me drawing out a rough architecture of some major change I worked on, and then talking about the benefits or disadvantages we uncovered.

    I’ve dealt with a lot of people who knew all the right buzzwords, but when you got right down to it you could tell that they had no clue why. That’s something a coding challenge won’t uncover, but a good discussion will.

    • Brilliant! I hope ALL recruiters READ this and LEARN from it. Failing that, at least some of the good ones would be brilliant and then they would become the best!

      • Randy A MacDonald

        Forwarding this article, as I have, to recruiters one has encountered, is a good strategy.

      • doug_b

        Recruiters = mindless bottom feeders.

        • Paul Pettersen

          Good ones do exist. But mostly those who have no skills to do any real job, recruits people who can!

    • Flemming Bonde Kentved

      I partly agree. Ofc the buzzwords should give no real points before they have been put to a test. Let’s say I need a programmer to develop c# business applications. Then a buzzword would be Entity Framework, ofc I then want that person to explain what it’s good for and also ask some questions on how to do different things. They might not need to sit and program, with me ready to judge, but I might show them some code and ask them to explain what it does behind the scenes.

    • Over the last few years, I’ve found myself conducting a lot of interviews and designing the format of the interview process. I always feel like what I’m doing (to paraphrase Churchill) is the worst process I’m aware of, except for all the other processes I’m aware of.

      What I’ve roughly settled on is giving candidates a brief programming exercise, which serves mainly as a discussion starter. After they complete it, the interview portion is a discussion centered around them explaining their rationale for design decisions and approach (surfaces plagiarized solutions and allows them to demonstrate conceptual grasp of principles). Then I typically ask to pair with them and toss them a curveball change in requirements (anything from, “I noticed that your solution doesn’t handle edge cases” to “refactor this entire class to use no casting”). This allows me to see what working with them is like and to see if they get weird in the face of relatively gentle criticism.

      There are no quiz questions about OOP, no trivia questions, and nothing that involves even a hint of some kind of “answer sheet.” Being able to rattle off the three main paradigms or the SOLID principles doesn’t tell me a lick about how someone would be as a teammate.

    • I too hate programming questions with a passion.

  • Robert-T

    Excellent article. I seldom read articles thoroughly and almost always lose concentration, resulting in skimming through and skipping most of it. But your article here is very good. I’m not in a position to hire anyone but if I were I would utilise what you have suggested. I also think throwing a candidate into the deep end when they start work is important. If they can do what they claimed during the hiring process, then that should be apparent from the outset when they start work. If they cannot do the work then they should be dismissed. This is probably a side note but I think recruitment agencies should be consolidated into one government run recruitment organisation which collaborates closely with the Education sector and Unemployment Benefits sector. Each citizen should have a portfolio of skills and experience and educational achievements. I think this is better than recruitment agencies because agencies don’t always have the employer or prospective employee’s interests in mind and just want to make profit by making the hiring process a money making scheme for themselves.

    • Mr. Peter

      Try the Soviet Union. I believe their model would work for you.

      • It’s just a very obvious trolling bored teenager. No use responding.

    • I’m not sure about the consolidation you mention, as it seems to require a significant bit of legal/governmental reform to be enacted. Notwithstanding my personal politics, I tend to favor approaches that can be achieved with action rather than lobbying.

      That said, the first part of your response resonates with me. In my ideal world, I would ‘interview’ candidates by paying them to work on a small, relatively non-critical assignment. The interview process as it exists today has a peak predictive efficiency of something like 30% (per Lazlo Bock’s studies), and I attribute this to it trying to predict the future. Rather than do that, I’d favor an approach that just does things on a trial basis and keeps the cost of being wrong small for everyone. Of course, easier said than done.

  • Randy A MacDonald

    As someone who sees the ‘situation that needs to be done in X manner’ as patently false,I question the value of the most important question as it is stated.

    • Do you mean, “what can the candidate do?” If so, I don’t mean this in the sense of order-following, but in the sense that the person can be trusted to operate autonomously. Frankly, if I’m hiring software developers, I have little use for people that need prescriptive direction, unless the hire is someone that’s pretty early in their journey.

      • Randy A MacDonald

        That begs the question: how far along in their journey should they be? and in what journey?

  • rockaUniverse

    The interview process and how it is conducted should depend on the position being interviewed for. The process will be different for a beginner programmer vs a senior programmer. For a beginner programmer, the interviewer must be realistic and knows that the beginner programmer will not have the experience nor a solid portfolio of projects to boast about. Instead, like Minnesota Steve said, he should be tested in his/her approach to put forward a solution, step by step, to a particular problem and even putting together a design in tackling a problem. Answering programming questions do not prove anything as questions will also be raised when working on a project you will not have an answer to but would have to research it. I think companies themselves don’t even know what they are looking for and are not visionaries in utilizing the resources of people’s background and what they potentially can offer. What do I mean by this? Remember programming is really a tool used to solve a problem and without a solution there is nothing to code. A person with a multifaceted background, say engineering with computing, would have different approaches and higher appreciation for tackling certain problems vs someone with just a pure programming or computer science background. Knowing how to tapped into the diverse background of others would prove a greater long term benefit than someone who is just locked into one area who only codes based on instructions given without any input in planning and design. When it comes on to evaluating candidates, I believe the interviewers are one dimensional and only focus on what the person have done in terms of coding or building applications.

    • Out of curiosity, are you familiar with the shu-ha-ri concept? It seems like there are some parallels to what you’re saying and that approach.

  • doug_b

    Rarely do I read anything to do with programming. I’m the ‘old guy’ on the block. Got my MS in comp sci 1971. At that time it was a career. Now, I’m not even sure it’s a job.

    I would rather my daughter come home with a plumber or carpenter than a programmer. Big business and nano-management have killed the profession (unless you like to kiss a**). It’s not a profession for thinking adults, more like guy’s who never really want to grow up.

    Next time you get your car repaired, dentist, doctor, lawyer, be sure to give them a test.

    • I’m not sure I agree with how dire your assessment of programming is, but I do agree with your main premise. A frequent topic of mine on this blog, actually, is how I think programmers should stop accepting a status quo where they’re subordinate to large management hierarchies and career project managers. The comparison to doctors/dentists/lawyers especially resonates with me, as I think that programming, as a professional, should look similar to those professions. Lawyers don’t bring in “lawyer managers” to give them orders and tell them what to do — they hand planning and logistical work out to subordinates and vendors. I think that’s what programming should look like — autonomous, partnered consultancies that delegate project planning and other logistical concerns.

      • Good point, Erik, but I would throw in that a lot of these professionals can get stale, especially hidebound in the case of Doctors. Also, lawyers (as I’ve gleaned from binge watching 4 seasons of the Good Wife) can be terribly prone to mismanaging their careers.

        • pmarfleet

          And politicians (based on all series of The Good Wife)

    • rockaUniverse

      I can appreciate what doug_b is saying and what the current profession of programming has become. In today’s society, the profession has virtually eliminated the purpose for having formal training as a programmer and would rather create a jungle where anything goes. A computer science degree bachelor or master is almost completely ignored when looking for a job. It shouldn’t be so because this is what should be preparing you to follow standards, build professionalism, be innovative, be creative and be apart of solving problems at the highest level. It’s been documented that only 30% of software are delivered on time and usually will be full of bugs and issues. Why is that? I don’t think the profession is even respected enough by those who are in it. When you are building a bridge, you have to ensure your design is correct as any failure will result in lawsuit, damages and possibly deaths. In the profession of software engineering, these standards are not followed and accountability is not given. I respect those who are self-taught in the field but even being self-taught should come with a certain level of professional standards and a formal way of how things are done. That’s the reason why professionals are licensed and certified. The schools are also to be blamed for not gearing their courses to fit the current events of what businesses are looking for . Due to this lack of control in the field, the reward of a starting career of a software developer/engineer has been significantly diminished where people can pay what they feel like paying and showing no regards for the time, resources and efforts people have invested, even if it is in a beginner’s position. There should at least be a minimum and I think the informality of the field has done more harm than good where anyone can hold the title of a software developer/engineer without going through the rigors of being formally blessed for the field.

  • I’ve got a three-question test I give – one SQL question, one algorithms one, and just for the sake of it, a Boolean logic one, to see if they’ve ever used anything lower-level than JavaScript, and all the questions require a certain degree of lateral thinking if you want to find their ridiculously simple solutions.

    But I’ve found that the most successful hires I’ve made in the past have nothing to do with that: my experience is that real programmers recognize each other like vampires. One of my most successful hires required a five-second interview on the phone before I knew what I was dealing with; another was similar but face-to-face… of course, the problems with this approach are that (a) such people generally require quite a lot of payment, unless you’re lucky enough to get them right out of college (which I was, in one case) and (b) such people are about as common as rocking-horse shit…

    • I think the idea that top-notch folks can sense their own is spot on. I wish I could make it algorithmic, somehow, though. I feel the same as you — that I’ve developed a pretty good sense for the kind of people I’d want to work with and that I’d trust. But “I have a sense” is a lot less empirical than I’d like. (Seems like a hard problem to solve)

      • Jmoore

        Hi Erik,

        I recently posted something about this exact thing … I’m still pretty confident there’s something in watching a fellow coder work that puts an empirical spin on it. If only you could figure out a way to eliminate the ‘stressed’ nature of over-the-shoulder observation…


        Great article!


        • Glad you liked! I think the idea of a screen-cast/recording of someone approaching a problem is an intriguing one, and one that had never occurred to me. I imagine that there’d be some logistics to sort out, but it seems like it could provide some interesting data points. (This also appeals to me, since I do a lot of screencasting)

      • I know. It’s hard to define, let alone to try to formalize. But somehow, it seems to work for me…

    • MarcelDevG

      “real programmers recognize each other like vampires”
      Ha, that’s a very good one!

    • This made me laugh!

      The problem with interviewing (in any field) is that the decision is always made in nanoseconds. The rest of the time interviewing is trying to find reasons for the decision we’ve already made.

      The gig I currently have was all technical c#, but 90% of what I’m doing is JavaScript. I passed the technical interview easily, but really, I think I was hired because I sounded like I could do what they needed to have done.

      Prior to this gig, the interview was “I know nothing about JavaScript, start talking…” Really, just a way to find out if I knew anything about it. As far as technical interviews go, this was the best interview I’ve ever had and if I were interviewing is the style I would use.

      “Your resume says you’ve done ____, tell me about it.”

      Even if I knew nothing about the topic, I should be able to tell if HE knows something about the topic simply by listening to HOW he talks about it.

      • It’s funny: I hired one guy once who appeared to be the best of a bunch of not-really-satisfactory options, but there was a time crunch. And he could talk about things very intelligently. The problem was that he was very, very good at talking – he couldn’t actually do anything. He lasted a short time, then we fired him… and the next thing you know, we got a call from another company asking if it was true he’d left us because we switched to PHP (it wasn’t) and what did we think of him – so we told them something about how he was a really nice guy (he was). Later we put it together: his entire career consisted of talking his way into a job, keeping it for a few months, getting fired and then doing it again. He’d turned it into a career: as a talking, he was amazing!

        • A long time ago, I made an off-handed remark to my wife (not in the industry) about how a non-trivial chunk of people with the title “programmer” didn’t know how to program. She was absolutely aghast and amazed at this, and I realized how telling it was that I wasn’t, after being in the industry for a long time.

          • Oh, it’s stunning. It’s frustrating beyond hell when I’m interviewing people… but then I realize it means I’ll never be unemployed!

          • haha, that’s an amazing way to look at things!

        • Jerome

          I interviewed couple of guys from a country in Asia (living in the USA). They had 6-8 pages of resume! all of them had ‘unit test’ written all over the resume.
          So…I asked about unit test. they did not know about using Mocks.
          You want to tell me that you do unit test in 8 different places, non of them use Mocks? yeah…..right….

          Then we went to the white board. easy questions. they did not know any of the questions.
          After that, their recruiter talked to us and said “they felt it went well, but it was too technical”…. too technical. because they thought they get talk into the job without showing a bit of actual knowledge.

          • My favorite was an old Russian guy who came in – he was at least 65, but could’ve been far older…

            For the SQL question, he said, ‘I kyennot do dis”, so I asked why and he said, “Is YesQL. I kyennot.”

            For the algorithms one (produce an array containing the numbers 1 to 20 in random order) he said, “I kyennot do dis,” so I asked why and he said, “Is not byizniss quyestion!”

            But for the Boolean logic question (“if a=23 and b=36, what are the values afer: a^=b; b^=a; a^=b;”) he wrote down “a=36; b=23” with no hesitation at all, and said, “Iz obvyuss”.

            I think he was so old that he thought entirely in binary!

  • Paul Pettersen

    The worst part of the interview process is the trivial technical section. You know 101 useless terms you barely use when developing but they seem to think you need to know them? I’m coming from the dark side, non-object oriented programming. I exclusively code OOP C#.Net now, but I’d prefer to show them I have these skills. I don’t care how much I looked at the terminology, every interview offers a strange new question. Probably the thing that drives me nuts the most are HR people who think they know IT and have no clue. I just came off a contract where I was offered to program in C#. I jumped at the offer because I wanted to learn the language. The HR person told me it was java-like! I had no technical interview at all. What was supposed to be the technical interview we discussed local restaurants! The most insane thing about this place was after 1 1/2 years on the job, the HR person tried to accuse me of lying on my resume that I knew C#. I made her get it out. It said C++! All I got was “Oh sorry!”!

    • Wow, yeah, having HR people do technical screens is definitely an anti-pattern, in my book. I think you see this because a recruiter/HR screener is a cheaper hourly investment than a developer, but, then again, so is a landscaper or a hair dresser, and companies don’t ask those people to screen developers.

  • Honestly, I think coding exercises during an interview are a waste of time. No, I won’t code for you an arbitrary problem that has already been solved to perfection in a library somewhere. You want to see if I can code? Check my github. There’s plenty of code written by me that solves a problem I had and took the time to solve properly. There, you will see how clean I code, how I design, how I document and test.

    Asking me for the n-th time to do a tree traversal is a waste of time. I don’t care about coding a tree traversal. I have a library do it for me.

    • Personally, when interviewing people, I’d rather look at a portfolio. But you have to bear in mind that 90+ percent of people interviewing don’t have one to offer.

      • It is hard to have a portfolio when the code you write can’t walk out the door.

      • Jerome

        or they lie.
        Honestly I see guys with blogs that are copy paste from other guys.
        now they walk around like code guros that have a nice blog…

    • JohnLWebb

      Stefano I do understand your point (I was the same), but the problem is that most of the people will not have any code that they can show. This includes even the code that is made in private time (maybe he’s freelancing or has a commercial product, site, etc.).

      Also here are few situations that made me realize that coding tests are unavoidable even with a portfolio:
      – A person with a very clean and intuitive code in its repositories on a Bitbucket, it turns out he plagiarised a work from one of his acquaintances.
      – A person with a nice GitHub profile, it turns out that most of the commits in that account were not his.
      – A person that only “contributed” to a single open source project and almost all the commits were not code related but rather documentation, setup, continues integration. It turns out he is the project’s original author with an extremely good grasp of best practices and team management.

      Now when I said that the coding tests are mandatory I wasn’t referring to those classical algorithm or data structure questions. I prefer more the programming tests similar to TestDome-s.

  • Great post. The next time I am interviewing a candidate, I will ask this question: “Tell me about the last time you DIDNT know how to solve a programming question, and what you did about it.”

    • I once heard an episode of Freakonomics where they talked about the tendency of some people to be completely unable/unwilling to admit a lack of knowledge about something. They proposed asking a question that was obviously unknowable just to see if candidates were the sort of people who would bluster/bluff or who were comfortable saying, “I don’t know.”

  • David S

    Another insightful post! Thank you!

    I have an unusual perspective on this one: I think I’m one of the few people in the world who genuinely enjoys job interviews (As the interviewee, not the interviewer!) I guess I’ve been to enough of them that I’ve gotten pretty good at them. It’s become an exciting challenge to present myself well, think fast for the trivia questions, and gain perspective into a new company.

    However…. I remain not entirely convinced that the typical software company’s job interview process adds any value over taking all the candidates they’ve decided to interview, and picking names out of a hat.

    • Thanks for the kind words. While I can’t claim to enjoy job interviews, I do sort of understand what you mean about enjoying sort of a mastery of the game. I’ve written more extensively about the interview process in the book I’m working on, but the tl;dr version of that is to agree with you that interviewing is probably not significantly better than random selection (after some screening and resume fact-checking).

      Frankly, I think that the most effective ‘interview’ results from people on a team putting forth a person that one or more of them knows. The ‘interview’ is then a discussion of mutual fit from a comp, interest, and compatibility perspective.

  • 123

    what does ”fly someone out” mean?

    • I meant that the company was going to pay for my travel to interview with them. I was living near Chicago, and the job was in Boston.