A Scientific Approach To Hiring

Hiring managers often fear making a poor hiring choice, and rightfully so—it can cost the company more than just money. To mitigate this risk, many hiring managers have developed intricate hiring processes. But do these processes really work?

Unfortunately, research indicates that complex and arbitrary procedures do not consistently predict a potential hire's job performance unless they are grounded in proven selection methods.

The good news is that you can enhance your confidence in making a sound hiring decision by utilizing a streamlined process rooted in proven selection methods that reliably predict an individual's on-the-job performance.

We delve into research findings specific to hiring technical talent. Our focus lies on advocating for best practices and discouraging ineffective methods.

Selection methods are the tools used to assess candidates. Validity, a measure of how accurately a method predicts future job performance, often ranges from 0 to 1 (e.g., 0.61), also interpretable as a percentage (e.g., 61%).

While data-backed hiring might seem impersonal, it significantly reduces risky hires, eliminates bias, and enhances workplace diversity.

Our recommended hiring process is discussed in detail in another article; here, we elucidate research findings, high-validity methods, and considerations for candidate experience.

Research Insights: Our insights are rooted in extensive research spanning a century and diverse industries. A 2016 meta-study consolidated this research, revealing key findings:

Certain methods exhibit higher validity. Combining specific methods results in enhanced validity. The highest validity arises from combining methods with low correlation. A 2020 study analyzing the validity of brainteaser questions served as a supplementary source. Focus Areas: We concentrate on the following methods, commonly used in technical talent acquisition:

GMA Tests: Assess general intelligence and problem-solving skills.

Integrity Tests: Identify counterproductive behaviors and traits like theft, drug use, or emotional stability.

Work Sample Tests: Candidates perform job-related tasks mirroring actual work conditions.

Job Knowledge Tests: Evaluate candidates' depth of job-related knowledge through specific questions.

Unstructured Interviews: Varied questions tailored for each candidate.

Structured Interviews: Consistent, predetermined questions to minimize bias.

Brainteaser Interviews: Include three types: justification-based, definitive-answer, and oddball questions.

Years of Previous Job Experience: Assess candidates' work history relevance.

Years of Education: Consider candidates' educational level.

Interests: Evaluate relevant hobbies for job suitability.

These insights empower a refined approach to hiring technical talent, ensuring a more accurate and fair selection process.

GMA tests GMA tests have the single highest predictive validity at 0.65. If you do nothing else in your hiring process, administer a GMA test—it gives you the best chance at predicting an applicant’s future job performance. The other selection methods should supplement your use of a GMA test.

An important aspect of these tests is that, in addition to measuring general intelligence and problem-solving ability, they are also a predictor of learning ability (Schmidt and Hunter, 1998). Learning ability is a large part of future job performance and internal promotion. Your team will benefit from having employees who are smart, can problem solve, and will continue to learn on the job as they are assigned new projects and challenges.

Integrity tests Integrity tests are somewhat middle of the pack in terms of stand-alone validity (0.46). However, integrity tests have low correlation with cognitive ability tests, so when used in combination with a GMA test, the validity increases to 0.78. Using these two selection methods in tandem drastically improves your ability to predict the future job performance of candidates.

Integrity tests, aside from highlighting risky behaviors, also give insight into personality traits. High integrity often correlates with high conscientiousness and high agreeableness. Conscientiousness in hiring is generally defined as wanting to do your work well and thoroughly, and agreeableness in hiring is defined as getting along well with others. It’s obvious why you’d want to interview candidates that score well on an integrity test because it’s an indicator of the developer’s personality traits and a decent predictor of their job performance.

Unstructured vs structured interviews When used on their own, unstructured and structured interviews have the same predictive validity (0.58). However, when used alongside a GMA test, structured interviews outperform unstructured interviews. This is because structured interviews don’t correlate as closely with GMA tests; structured interviews and GMA tests measure different things, and so the two used together are a stronger predictor of job performance.

The debate about the benefits of unstructured and structured interviews is ongoing, but what the data supports is using employment interviews to evaluate candidates. Interviews as a selection method, when used with GMA tests, have the next two highest validities. Facet favors structured interviews because it allows us to fairly evaluate candidates and efficiently train and prepare our interviewers.

Interests Earlier meta-studies reported low validity for interests as a predictor for job performance. However, the 2016 meta-study reanalyzed data and found that vocation-related interests do predict job performance with some level of validity (0.31). While significantly lower than other selection methods when used alone, evaluating interests alongside a GMA test provides the fourth highest validity (0.71).

Remember that for this to be accurate, the interests that you as a hiring manager need to consider are vocation-related—in this case, that might look like developers having personal coding projects or contributing to open-source projects.

Take this with a grain of salt—Stack Overflow reported that 72.87% of professional developers code as a hobby. If you are extrapolating that simply because they code outside of work, the candidate is a good developer, your bar isn’t high enough. That number of developers who code as a hobby potentially speaks more to passion for coding rather than indicating technical expertise.

Work sample tests vs. Brainteasers On its own, a work sample test has a predictive validity of 0.33, and when used with a GMA, that number rises to 0.65. This data may have given rise to the frequency of using coding tests like Leetcode, HackerRank, or Byteboard to evaluate developers. As a hiring manager, you might think you are doing a work sample test by utilizing these coding tests, but you aren’t. You aren’t modeling the environment or even the types of questions that will be required by the job, and so these coder tests are no better than brainteasers.

Famously, Google used brainteaser questions to evaluate developers during interviews. But in 2013, their chief of human resources announced they were discontinuing the practice, saying they were a waste of time. Brainteasers have not been extensively tested, but a 2020 study researched the predictive validity and suggested that brainteasers correlate with cognitive ability rather than job performance.

The predictive validity for any type of brainteaser when used with a GMA test do not raise to a statistically significant amount because they are closely correlated. Besides that, their predictive validity is quite low compared to other selection methods. Your time and finances as a hiring manager would be better spent investing in a tailored work sample test rather than utilizing coder tests that function as brainteasers. If you do this, you’ll have significantly better hires.

Job knowledge tests Job knowledge tests are also middle of the pack when used on their own (0.48), but they rank highly when used with a GMA test (0.65). Job knowledge tests might be a good alternative to work sample tests (and certainly to brainteaser coding tests) because job knowledge tests are easy to administer in-person and don’t require the candidate to take a coding test during their personal time.

You can expect the validity of your hiring process to increase if you administer a job knowledge test during a structured or unstructured interview alongside a GMA test.

Job experience and years of education Once you have reached a certain threshold, your performance is more impacted by your ability to learn on the job than by what you’ve done in the past. Ability to learn is best tested by a GMA, not by your previous job experience (0.16 validity) or how many years you’ve gone to school (0.10 validity).

When used with a GMA test, these two selection methods have a drastically improved predictive validity (see Table 2). This is because the GMA test is an accurate predictor of job performance, not because previous experience or education are good predictors. Using these selection methods on their own to evaluate if the candidate will be a good hire is only barely better than randomly selecting a candidate’s name out of a hat. And if you are using a GMA test as a baseline, you are better off using one of the higher-validity methods to better predict job performance.

Combining Selection Methods You’ll notice that, when combined with a GMA test, all the selection methods experienced an increase in validity. Previously low-validity selection methods, like brainteasers or job experience, become somewhat effective at predicting job performance.

That increase shouldn’t be used to justify using low-validity selection methods. The predictive validity that is statistically significant is a GMA plus an integrity test (0.78) and a GMA plus a structured interview (0.76). And, as noted in the 2016 meta study, “A further advantage of these two combinations is that they can be used for both entry level hiring and selection of experienced job applicants.”

These two combinations have the highest predictive validity because the selection methods have very low correlation, meaning they don’t test similar things. Either of these combinations will allow you to test for multiple characteristics in candidates with a high probability of predicting future job performance.

As you plan out your hiring process, combine selection methods that don’t correlate and are efficient to administer. Your hiring process will maximize the probability that any given candidate will be a good hire.