Do pre-hire assessments impact turnover, applicant faking, and completion rates? with Carter Gibson of Modern Hire.

Welcome back to another episode of the High Volume Hiring Podcast, the podcast that features news tips, case studies, and interviews with the world’s leading experts about the good, the bad, and the ugly when it comes to high-volume hiring. 

The High Volume Hiring podcast

In this episode, host Steven Rothberg, College Recruiter’s Founder and Chief Visionary Officer, interviews Dr. Carter Gibson, a Managing Associate at Modern Hire. Modern Hire specializes in pre-hire assessments that supports over half the Fortune 100. Carter’s algorithms assess millions of candidates per year and his research appears in some of the top journals in the field.

Pre-hire assessments and interview tools

Dr. Carter Gibson discusses some of the ways that his company has been utilizing collected data sets to become more effective, and in turn help their clients achieve hiring success. He addresses how comployer’s hiring goals have changed since the onset of the COVID 19 pandemic and how honing into these goals allows your organization to get the most value out of pre-hire assessments. Finally, he offers some advice to those employers who are resistant to adding assessments to their hiring process.

Check out the full episode here:

Subscribe here to follow the High Volume Hiring podcast and stay up to date with new episodes.

TRANSCRIPT

Steven (00:10):

Welcome to the High Volume Hiring Podcast. I’m Steven Rothberg, the founder of College Recruiter Job search site at College Recruiter. We believe that every student and recent grad deserves a great career. This podcast features news tips, case studies, and interviews with the world’s leading experts about the good, the bad, and the ugly when it comes to high volume hiring. Thanks for joining us. Today’s desk is Dr. Carter Gibson, a managing associate at Modern Hire. He sits on the professional services team designing, implementing, validating, and supporting pre-hire assessments, as well as overseeing ongoing research on topics such as turnover, applicant faking and completion rates. Modern hire specializes in pre-hire tools that supports over half the Fortune 100 Carter’s algorithms assess millions of candidates per year, and his research appears in some of the top journals in the field. Dr. Gibson, welcome to the show.

Dr. Carter Gibson (01:08):

Thanks for having me, Steven.

Steven (01:10):

So before we dive into a few questions that I have that I think the listeners are gonna be pretty interested in maybe you could tell us a little bit about you modern hire, things that I didn’t cover.

Dr. Carter Gibson (01:24):

Yeah, of course. I think you did a pretty good intro there. I’m an industrial psychologist by training. I work for a consulting firm and like you said, modern hire and we specialize in pre-hire tools, so that’s things like interview technology, can self scheduling. Most importantly for my role is the assessments. So we do work with a huge number of companies, roughly half the Fortune 100. So that means we have a lot of assessments with over a hundred thousand candidates a year and several that are well into the millions. I’m on our optimization team, and my role specifically is I manage the algorithms behind those assessments. So I either lead or advise or review most of the changes to those algorithms as part of business impact studies, what some people might call predictive validations. And you mentioned this, the other part of my job is leading is research and looking at consistent trends across our clients and questions they’re asking to lead. A lot of the research we find is that there is a lot of academic work on some of these topics, but it often lags a few years behind. And the truth is we just have really amazing data sets, so we try to use that to make us more effective and by extension help our clients.

Steven (02:32):

Cool. That’s great. And one of my friends met him through work. He’s become also a personal friend, is a co-worker of yours and I would be remiss if I didn’t give a shout out to him. But Mark Wenzel, thank you very much for the introduction. And one of the things that Mark has told me over the years about the algorithms used by Modern Hire as well as some of the other assessment companies, not, but certainly not all of them, is that the modern assessments are really looking towards predicting the productivity in the workplace. And my words not Mark. So if there’s a better way of phrasing that, please <laugh> correct me. Whereas the old fashioned assessments, those that many of us grew up on, the Meyers Briggs, whatever, the sort of the personality, they were never scientifically validated, they were never really suitable for the workplace. Is that a fair summary?

Dr. Carter Gibson (03:34):

Yeah, I would agree with that, that there are a number of assessments that end up in pop culture. Myers-Briggs is probably one of the most common ones that don’t really have a whole lot of organizational value outside of maybe team building or something like that. But when we’re talking about selection specifically in high volume context, we are drawing on probably a hundred years of research. But the truth is, I mean, a lot of things when it comes to just being able to take assessments online has really exploded their popularity and their practicality for use in organizations. So the field has really changed a lot over the last decade or two

Steven (04:08):

And certainly within the last decade or two. And you would definitely know that better than me and probably most of our listeners. I would even say from my perspective that there’s been a massive change even in the last five years if you haven’t looked at some of the assessments in 6, 7, 8 years, and your recollection of them was, Well, this sounds fun, but I don’t see how it’s gonna help us figure out who’s gonna do well in the job. Then I think you, oh, it’s yourself to have another look. As far as the Meyers Briggs, another element that I would say is true about that is it’s a fun conversation starter with your spouse and very often will end a conversation I’m an E N T J, which means that I have this need to tell people what they’re doing wrong and I think it’s for their benefit. And my spouse does a very, very good job of ending a conversation with me saying, Oh, you’re just being ntj. So there’s that first question. So one of the things that I think that a lot of the listeners wanna know is the goals of hiring and how we evaluate program success. Have those changed since the onset of the COVID 19 pandemic?

Dr. Carter Gibson (05:26):

Yeah, I mean I think a lot of people point to Covid as the pandemic, as this giant agent of change. I don’t know if it fundamentally changed things, if it sped up a lot of trends that had already been occurring. You talked about how the nature of assessments has changed a lot that I think you just implicitly referred to gamification, and I can touch on that a little bit and it feels like the field has moved away from that for a number of reasons. We talk about the goals of hiring. I could take this in two directions. One is that how we’re being evaluated as vendors, what does accountability look like for our tools has really expanded for the majority of assessment history, it’s been about predicting performance. That’s the goal of firing is to identify high performance talent. But the truth is it’s gotten a lot broader in terms of the metrics people care about.

(06:11):

In fact, we we’ve launched assessments that weren’t even trying to predict performance that exclusively predicting pre-hire retention. So they were turnover predictors, not performance predictors. And that’s certainly something that’s changed the last few years. You think about the pandemic and the labor shortage that came afterwards one of the ways that you don’t have to source as many candidates is if you can keep your current employees in place for longer. So that was just one of the mechanisms and you think about a lot of entry level jobs, What I can show you is that even a low performer at two months is way better than a high performer at one week just because you’ve learned and been trained on the job. So for these companies, they’re like, You know what? I’d rather have a medium performer that stays here six months than a high performer that stays here one month.

(06:52):

So that’s been a major shift is that for a lot of, especially high volume or high turnover entry level roles, that turnover is probably more important than performance. We also saw a huge emphasis on time to offer or speed to hire. What happened at the very beginning of the pandemic is there was this massive competition. So a lot of companies stopped hiring, but then on the other end, there were a huge number of companies that had massive increases in hiring. So think about grocery stores, your Walmart’s, Amazon’s distribution centers. We saw some of our clients have 15 fold increases in candidates. So if they were getting 1,005,000 candidates a day, they may have 50,000 candidates a day at the beginning of the pandemic. So speed to offer became this huge source of competitive advantage. You saw companies literally advertising that you can have an offer or you can start the job within a week or an offer within 24 hours and that type of thing.

(07:45):

I mean, to get to that, you necessarily have to make compromises. So I’d point to turnover, I’d point to speed to hire another one. That’s interesting because it used to just be an aspect of the test. Those are you that are familiar with hiring. There’s a whole legal framework. When I say a assessment or hiring process needs to be fair. There’s very specific operationalization. So things like for fifth ratios and effect sizes and significant tests. And five years ago I would’ve said that it’s usually just kind of a check box. You’re like, Yeah, the test is legally defensible for many of our clients, diversity has become almost an outcome in and of itself. I think about for many of our clients, every so often they’ll present to C-suite executives, the top of the company, and they may have one slide or two slides and two minutes of their time.

(08:31):

They’re important people and you really don’t get much time with them. And several years ago you would’ve presented a performance slide. Hey look, the assessment predicts performance these days we’ve had clients that would’ve chosen an ethics or fairness slide. So if you think about a study, you could say the year before an assessment was implemented, what were the demographic breakdown of all your hires and then the year after? And if you could show that you doubled your number of Hispanic hires or something along those lines that may be more impactful or important to an organization than predicting performance. So that has been a pretty serious shift these last few years,

Steven (09:07):

And that’s really interesting. That’s the first time that I’ve heard anybody articulate it that way. And I would say, and see if you agree with this, that things like diversity and performance are certainly not mutually exclusive. They tend to go hand in hand. I think there are a lot of studies that show that the more diverse a workplace is that the more productive that workplace tends to be. So the organizations that you’re talking about where you might go into that C-suite and get your 90 seconds and you’re showing that diversity slide, there are multiple reasons why they wanna see that data. Some of it just public relations wise, but underlying that I think also must just be the implicit understanding that it does lead to productivity. Is am I off base on that?

Dr. Carter Gibson (09:58):

Yeah, that could certainly be it. Maybe I take the cynical route and I go the pr, but I think some of it, it’s not always that cut and dried. I think the truth is a lot of the legal standards in this area are a lot more lenient than a court of public opinion would be. So not to go too far in the weeds here, but you think about a four fifth ratio and with that is saying if your assessment is fair, imagine that white applicants were passing your assessment 80% of the time. The four fifth ratio basically says as long as the protected classes, say African Americans or Hispanics, were passing at 80% of the rate of the majority group that it’s still fair. And in that case, you’d have a pass rate of 64% versus 80%, and that’s legally defensible. You’re right on the edge, but you’re kind of there. But imagine if those numbers got out in the Wall Street Journal or the New York Times or something like that that said this major company white applicants are past 80% of the time Hispanic applicants pass 70% at a time. Do you wanna go out there and say, Guys, this is legally defensible, this is within legal guidelines.

(11:00):

A lot of companies are far exceeding those standards. So I think there probably are a number of reasons for it, but it’s been something that resonates with a lot more clients these days.

Steven (11:12):

Interesting. 187 years ago, I used to work for a, was then a Fortune 50 company, I’m Honeywell, and down the hall from me, we had an office of affirmative action and it was all about compliance that there wasn’t anything there about diversity, inclusion, equity, and it was before even like DEI was, the words were being used. It was a different era. But you’re absolutely right. I can just imagine some big company, some article coming out in the New York Times or the Wall Street Journal that basically said something like, African American women are a third less likely or a quarter less likely to be included in the finalist pool or hired by XYZ company. And even if there’s no employment law issue there, it is definitely gonna be harmful to that business. Another question for you one of the reasons that was really looking forward to talking with you is because modern hire, your clients skew so heavily to the largest organizations that the half of the fortune hundred, et cetera. What are they doing now with their assessments that whether they’re modern, higher clients or not, that they weren’t doing just a few years ago?

Dr. Carter Gibson (12:39):

Yeah, I think that’s an interesting question simply because you think each one of these companies is an island and they’re all kind of coming to making the best of their own situations. And the truth is a lot of them end up coming to the same conclusions. They’re asking for the same things. So when I say a client’s interested in diversity it’s like a trend that I’m seeing across a bunch of them. One of the most consistent trends that I’ve seen the last few years across all these clients is really honing in on what I would call validity per minute. So you think about your assessment and it’s trying to predict various things, whether that’s performance or turnover or helping with diversity or something like that. They, there’s been a massive push to shorten assessments. If you took the average assessment length in 2010 and 2015 and 2020, you would see a steady trend towards much, much shorter efficient assessments.

(13:30):

When I started there, we had plenty of assessments that were over an hour long, maybe an hour and a half long. Some of them might have even been proctored in specific cases. You really don’t see that anymore. Even at high level position the longest, you’re gonna see those, for the most part an hour, maybe 45 minutes entry level positions they may launch at 20 to 30 minutes, but the client starts pushing to shorten it. They want it down to 15 minutes or 10 minutes. When you see some vendors out there that are pushing two minute assessments now, I don’t really believe you can do much in two minutes. I think that’s, they’re trying to give the market what they want and not providing a whole lot of value, but there is a massive demand for very efficient assessments that don’t take very much time.

Steven (14:23):

I would think that an assessment for say a part-time warehouse job hourly would be far shorter than even an entry level role with a big four accounting and consulting company. With those kinds of couple of examples roughly how many minutes are you seeing now taking place for the warehouse versus the accounting and finance kind of positions?

Dr. Carter Gibson (14:55):

So what I would say is for my professional opinion is that you need to earn those reductions in time. I think a lot of people think of an assessment as this tool that you launch and it’s just kind of fixed. My opinion is that it should be changing and adapting both on the algorithm side about what’s being scored and how it’s being scored, but then also on the content side. So my recommendation for an entry level role, if you started at say 25 or 30 minutes and then as you gather more local data, whether that’s local turnover data or local demographic data or local performance data, that’s when you start really cutting that down because if you’re rolling out 15 minutes, you can’t be sure that all of that’s going to be working and then you whittle it down so that you do get those high validity per minute estimates.

(15:38):

I would also ask, what are you most interested in? If you’re saying we only care about turnover, that’s gonna be a shorter assessment if you can really hone in on exactly what you want to predict. But if you say, Hey, we care about all of these things, then for entry level you should probably start 20 to 30 minutes and then shop from there. Now you talked about a more professional campus job, those types of assessments, you have a little bit more time. Maybe you start that at somewhere around 35 to 40 minutes and then cut that down once you start getting more local data. That’s what I was always recommend. I think some companies put the cart before the horse and they say, We want 20 minutes right now. And I’m kind of like, Well, you need to earn that. That just takes time. There’s no way to force that from day one if you really care about those outcomes.

Steven (16:20):

Interesting. Yeah, I would imagine that a big part of your job with any client, and I hear you, it would be different role to role and maybe probably also different geographic areas, et cetera, is looking at each of the assessment questions and it’s like, okay, well we’re asking this question, Does this actually differentiate our results? Are we seeing different results based upon the answers to this question? Is it adding any value? Is probably a better way of putting it. If we get rid of this question, does it change the outcomes? And if not, it’s looks like we’re gonna be eliminating that question. It’s just, it’s redundant and repetitive a as you might say.

Dr. Carter Gibson (16:59):

Yeah, I mean you’re looking at a huge number of things. It may be criterion data. Does it predict performance? Does it predict turnover? It may be group differences. Is it a fair item that different demographic groups are answering it in a similar way? Sometimes it may even just be an efficiency thing that the, it’s taking longer. A cognitively loaded problem solving item or situational judgment item. You may be like, Well, it’s doing pretty well, but it’s taking two and a half minutes to administer that one item. And sometimes it might be an item difficulty question. You might cut a simulation because it’s sometimes hard to get it right. It may be too easy where 90, 95% of people are doing too well. Or it may be too hard where it’s just incredibly difficult and you say like, Hey, it’s difficult. Maybe that’s good for the assessment, but then that’s not good for the candidate experience.

(17:42):

That’s a whole nother thing that you have to consider in all this is like what do candidates actually enjoy doing? Cause a lot of content out there, like personality is a good predictor and tends to be fair. So it’s pretty efficient item by those understandings. Candidates tend not to enjoy it as much. So it’s like what are the levers you’re trying to pull? Are you trying to build the most? So there’s always gonna be tensions and compromise. Do you wanna do lots of measurements hard to do that fast? Do you want something super cognitively loaded and difficult? Well then it’s probably not gonna be, candidates are gonna get frustrated. There’s all those different tradeoffs. So it’s really about tailoring that assessment to get to the point where it’s exactly what the company wants. And I think what’s cool, but my job is you look at different companies and after three or four years after launch, the assessment looks so much like that company, you can look at what they value and care about by what their assessment has become.

Steven (18:29):

Interesting. And I personally, I hate personality tests. I always fail them, but that’s another issue for another podcast, I think. But well, if you got time for one more question, this is one that I hear all the time from employers who are resisting adding an assessment as part of their hiring process. And that is their concern that if they add some kind of an assessment whether it’s at the very front end of the process or at the back end of the process, that they’re gonna see such a massive drop off in the number of applicants that they won’t be able to hire the people they need. I remember this was one of the very first conversations that I had with Mark Wenzel, and I think he kinda shared some sort of general numbers with me that there’s a bit of a drop off, but in terms of the number of applications, but there really isn’t much of an impact at all on the speed to hire, the number of people you hire, et cetera. And that conversation goes back 10 years. So things may have changed, but if you’re talking to an employer who is looking at adding an assessment into their hiring process and you hear that objection of, Well, we’re not gonna then have any applicants, we’re not gonna be able to hire any people. How do you address that?

Dr. Carter Gibson (19:53):

Yeah, and honestly, dropout fallout, completion rates, attrition. I think what’s interesting is that I see companies use different words for it because there’s not necessarily fixed clear language for how they should be talking about it. And it’s probably one of the single most common questions we get from companies. So we’ve actually done a huge amount of research on this topic. I mean, a lot of it’s publicly available. We published it in a number of journals over the years and I think the assessments end up becoming almost an unfair scapegoat. I think one of the cool studies we did is we looked at dropout through the full application from when someone first clicks on the apply link all the way to when they start in the job. And the numbers were startling because dropout happens everywhere. In fact, one of the things that we hear is like, Oh, applicants don’t wanna take an assessment, but obviously the other stuff filling out their name and address and stuff, they know they have to do that.

(20:42):

So no one dropout during those pages and it was a large distribution center, it was over a million candidates a year we found is <laugh>. They lost about 10% of their applicants on the first page. So I think there’s a disconnect where companies think an applicant is serious as soon as they start the application. The truth is applicants are still browsing potentially even to when they start the job. So they click on the job, they click apply, and it’s like, what’s your name? 10% of your candidate pool is like, Nah, I don’t really wanna give them <laugh>. So it’s like it’s happening all over the place. And the distinction I would make is between good and bad dropout. So good dropout is if people that you don’t want for that role, whether they were gonna be low performers or turnover risk leave and drop outta your process.

(21:25):

Whereas bad dropout as if all your high performers were leaving, they saw something they didn’t like and they walked away. And we’ve actually found, we did a study a few years ago that showed that the dropout happening during the assessment was actually good dropout. And the way we did this study is so if somebody, they complete part of the assessment, so our assessment’s gonna be like multi method assessment. So there’s gonna be like situational judgment, maybe a simulation and some bio data and maybe personality, something along those lines. We take everyone who had completed that first section of the assessment, maybe 15, 20% of the total assessment use that as a predictor of dropout. And what we have found across millions of applicants every time we’ve looked at it is that it’s people that were scoring lower is that early performance on the assessment is a fantastic predictor of dropout.

(22:10):

High performers are like 25, 30% more likely to complete an assessment than people that we’re doing poorly on in the early on. So it’s kind of like the assessment is helping you in two ways. One, it’s helping you sort your candidates from those who finish it and telling you who’s likely to be a good hire, but it’s also acting as a preview, a realistic job preview for candidates where they’re self-selecting out, they’ve learned something. What I would say is a lot of companies focus on the assessment as a source of dropout when we’ve shown empirically that that’s often good dropout is that once they complete the application, that period from application to hire there was a client that we had that collected data on why people were leaving after that and two thirds of ’em, cuz they accepted another job offer. And we actually showed that that was actually higher quality applicants. So if you think about your applicant pool every day, it’s sitting there post application. This is why I think speed to hire is such an important metric is that the high performers are the ones taking jobs other places. So

Steven (23:03):

Yeah,

Dr. Carter Gibson (23:04):

Clients often focus on this, Hey, we gotta get this 30 minute assessment down, but then they take two weeks to get an offer out to people and that’s where your dangerous dropout is actually happening.

Steven (23:12):

Yeah, they have. They’re the ones with the options. Well, Dr. Gibson, thank you. This was so enlightening. For listeners who wanna learn more about you personally or Modern hire, where should they do that?

Dr. Carter Gibson (23:25):

Yeah, actually the Modern Hire website hosts all the publications, all the papers that we’ve written on these topics like turnover and completion rates. So that would be a great source to access anything that we’ve written, any of the white papers or academic articles.

Steven (23:39):

And that’s modern hire.com?

Dr. Carter Gibson (23:41):

Yes, I believe

Steven (23:41):

So. Awesome. Well, thank you so much for joining us today on the High Volume Hiring podcast, a co-production of Evergreen Podcast and College recruiter. Please subscribe for free on your favorite app, review it and recommend it to a couple of people who wanna learn more about how best to hire dozens or even hundreds of people. A special thanks to our producer and engineer, Ian Douglas. I’m your host Stephen Rothberg of job search site college recruiter. Each year we help more than 7 million candidates find great new jobs. Our customers are primarily Fortune 1000 companies, government agencies, and other employers who hire at scale and advertise their jobs with us. You can reach me at stephen@collegerecruiter.com. Cheers.