Liz Ryan
often polls her followers. Generally speaking, she's an excellent pollster. She doesn't write double-barreled questions, and they are topical and relevant to modern events. But there's a common misconception that just because someone creates a poll, that said person is a researcher. Let's take a look at Liz's poll for context:
In under five hours, she has nearly 7,000 responses. Most researchers can only dream of sample sizes that large, especially for an ad-hoc poll. But, what can and - more importantly - what can we not learn from the data? Let's dive in and discuss.
- As of this writing, the poll is not closed. So, we can infer that the results will change. Also, since LinkedIn doesn't allow for a quota, we don't know the final N (sample size). So, we know how many people have taken a poll, and if we did more research, we could determine the average response to most of her polls when they close and estimate how many responses remain to be gathered. But we must label any presentation of this as "preliminary findings." This is crucial, because oftentimes (with much smaller N sizes) researchers can be pressed to close studies early to save time and/or money with the belief that what's been gathered is representative of the results if the study was allowed to complete. This is erroneous and speculative.
- We do not know the demographics of the audience - this is crucial. While Liz didn't and isn't obligated to explain the reason she conducted her poll, the importance of a gap in one's résumé is purely academic if the people answering aren't hiring managers. Most humans, by nature, are guarded about their personal lives. "It's nobody's business!" is a common comment beneath the poll. But LinkedIn isn't a robust polling platform, so I must make certain not to fault Liz for not restricting this question to people who are or are not hiring managers. Indeed, she might not care at all. But, if I were to use this as a secondary source as a researcher, I would need to list many caveats about how the data could and could not be interpreted.
- What actions can be taken based on the data? Extraordinarily little. Researchers call untargeted surveys polling the "general population," as "gen pop." But even when we target the general population, we typically screen people in or out based on a host of criteria, and collect additional demographic data such as age, gender, geographic location, et cetera, so we might look for patterns. In Liz's poll, hypothetically, are people over say, 50 more likely to care about gaps in résumés than those under 50? Is there a statistically significant difference by gender? Do people in different industries view this differently? None of these questions can be answered without a screener. What we can tell is that by a roughly 2:1 ratio, people of unknown backgrounds believe that a gap in a résumé should not be discussed during an interview by the potential employer. What are the implications of this? That remains unknown based on this poll. Assuming that hiring managers are of a similar mind as the general population, there's a 2/3 chance that you will not be asked about a gap in employment. But, if hiring managers account for 90% of the people who do care, there's only a 1/10 chance that you will not be asked about a gap in employment. So, caveat emptor is the simplest advice on interpreting these results.
- There are no definitions, so the data is skewed based on interpretations of a "gap." This is really the fatal flaw of this poll from a scientific research perspective. As someone who has hired people before, and speaking only for myself, I didn't care about gaps of less than three years. Anything beyond that and I wondered if the candidate possessed the skills needed to perform research work. Typically, this centered around technology as most research methods haven't changed much. But another thing missing from the question is context (sometimes called "set up" or "introductory text.") Just where is the gap in employment? Again, using myself as an example, a ten-year gap wouldn't even be on my radar if it were over three years ago. However, if I was presented with a candidate who is coming off a five-year gap and has no notation of recent training or education, I might ask the question, "How have you kept up with changes in research technology over the past several years?" This then brings up the question of what Liz means in her initial question with the word "doing." I do not care if the person I was interviewing took a sabbatical, was raising kids, caring for a sick family member - whatever, that truly is inconsequential to me. But the candidate's efforts to remain current on the skills needed to perform the job? Effort implies "doing" something; therefore, I am asking what this person was "doing" during the gap. Thus, the question persists: how do I answer Liz's poll truthfully?
So, how might we write the question given the limitations of LinkedIn to get a more useful answer? Here are some examples:
- Should the personal details of what you were doing during a gap in employment of greater than a year be important to a potential employer? (yes/no/undecided)
- How would a potential employer who is interviewing you who asks about the personal details of what you were doing during a gap of employment in your résumé (greater than one year) impact your willingness to accept an offer from that employer? (increase/decrease/no change)
- Is it appropriate for a potential employer to ask you how you remained abreast of relevant job skills during an employment gap of two years or more during an interview? Assume that no details about why you took the leave are asked. (appropriate/inappropriate/no opinion)
I have said and must repeat that I love it when people ask questions, regardless of their backgrounds. However, the way those questions are asked can convey an unintended meaning and generate results that aren't reflective of what the author intended. Clarity in writing the prompt is crucial, and if you are going to use the data to take action, please try to #hirearesearcher to write or review it for you.
(You can see the final results of Liz's poll after it closes by clicking on the link in her name at the beginning of the article, if interested. Also, my thanks to
Jeff Sauro
and the folks at
MeasuringU
for their excellent blog, which I frequently reference to introduce research to non-researchers)
