The Myth of the Data Scientist Shortage
Technologists and business folks alike overstate the shortage of data scientists. They just need to know where to look.
Data scientists—people who can manage and analyze big, unstructured data—were once as scarce as vegetarian dogs. If your business wasn’t based in Silicon Valley or Boston, if you couldn’t offer massive stock options, and if you didn’t have a sexy business model, you were unlikely to be able to hire any. When I interviewed 35 of them in 2013 for an article in Harvard Business Review, my co-author (D.J. Patil, now a data scientist in the White House) and I wrote, “The shortage of data scientists is becoming a serious constraint in some sectors.” The most common educational background among the 35 data scientists I interviewed was a Ph.D. in experimental physics, and there aren’t a lot of those sitting around.
But now the world of data science has changed dramatically. There may not be a glut of data scientists, but they are much easier to find and hire than they used to be. If you’re based in Omaha, you’ve got a good shot at finding some good ones. If you can offer only a decent salary, you’ll probably be OK. And even the most traditional business can hire them these days. In short, there is no excuse for not building a data science capability.
Here are some common excuses companies use for not employing data scientists, and why they’re no longer valid:
“Universities just aren’t turning out data scientists.” Au contraire. There are more than 100 programs at U.S. universities alone that focus on analytics or data science. Some schools, like Northwestern University, New York University, and the University of California at Berkeley, have more than one degree program in data science. These programs are already churning out thousands of graduates.
There aren’t enough quantitative Ph.D.s to go around. First of all, you probably don’t need a Ph.D. data scientist. There are plenty of Master’s degree graduates out there who will have all the skills you need. Moreover, you’d be surprised how many unemployed or underemployed Ph.D.s there are in quantitative and scientific fields. There are programs that provide data science internships to Ph.D.s like the Insight Data Science Fellows program, and programs at vendors like SAS and Microsoft can ensure that the Ph.D.s have all the latest skills.
Data scientists don’t want to work in the hinterlands where my company is based. Think again. There are universities with programs in data science or analytics in Alabama, Kansas, Nebraska, South Dakota, West Virginia, and many other states far removed from the east or west coasts. At least some of the graduates of those programs are willing to work where they went to school.
Data scientists fresh out of school won’t understand my business. That may well be true. So train your own employees in data science. Cisco Systems, for example, worked with two universities to create distance learning education and certification programs in data science. More than 200 data scientists have been trained and certified, and are now based in a variety of different functions and business units at Cisco.
My company isn’t hiring anybody, but we still need data scientists. Again, you can retrain existing employees. You could take the Cisco approach and create a custom program. Or keep in mind that many of the university programs are online and can be taken part-time. If you make it known to your employees that your company needs and values data science skills—and that you might pay for some of the education—you will probably have some certified data scientists within a year.
Of course, it still may be difficult to find highly productive and effective data scientists, as with any sort of job. But there are now many potential candidates out there. No matter what your business is or where it’s based, chances are good that you can find someone to help with your difficult data problems.
— by Tom Davenport (tdav), senior advisor, Deloitte Analytics, and distinguished professor, Babson College.
This article was originally published in the Wall Street Journal (CIO Journal) on August 11, 2016.
Student at University of San Francisco School of Management
4 年Dear LinkedIn fraternity, With positivity as the core enabler of the next wave that we see emerging from the current crisis, my team has articulated what we all collectively think/agree as the first major focus—Remote Work Model. A quick read for 5 minutes can prove worthwhile, and here is the article, straight on its message – https://kairostech.com/coronavirus-recovery-bytes-from-kairos/ Do comment on what you think and we will respond to you ASAP. It will be good to everyone as you get to share it in your relevant networks!
Quantitative Expertise with knowledge of Banking, FinTech and Media products. Enhancing current KPIs and creating new Metrics. Actually resolving regulatory findings. Absolutely Dislikes WFH.
7 年I agree with Tom's points but I can see why there is a view of a shortage: inflation or exaggeration of skills. Also, the definition of a data scientist may vary greatly from company to company, and even within a company. I stopped counting the number of times a candidate labeled himself/herself as a data scientist, solely on the ability to create Excel pivot tables. Or the individuals who are experts in SAS because they copied and pasted someone else's code. Also, not forgetting the "robust statisticians" who perform "correlation analysis", which is no more than comparing rolling averages to previously defined thresholds. I'm great with a carving knife. You should see me cut a turkey during Thanksgiving. However, I would never tell anyone that I'm an expert surgeon.
Co-Founder of Hiflylabs; Championing the Modern Data Stack with decades of experience in data consulting
8 年Tom, you miss mentioning the option of external providers. There is also an excuse for this saying “consultants are going to be too expensive for me”. Given the high salaries of potential candidates and the months/years needed to build a performing internal staff, ready-to-go external teams may prove a cost effective alternative. They should arrive with a motivated and creative team and also with relevant cross-industrial experience. You even have the option to evaluate them in a competitive way. Pick a few service providers, give them the same problem and let them go for the gold.
Data Systems Development at Moodys
8 年In my experience, the amount of low hanging fruit laying around a given company is truly staggering. The 80/20 rule is alive and well, and the skill to exploit it isn't as expensive or difficult as senior executives might fear. 1) Hire a few wizards; pay them well. 2) Give them the tools they need. 3) Give them the access to as much business knowledge as you can. 4) Give them a seat at the table. 5) Profit. A $500,000 investment (team, tools, & time) can net you millions in efficiency enhancements within a quarter or two.