Building a "BIG DATA" Software Team
Building any software team can be a challenge. Building a team for software that solves "Big Data" problems can be especially challenging. In my experience, the biggest challenge is the number of different tools, databases, techniques, and other related technologies. Hadoop, Hana, ML, various modeling languages, visualization tools, data refineries, key-value stores, and more, are incredibly powerful technologies used to solve and present complex problems often involving massive amounts of data. However, many of these are specific to the Big Data context and they are not a significant part of a Computer Science curriculum nor are they widely used by software engineers in general.
When recruiting, an obvious choice becomes quickly apparent. Do I recruit a more traditional software engineer who has limited exposure to Big Data technologies, or do I recruit someone who has significant expertise but likely is not a traditional software engineer? Answering this question is not easy because many Big Data projects are a combination of traditional software with Big Data technology in such a way that specific expertise is critical, Hadoop for example. While one can understand the basics of Hadoop and MapReduce fairly easily, the expertise required to architect a cluster, and the software the leverages that cluster for production use, is not easily obtained.
At Cognilytics we've taken the approach of hiring traditional software engineers, primarily because we already have a significant amount of expertise on all things Big Data. We find it's easier to bring a software engineer up-to-speed on the relevant Big Data elements rather than to search endlessly for the Unicorn who knows everything. For us, senior leadership is critical for establishing higher level architecture so that the software engineers can focus on implementation rather than on second guessing the endless permutations of available technologies.
If your company doesn't have Big Data expertise in house like we do, I suggest two things. First, limit the number of technologies that you use to only what is required for your problem. Do you really need MapReduce? If not, why are you using Hadoop? Second, hire someone who knows the landscape fairly comprehensively and can help establish that critical high level architecture early on and then find experts for each piece of that architecture that you don't have in-house people for. Ok, there's a third option. You don't have to bother with any of this, just work with a company who has done it many times, like Cognilytics. ;)
Art Messal, Manager of Engineering, Cognilytics Inc.
I seriously apologize for how many times I used the buzzword, BIG DATA.
Polymath, U.S. Foreign Policy & National Security Strategist, Policy Analyst, Intelligence & Foreign Affairs Advisor, Writer, Master’s degree
10 年Great post Art. In early 2013, I started looking to bring together an analytics team - relying on various tools, methods, and techniques that us intelligence analyst's used in our art of practicing counterintelligence, counterterrorism, political/military intelligence. I always saw the most powerful tool not the databases we used, but the human mind that was more powerful than any machine built. However, at the time, there wasn't enough software engineers that I needed to help bring the main part of the technology into the design I was trying to develop. You software engineers are geniuses and definitely needed for any big data development team, or enterprise.
Retired! Unless someone needs a Firmware Engineer Guru .. always looking for a true challenge.
10 年Sounds right to hire software engineers. Too many chefs and not enough cooks only leads to half-baked solutions.