Building an All-Rounder Big Data Team

Talent wins games, but teamwork and intelligence win championships." --Michael Jordan

Alone we can do so little, together we can do so much." --Helen Keller

The importance of teamwork in any area can never be over-emphasized. In fact, it would not be incorrect to say that at times, the difference between success and failure is ….a great team.

Has the relevance of team work diminished in these days of advanced technology, automation, robotics and Big Data? I don’t think so. In fact, it’s the opposite. With advancement in these areas, team work becomes all the more important. Simply because

None of us is as smart as all of us." --Ken Blanchard

In fact, if you look closely, the architecture of Big Data is itself based on Team Work!!

Just having a team is not enough, an All-Rounder team is a must. So, what does it take to build an All-Rounder Big Data team? Not much really. Having successfully built many such teams over the last several years, I can confidently say that the following 5 principles are important –

i) Look for the ATTITUDE of the members above everything else – Big Data is relatively new and is an evolving field. Keeping up to date with the changing technology is not easy but a must. People who have a good open attitude and an inherent ability to adapt themselves, are the only ones who will go the long way. Technology experts who do not have the right attitude will not help the team much.

When I began with Hadoop, the ecosystem was confined to HDFS, MapReduce and a relatively smaller ecosystem like Hive, Pig, Hbase, Impala, Oozie and the likes. Today, there are several dozen Big Data projects (30+ projects waiting on the Apache platform, as I write this!). Frameworks like Spark are already replacing MapReduce! Without the right attitude, surviving and growing in this fast paced environment is a challenge.

ii) Have a GOOD MIX OF EXPERIENCE in the team – A general tendency is to look for senior, experienced members. Not a bad idea. However, juniors bring an entirely different perspective to a Big Data team. Their inherent ability to be on their toes and try new things adds great value. After all, a Big Data system is often very complex with several interfaces and learning by trial-and-error cannot be done away with.

I am a fan of the newbies in our team. Every single day is a learning experience – not only for them but also for me! The way they lap up new skills is simply amazing!

iii) Keep a SEPARATE BUDGET for skills enhancement – Relying on the common corporate budget is a no-no for Big Data teams. Skills need to be enhanced at a rapid rate and at times, the early movers gain a competitive advantage. It is essential to convince stakeholders and obtain a separate budget for skills enhancement. This includes not only software and training but also hardware.

Just a few months ago, we purchased 8 GB laptops for our team to try new stuff, only to realise that they barely suffice. We quickly had to ramp up hardware and also consider the cloud option.

iv) Do not shy away from REPLACING PEOPLE – I might sound preposterous but working on Big Data is not everybody’s cup of tea. People from different backgrounds jump into this area, sometimes because of the hype created; only to realise that the sailing isn’t smooth. It’s always a good idea to give team members a chance; in fact several of them. However, it is also essential to be realistic and assess whether a person is coping up well or not. If things are not working out with an individual, it makes sense to offer her/him an alternate area of choice (away from Big Data) and allow her/him to grow, rather than put in incessant efforts only to yield no results. Remember, time is of essence in the Big Data arena so it’s good to FAIL FAST (if at all) and look for alternatives.

v) DOMAIN KNOWLEDGE is important – Often, when talking of a Big Data team, the emphasis tends to veer towards the technical areas only. It needs to be kept in mind that any Big Data solution is a means-to-the-end so needs to be treated as such. Knowledge of the domain area will be essential to adopt the right approach to deliver a Big Data solution. Having an expert Business Analyst in the team will be the best thing to do.

We work across domains so adding appropriate Business Analysts is not always possible for us. However, as an alternative, we make sure that there are at least a few people in our team who invest time in understanding the domain. They might not be experts but it does help in a big way.

With these pointers, do you think you are ready to build a Big Data team? I would love to hear your views so please do share them. Happy Team Building!

About Me - I am the Big Data and Delivery Head at Ellicium Solutions Private Limited, a company focused on providing innovative solutions in Big Data and Analytics. Apart from architecting Big Data solutions, I enjoy sharing insights with fellow people.

For further details about our offerings, please visit our website www.ellicium.com