Data Engineers to Data Scientists to Chief Analytics Officer: Creating a Successful team

In the last few weeks and days, while I have been interacting with a lot of individuals interested in Data Science, some questions that are asked have a similar theme. So, I thought I spend some time to explain some of the regular questions:

ONE: How important is the Data Engineer’s role in supporting the Data science teams?

• Data Engineers have always been there. Actually, they are part of the team whose role is very critical in accessing the data, manipulating it and getting ready for the data scientists to apply the science to it. Though the data scientists usually have the skill to access the data, but it is an expensive proposition for data scientists to do so.
• Data Engineers are true IT people who can very quickly access all kinds of data, whether structured, semi-structured or unstructured data and are able to create automation around data flows. They make sure that the platform is available and the data is there for the data scientists when they require the data. Infact, when I was working at one large retailer, I hired a very experienced data engineering team who not only had a very good understanding of the data but was responsible for the entire platform. Their expertise ranged variably from UNIX, R, SAS, SQL, noSQL, Hive, etc. to knowledge of scraping data and storing it on regular basis. This is indeed a very important role.

The Venn diagram below explains the clear differentiation in skills of Data Scientist and a Data Engineer:

On the other hand, Data scientists consume that data, run statistical models on them and then create a value story to be consumed by business. Data Scientists have a very good understanding of business and a bit of technology. The most important skill a data scientist has is the ability to convert analysis, outcome of a statistical model into plain English, which can then become an integral part of the business execution plan.

TWO: There is an emergence in new Data service offerings in the market, what are these offerings and how are they being used?

Due to the availability of big data technology in the cloud environment, a lot of companies have seen potential in creating a large platform and store large volumes of public data available readily in a format which can be accessed easily. The platform is already installed with software packages that their clients can use by bringing their private data and process it securely. There is a big market for such companies as they reduce a lot of overhead for clients who do not what to make large investments upfront and want to outsource.

Below is a basic representation of how does the platform of such Data Service Offering looks like:

The above environment gives various organisations access to large volumes of data, analytics software and templates based on the experience from the industry.

Large number of organisations are using such platform for many reasons which mostly solve their reporting needs, e.g.:

1. Visualisation
2. Correlation of data with external sources
3. Dashboard reporting

Unfortunately, these are very IT focused solution which only solving IT problems. Business problems are still left untouched as it requires business knowledge to be embedded into the solution, which is hard because that is not the focus.

 "The service which is able to provide business solutions incorporating the business knowledge will be the shining star in this space and will get ahead of all the other service providers by miles."

THREE: What is the importance and role of Chief Analytics Officer?

The biggest challenge is that the Senior Leadership in an organisation does not understand the value that can be delivered. As we know that businesses work on profit and loss, so when the leadership does not see the value, it becomes difficult for them to justify funding. There are three components to this:

• I usually say, “Nothing succeeds like Success” and the most important role of the Chief Analytics Officer is to demonstrate and show value and success to take the leadership on a journey. In organisations, where Chief Analytics Officer is able to demonstrate value, the question of funding never arises. It is no more a cost center but a profit center. The Chief Analytics Officer is usually a story teller who understands the art and science of analytics and showcases the value created through the work done by the team.
• Secondly, implementing these projects successfully is the key which requires identifying the right resources and talent for these projects and that only comes with experience. Chief Analytics Officers have to be hands on, they should be able to ask the right questions while recruiting, hence, hire the right talent leading to successful implementations.
• Finally, taking the business teams along with you, engaging with them at the very beginning of the project, so they execute with full confidence. Change management is not even considered in this domain but it is one of the most important component for a successful delivery.

Chief Data Officer is NOT a Chief Analytics Officer; They are two different roles, require different skills and have different responsibilities!

The key to success is to get the right mix of leadership and resources, organisations that have got it right are far ahead of their competition.

Please follow me on Twitter, Linkedin or my blog Cognitive Today where you can find many articles on Data Science, Deep Learning, Machine Learning and Advanced Analytics.