7 recommendations for big data management

Enough has been said and written about Big Data – the technologies, use cases, infrastructure, tools, storage, platforms etc.  As the Big Data slips off the perch – top of the hype curve -the big data space will witness more successes (use cases) and less clutter (technologies and tools).  We will see more maturity in the implementation of the ROI inducing use cases across industries and sectors.  This, coupled with consolidated and more standardized (de-cluttered) tools and technologies, will normalize the inflated expectations, which are common at the top of the hype-curve, to more realistic potentials.  As more organizations begin to realize big data ROI, they will attract more organizations to join the band-wagon.  In all this movement, the data (structured, unstructured and semi-structured) will continue to get generated and captured by organizations at faster pace.  Many organizations are already past the 100 terabytes of data threshold and pushing deeper into petabytes territory.  This makes strategic management of big data paramount. The architecture supporting big data management has the potential to enable or inhibit future extreme scalability, build complete business views, permit unforeseen forms of analytics, establish big data as an enterprise asset, catalyze real-time operation, create competitive business value etc.  Soon, it will be also be critical to build skills, teams and best practices to manage such big data architecture and platforms along with traditional datawarehouse technologies in a single multi-platform infrastructure. 

 I am working on a separate blog on the multi-platform infrastructure but, here, I will summarize my top 7 steps for managing big data.

1.Build big data strategy to extract business value: The vast majority of businesses agree that big data is an excellent opportunity to obtain customer insights, monitor business operations in real-time, predict business outcomes and so forth. However, it is so only if you seize this opportunity.  

I believe that a three-step big data strategy will allow organizations to exploit the available opportunity – a) Use big data to deep-dive for advanced analytics.  b)  Integrate new big data with legacy enterprise data to extend views of customers and other business entities. c) Enable ability to capture, ingest,  analyse and enlighten in real time business processes based on streaming big data.  

Needless to say the big data strategy will exist to support the business strategy and not vice versa.  Further, I imagine that significant proportion of organizations with big data dreams must already be armed with a semblance of this strategy.  If not, it is time to do so.

2.Plan for ALL formats of big data: As your big data strategy goes through the three-steps mentioned above, the organization’s use and demand of different data format evolves and matures.  Anticipate the business to grow from using relational data to move on to other structured data, such as network files or log files. Subsequently XML messaging data will need to be stored as semi-structured data to be followed by unstructured data, such as PDF files or weather reports.  In order to cope up with these diverse data format, select relevant data platforms that can support such demands.

3. Roadmap from legacy data platforms to new data platforms: Clearly, with multiple data formats to support, big data management is definitely a multi-platform solution. As such, it would be prudent to create a road-map from the current portfolio of data platforms and data management tools to the future state.  This step will create a unified and joined-up big data infrastructure.

4. Technology strategy to enable big data strategy: On the back of the big data strategy / roadmap, spell out an enterprise-wide technology strategy for big data management. Some elements that will go into such a strategy would be road maps, preferences, governance etc . In the technology strategy document you could detail out a road map for maturing from structured to semi-structured to unstructured data formats – the three-step big data strategy.  Since the idea should be to avoid siloed data repositories, the technology strategy should layout preferred platforms and interfaces for capturing, storing, and processing each data format.  These platforms must co-exist as a single technology stack rather than as individual technologies, and thus, the technology strategy must also address the integration roadmap for these platforms.  Finally, the strategy must also anticipate and address the capacity, storage and scalability challenges of the future. 

5. Ensure trained staff: Data exploration, advanced analytics, real-time monitoring, multi-platform DW architecture etc are relatively recent concepts and technologies that require new or, atleast, enhanced   It is critical for the organizations to hire relevant resources or train existing resources to adequate level.  Any corners-cut here could result in, at best, delayed delivery of the projects or, at worse, failed projects. 

6. Strong collaboration amongst the BDM organization: For big data to be fully utilized, it must be accessed and leveraged by multiple business units and users.  Without doubt, in the future BDM will consist of diverse technology teams which will need to work collaboratively and efficiently to reach out to all the big data stakeholders.  It would make sense to have a conductor to orchestrate the music by all the technology teams to create music – a single owner to oversee the activities by these diverse teams.

7. Governance – One lesson from the data management days is the proliferation of siloed data marts. In order to allow for smoother integration of the big data into the EDW and the BI/DW environments, either appoint a bu owner for each of the big data platforms or a central IT that is responsible for shared enterprise infrastructure.  A clear governance model will aid data completeness, quality and integrity.