Big Data Pipeline Implementation Use Case
With the Evolution of Digital Footprints of Organisations, the classic Data ware House Architecture is changing into a Big Data Pipeline based Architecture which is able to Utilise IOT, ML, AI, BigData and Traditional data in a seamless manner. Our team has shown how we have developed Big Data Pipeline Architecture based platform that can handle a very large set of streaming data. In an actual deployment case study, the data Streams through Kafka layer are moved into a Data Lake (in HBase, Hadoop, Cassandra or Azure or AWS) as Transient Data.
The data is passed through Data Transformation layer and saved into Raw Zone of Data Lake. The faulty data can be treated in the Data Unification layers and entire cleaned data after the Compute Zones can be pushed into the Trusted Zone of Data Lake. Compute Layers have our Predictive Tool that has integration with R, Scala Custom Code, Shell Scripts etc & integration with Spark ML. The Trusted data from Data Lake can be extracted, Transformed and Loaded into our Elastic Cube and Cassandra to give the top end Visualisation in the form of Reports, Self Service BI, Dashboards, Infographic based Advanced Analytics Dashboards etc. Having implemented this (at Beta Level) for an Enterprise customer with 50 Million active users, we are now working on Banking and Financial Sector, Agri Data Science, WorkForce and Health Care Case studies. For more details please click the following link
- https://bdbizviz.com/whitepaperform.html?id=3
Scalability, Benefits & ROI of the above Solution
- BDB system works in one platform/integrated ecosystem on Cloud rather than working in SILOS like other vendors. There is no need for any other product with BDB Platform, it comes with Big Data Pipeline, ETL, Query Services, Predictive Tool with Machine Learning, Survey Platform, Self Service BI, Advanced Visualization and Dashboards
- Provided real time attendance & assessment reporting to its entire 40 million active user base – Teachers, Students, Administrators etc.
- Substantial Reduction in Cost of Operation
- Saved 2 years of time and multi - million Dollars of effort to create an Analytics platform.
- Creation of New Market Opportunity - Instead of selling others licensing package, customer is able to sell their own Subscription Services and Licenses – Market opportunity 10x of investment Additional Analytics Services revenue with BDB as Implementation team
Big Data BizViz (www.bdbizviz.com) provides end to end Big Data Analytics and Visualisation to real time or near Real Time data coming from different sources.