How Big the BigData Problem is ? How Companies overcome this Problem ..

Hello Connection !!

Here is my article on How the BigData is a great problem for Every Company...

Have you ever seen one of the videos on Facebook that shows a “flashback” of posts, likes, or images—like the ones you might see on your birthday or on the anniversary of becoming friends with someone? If so, you have seen examples of how Facebook uses Big Data.This how Data is Business for Company..

• This is the world of Technological Advancements with Internet.In 2020, it's estimated that for every person on earth, 1.7 MB of data will be created every second

BIG DATA :

Big data is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software. Big data refers to the massive volume of information that is difficult to be processed using traditional database techniques. It can be either structured or unstructured, and both kinds have innumerable benefits. Big data holds tremendous potential for improving our lives. It can create connections, recognize patterns, and finds use in many innovations like self-driven cars or customizing, developing, improving medical treatments, and accurate weather predictions.

• "Big data" refers to data sets that are too large or too complex for traditional data processing applications
• 1.7MB of data is created every second by every person during 2020.
• 2.5 quintillion bytes of data are produced by humans every day.

• 463 exabytes of data will be generated each day by humans as of 2025.
• 95 million photos and videos are shared every day on Instagram.
• By the end of 2020, 44 zettabytes will make up the entire digital universe.
• Every day, 306.4 billion emails are sent, and 5 million Tweets are made.

Characteristics Of Big Data

Volume

Volume is absolutely a slice of the bigger pie of Big data. The internet-mobile cycle, delivering with it a torrent of social media updates, sensor data from tools and an outburst of e-commerce, means that all industry swamped with data, which can be amazingly valuable if you understand how to work on it.

Velocity

Each minute of every day, users throughout the globe upload 200 hours of video on Youtube, send 300,000 tweets and carry over 200 million emails. And this keeps growing as the internet speed is getting faster.

?In One Second :

• 85,822 Google searches
• 9139 tweets
• 1017 Instragram Photos
• 2,949,441 Emails

Source :

Here is the link to visualise the BigData Problem.

Variety :

The are mainly two variate of data generated

1. Structured Data
2. Unstructured Data

• The amount of Structured data decreasing year by year which make a huge amount of increase in Unstructured/ Semi structured data.

How companies like Facebook and Google Mange BigData ?? ??

The two problem of the Bigdata of volume and velocity are solved by Distributed Computing. We can use Hadoop as a product to create a Cluster that can solve the problem of Volume and Velocity

They rely too much on one technology, like Hadoop. Facebook relies on a massive installation of Hadoop software, which is a highly scalable open-source framework that uses bundles of low-cost servers to solve problems. The company even designs its in-house hardware for this purpose.

• Mr. Rudin says, “The analytic process at Facebook begins with a 300 petabyte data analysis warehouse. To answer a specific query, data is often pulled out of the warehouse and placed into a table so that it can be studied. The team also built a search engine that indexes data in the warehouse. These are just some of the many technologies that Facebook uses to manage and analyze information.”

How Google Solved Big Data Problem?

This problem tickled google first due to their search engine data, which exploded with the revolution of the internet industry. And it is very hard to get any proof of it that its internet industry. They smartly resolved this difficulty using the theory of parallel processing. They designed an algorithm called MapReduce. This algorithm distributes the task into small pieces and assigns those pieces to many computers joined over the network, and assembles all the events to form the last event dataset.

They are using Hadoop,Hbase,Sqoop,Flume,SPARK,Hadoop,MapReduce,Pig,Impala,Hive,Cloudera Search,Oozie,Hue to solve the problem of BigData..

?

Done this task under guidance of Mr.Vimal Daga sir in ARTH

Thankyou sir for this task I learned a lot of thing about BigData in the process of writing this article.

Thanks for Reading !! Open For suggestions