What is MapReduce?

MapReduce is a processing method and a Java-based distribution model for distributed computing. The MapReduce algorithm includes two important tasks of Map and Reduce. Map takes a dataset and converts it to another set of data, where single elements are divided into tuples (key/value pairs). Reduce the tasks that receive the output from Map as input and transform tuples into a smaller set of ones. As their name suggests, the Reduce task is always done after the map.

The major advantage of MapReduce is that data processing is easy at more than one computing node. Under the MapReduce model, initial data processing is called mappers and reducers. Dividing a data processing program into mappers and reducers is sometimes important. However, when we wrote an application in the MapReduce form, an application to run on more than hundreds, thousands, or even tens of thousands of devices in a cluster, is merely a configuration change. This scalability is something that many programmers have been attracted to using the MapReduce model.

The Algorithm

*In general, the MapReduce paradigm is based on the submission to the computer where the data is located.

*The MapReduce program runs in three stages, called map stage, shuffle stage, and reduce stage.

** map stage - map or mapper’s job is the processing of input data. In general, the input information is file or directory and is stored in the Hadoop file system (HDFS). The input file is transmitted to the mapper function line by line. Mapper processes data and creates multiple small pieces of data.

**reduce stage - This stage is a combination of the shuffle stage and reduce stage. The Reducer job is processing data that comes from the mapper. After processing, a new output set is generated that is stored in HDFS.

* During the work of MapReduce , the Hadoop sends the map and reduce tasks to the appropriate servers in the cluster.

* This framework manages all the details of the data passing, such as the issuing tasks, verifying task  completion and the copying of data in a cluster and between nodes.

* Most computational operations in nodes are done with data on local disks that reduce network traffic.

* Upon completion of the tasks, the cluster attempts to collect and reduce the data in order to produce an appropriate result and send it to the Hadoop server.