Amazon EMR (previously called Amazon Elastic MapReduce) is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark , on AWS to process and analyze vast amounts of data. Amazon EMR is based on Apache Hadoop, a Java-based programming framework that supports the processing of large data sets in a distributed computing environment. Using MapReduce, a core component of the Hadoop software framework, developers can write programs that process massive amounts of unstructured data across a distributed cluster of processors or standalone computers. It was developed by Google for indexing webpages and replaced its original indexing algorithms and heuristics in 2004.
Amazon EMR processes big data across a Hadoop cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3). The Elastic in EMR's name refers to its dynamic resizing ability, which enables administrators to increase or reduce resources, depending on their current needs. Amazon EMR is used for data analysis in log analysis, web indexing, data warehousing, machine learning (ML), financial analysis, scientific simulation and bioinformatics. It also supports workloads based on Apache Spark, Apache Hive, Presto and Apache HBase -- the latter of which integrates with Hive and Pig, which are open source data warehouse tools for Hadoop. Hive uses queries and analyzes data, and Pig offers a high-level mechanism for programming MapReduce jobs to be executed in Hadoop.
There are several ways enterprises can use Amazon EMR, including:
- Machine learning. EMR's built-in ML tools use the Hadoop framework to create a variety of algorithms to support decision-making, including decision trees, random forests, support-vector machines and logistic regression.
- Extract, transform and load. ETL is the process of moving data from one or more data stores to another. Data transformations -- such as sorting, aggregating and joining -- can be done using EMR.
- Clickstream analysis. Clickstream Data from Amazon S3 can be analyzed with Apache Spark and Apache Hive. Apache Spark is an open source data processing tool that can help make data easy to manage and analyze. Spark uses a framework that enables jobs to run across large clusters of computers and can process data in parallel. Apache Hive is a data warehouse infrastructure built on top of Hadoop that provides tools for working with data that Spark can analyze. Clickstream analysis can help organizations understand customer behaviors, find ways to improve a website layout, discover which keywords people are using in search engines and see which word combinations lead to sales.
- Real-time streaming. Users can analyze events using streaming data sources in real time with Apache Spark Streaming and Apache Flink. This enables streaming data pipelines to be created on EMR.
- Interactive analytics. EMR Notebooks are a managed service that provide a secure, scalable and reliable environment for data analytics. Using Jupyter Notebook -- an open-source web application data scientists can use to create and share live code and equations -- data can be prepared and visualized to perform interactive analytics.
- Genomics. Organizations can use EMR to process genomic data to make data processing and analysis scalable for industries including medicine and telecommunications.
As a cloud service, Amazon EMR can be deployed in a variety of settings, such as:
- Amazon EMR on Amazon EC2. Amazon EMR can quickly process large amounts of data using Amazon EC2. Users can configure Amazon EMR to take advantage of On-Demand, Reserved and Spot Instances.
- Amazon EMR on Amazon Elastic Kubernetes Service (EKS). The Amazon EMR console enables users to run Apache Spark applications with other applications on the same EKS cluster. Organizations can share compute and memory resources across all applications and use a Kubernetes tool to monitor and manage the infrastructure.
- Amazon EMR on AWS Outposts. AWS Outposts enables organizations to run EMR in their own data centers. This makes it easier to set up, deploy, manage and scale EMR in on-premises environments.
