Spark

Spark

Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.[2] The Dataframe API was released as an abstraction on top of the RDD, followed by the Dataset API. In Spark 1.x, the RDD was the primary application programming interface (API), but as of Spark 2.x use of the Dataset API is encouraged[3] even though the RDD API is not deprecated.[4][5] The RDD technology still underlies the Dataset API.[6][7]

Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.[8]

Inside Apache Spark the workflow is managed as a directed acyclic graph (DAG). Nodes represent RDDs while edges represent the operations on the RDDs.

Spark facilitates the implementation of both iterative algorithms, which visit their data set multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications may be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementation.[2][9] Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark.[10]

Apache Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone (native Spark cluster, where you can launch a cluster either manually or use the launch scripts provided by the install package. It is also possible to run these daemons on a single machine for testing), Hadoop YARN, Apache Mesos or Kubernetes.[11] For distributed storage, Spark can interface with a wide variety, including Alluxio, Hadoop Distributed File System (HDFS),[12] MapR File System (MapR-FS),[13] Cassandra,[14] OpenStack Swift, Amazon S3, Kudu, Lustre file system,[15] or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.

要查看或添加评论,请登录

Vanshika Munshi的更多文章

  • Key Data Engineer Skills and Responsibilities

    Key Data Engineer Skills and Responsibilities

    Over time, there has been a significant transformation in the realm of data and its associated domains. Initially, the…

  • What Is Financial Planning? Definition, Meaning and Purpose

    What Is Financial Planning? Definition, Meaning and Purpose

    Financial planning is the process of taking a comprehensive look at your financial situation and building a specific…

  • What is Power BI?

    What is Power BI?

    The parts of Power BI Power BI consists of several elements that all work together, starting with these three basics: A…

  • Abinitio Graphs

    Abinitio Graphs

    Graph Concept Graph : A graph is a data flow diagram that defines the various processing stages of a task and the…

  • Abinitio Interview Questions

    Abinitio Interview Questions

    1. What is Ab Initio? Ab Initio is a robust data processing and analysis tool used for ETL (Extract, Transform, Load)…

  • Big Query

    Big Query

    BigQuery is a managed, serverless data warehouse product by Google, offering scalable analysis over large quantities of…

  • Responsibilities of Abinitio Developer

    Responsibilities of Abinitio Developer

    Job Description Project Role : Application Developer Project Role Description : Design, build and configure…

  • Abinitio Developer

    Abinitio Developer

    Responsibilities Monitor and Support existing production data pipelines developed in AB Initio Analysis of highly…

  • Data Engineer

    Data Engineer

    Data engineering is the practice of designing and building systems for collecting, storing, and analysing data at…

  • Pyspark

    Pyspark

    What is PySpark? Apache Spark is written in Scala programming language. PySpark has been released in order to support…

社区洞察

其他会员也浏览了