?????????????? ???????????????????? ????????????????????????

?????????????? ???????????????????? ????????????????????????

Spark makes it easy to develop and create big data programs. Spark also makes it easy to turn your interactive exploration into production applications with ??????????-????????????, a built-in command-line tool. ??????????-???????????? does one thing: it lets you send your application code to a cluster and launch it to execute there. Upon submission, the application will run until it exits (completes the task) or encounters an error. You can do this with all of Spark’s support cluster managers including Standalone, Mesos, and YARN.

??????????-???????????? offers several controls with which you can specify the resources your application needs as well as how it should be run and its command-line arguments.

You can write applications in any of Spark’s supported languages and then submit them for execution. The simplest example is running an application on your local machine. We’ll show this by running a sample Scala application that comes with Spark, using the following command in the directory where you downloaded Spark:

??????????-???????????? \

--?????????? ?????????????????????? \ 

--???????????? ?????????? \ 

--????????????-???????? ???????????? \ 

--????????????????-???????????? ?????? \ 

--??????????-????????????????-?????????? ?????? \ 

./????????????????/????????/??????????-????????????????_??.????-??.??.??.??????         

This sample application calculates the digits of pi to a certain level of estimation. Here, we’ve told ??????????-???????????? that we want to run on our local machine, which class and which JAR we would like to run, and some command-line arguments for that class.

By changing the master argument of ??????????-????????????, we can also submit the same application to a cluster running Spark’s standalone cluster manager, Mesos or YARN. Take a look below:

??????????-???????????? \

--?????????? ?????????????????????? \ 

--???????????? ????.????.????.????:???????? \ 

--????????????-???????? ?????????????? \ 

--????????????????-???????????? ?????? \ 

--??????????-????????????????-?????????? ?????? \ 

./????????????????/????????/??????????-????????????????_??.????-??.??.??.??????         


Manoj Chandrashekar

UAB’24?? | Former Lead Data Engineer @7-Eleven | I Torture The Data, To Confess To Anything

2 年

要查看或添加评论,请登录

Manoj Chandrashekar的更多文章

  • End to End Pyspark Example

    End to End Pyspark Example

    We’ll use Spark to analyze some flight data from the United States Bureau of Transportation statistics. Inside the CSV…

  • The SparkSession

    The SparkSession

    You control your Spark Application through a driver process called the SparkSession. The SparkSession instance is the…

社区洞察

其他会员也浏览了