Spark On YARN cluster, Some Observations

1.     Number of partitions in Spark

     Basic         => n Number of cores = n partitions = Number of executors

     Good        => 2-3 times of number of cores.

   So normally number of cores are directly proportional to partitions

2.     Spark Dynamic allocation very well performs on YARN and this way it ensures that it doesn’t over utilizes the memory but this scenario might create problem if you want to juice up everything from cluster and control resources.

3.     If Memory utilization is high but number of cores are free

If you have 100 GB YARN memory and minimum allocation is 1G , then ideally yarn has 100 slots. So if your min allocation is 4G, then yarn will have 25 slots. Each slots belongs to the capacity yarn can hold and process data.

Decrease the YARN container size yarn.scheduler.minimum-allocation-mb, make sure your cpu utilization is less than 80%

4.     Spark Memory

We will calculate based on 10gb of executor memory

Spark Memory is Divided in following parts –

a)     Spark Safety Region

Spark Safety region is meant for preventing OOM and ideally its 90% of total allocation (spark.storage.safetyFraction)

- 9gb is ideally usable spark memory

b)     Shuffle & Storage

Spark.storage.memoryFraction = 60% = 5.4gb

Spark.shuffle.memoryFraction = 20% = 1.8gb

So for no caching, simply make

spark.storage.memoryFraction = 0

Spark.shuffle.memoryFraction = 1

c)     Shuffle Safety Fraction

Spark.shuffke,safetyFraction = 80% of JVM heap

When we do reducebykey, reduce happens locally first and the shuffle memory is used for storing the intermittent result

5.     Spark Executor Memory Allocation

Map Friendly like ETL job recommendation

a)     The good number for executors, if you have multiple nodes are 4.

4 cores ~ 400%

5 cores ~ 425 – 435 %

So more than 4 cores may be not that efficient but this can be further optimized/tuned.

Entire observation is based on my experience working and facing number of challenges during the deployment in production. If you are working on Single Node, then it will be an entire different scenario & not a distributed environment so God Bless You :-)