Resource Allocation is an important aspect during the execution of any spark job. If not configured correctly, a spark job can consume entire cluster resources and make other applications starve for resources. This blog helps to understand the basic flow in a Spark Application and then how to configure the number of executors, memory settings of each executors and the number of cores for a Spark Job. There are a few factors that we need to consider to decide the optimum numbers for the above three, like: The amount of data The time in which a job has to complete Static or dynamic allocation of resources Upstream or downstream application Introduction Let’s start with some basic definitions of the terms used in handling Spark applications. Partitions : A partition is a small chunk of a large distributed data set. Spark manages data using partitions that helps parallelize data processing with minimal data…
Understanding Resource Allocation configurations for a Spark application
