Last active
October 4, 2018 18:54
-
-
Save sumanentc/8f24f2add6e7656c3d39fa0bd3bc2619 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Property Name | Default | Meaning | |
|---|---|---|---|
| spark.dynamicAllocation.enabled | FALSE | Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload | |
| spark.dynamicAllocation.executorIdleTimeout | 60s | If dynamic allocation is enabled and an executor has been idle for more than this duration, the executor will be removed | |
| spark.dynamicAllocation.initialExecutors | minExecutors | Initial number of executors to run if dynamic allocation is enabled.Ê | |
| spark.dynamicAllocation.minExecutors | 0 | Lower bound for the number of executors if dynamic allocation is enabled. | |
| spark.dynamicAllocation.schedulerBacklogTimeout | 1s | If dynamic allocation is enabled and there have been pending tasks backlogged for more than this duration, new executors will be requested. | |
| spark.dynamicAllocation.sustainedSchedulerBacklogTimeout | schedulerBacklogTimeout | Same asÊspark.dynamicAllocation.schedulerBacklogTimeout, but used only for subsequent executor requests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment