Skip to content

Instantly share code, notes, and snippets.

View deepyaman's full-sized avatar

Deepyaman Datta deepyaman

62 followers · 0 following
View GitHub Profile
@deepyaman
deepyaman / spark_configuration_changes.md
Created March 27, 2025 14:55
Spark configuration changes in Dagster 1.10.7 
2,11c2,4
< spark.blacklist.application.fetchFailure.enabled
< spark.blacklist.application.maxFailedExecutorsPerNode
< spark.blacklist.application.maxFailedTasksPerExecutor
< spark.blacklist.enabled
< spark.blacklist.killBlacklistedExecutors
< spark.blacklist.stage.maxFailedExecutorsPerNode
< spark.blacklist.stage.maxFailedTasksPerExecutor
< spark.blacklist.task.maxTaskAttemptsPerExecutor
@deepyaman
deepyaman / dataproc_configuration_changes.md
Created March 27, 2025 14:52
Dataproc configuration changes in Dagster 1.10.7

Cluster properties

0a1,2
> autoscalingConfig.policyUri
> auxiliaryNodeGroups
1a4
> dataprocMetricConfig.metrics
2a6,8
> encryptionConfig.kmsKey