Skip to content

Instantly share code, notes, and snippets.

View deepyaman's full-sized avatar

Deepyaman Datta deepyaman

78 followers · 0 following
View GitHub Profile
@deepyaman
deepyaman / catfact_with_catfact.py
Last active July 15, 2025 20:42
Visualize catfact data with catfact
# https://dlthub.com/docs/general-usage/http/overview
import dlt
from dlt.sources.helpers.rest_client import RESTClient
catfact_client = RESTClient(base_url="https://catfact.ninja")
@dlt.resource
def get_breeds():
for page in catfact_client.paginate(
@deepyaman
deepyaman / spark_configuration_changes.md
Created March 27, 2025 14:55
Spark configuration changes in Dagster 1.10.7 
2,11c2,4
< spark.blacklist.application.fetchFailure.enabled
< spark.blacklist.application.maxFailedExecutorsPerNode
< spark.blacklist.application.maxFailedTasksPerExecutor
< spark.blacklist.enabled
< spark.blacklist.killBlacklistedExecutors
< spark.blacklist.stage.maxFailedExecutorsPerNode
< spark.blacklist.stage.maxFailedTasksPerExecutor
< spark.blacklist.task.maxTaskAttemptsPerExecutor
@deepyaman
deepyaman / dataproc_configuration_changes.md
Created March 27, 2025 14:52
Dataproc configuration changes in Dagster 1.10.7

Cluster properties

0a1,2
> autoscalingConfig.policyUri
> auxiliaryNodeGroups
1a4
> dataprocMetricConfig.metrics
2a6,8
> encryptionConfig.kmsKey