In release 1.3.3 Dagster introduced the ability for Dagster to display a link to compute logs instead of displaying the logs directly. This capability is important for Dagster Cloud users who do not want to send compute logs to Dagster Cloud, but still want end users to be able to access logs while debugging a run.
This capability is possible because of additions to the compute log manager. Users can implement their own compute log manager for full control over the link behavior or use the default dagster-aws implementation. The default implementation stores logs in s3 and displays a link to the log file:
Note in 1.3.3 the displayed link is directly to the s3 object. In 1.3.4 the displayed link is to the s3 console for the log object which provides a better experience for non-public s3 objects.
The way to enable the default dagster-aws option in Dagster Cloud varies by Hybrid agent. For the Kubernetes hybrid agent create a custom values.yaml file containing:
computeLogs:
enabled: true
custom:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
show_url_only: true
bucket: your-compute-log-storage-bucket
region: your-bucket-region
# additional options: https://artifacthub.io/packages/helm/dagster-cloud/dagster-cloud-agent?modal=values
To further customize the link behavior, implement the capture_logs context manager method on your own CloudStorageComputeManager subclass. The dagster-aws implementation can serve as an example.