Skip to content

Instantly share code, notes, and snippets.

View DaveRuijter's full-sized avatar

Dave Ruijter DaveRuijter

49 followers · 1 following
View GitHub Profile
@DaveRuijter
DaveRuijter / pipeline-pullrequest-administration.yml
Created October 3, 2021 19:13
Azure DevOps YAML pipeline for Pull Request administration, with SemVer based on the PR description
trigger:
- none
pool:
vmImage: ubuntu-latest
## Job to calculate semantic version
jobs:
- job: CalculateVersion
displayName: Semantic versioning
@DaveRuijter
DaveRuijter / pipeline-release-administration.yml
Created October 3, 2021 19:47
Azure DevOps YAML pipeline for release administration, with automatic SemVer based on GitVersion and automatic release notes publication
trigger:
- main
- master
pool:
vmImage: ubuntu-latest
## Job to create release and add tag
jobs:
- job: CalculateVersion
@DaveRuijter
DaveRuijter / release-notes-template.md
Created October 3, 2021 20:23
Template file for automatic release notes

🚀 Release v{{buildDetails.buildNumber}}

Associated Pull Requests ({{pullRequests.length}})

{{#forEach pullRequests}}

  • Pull Request [#{{this.pullRequestId}} {{this.title}}]({{replace (replace this.url "_apis/git/repositories" "_git") "pullRequests" "pullRequest"}}) {{/forEach}}

Associated Azure Board Items ({{workItems.length}})

{{#forEach this.workItems}}

  • Item [#{{this.id}} {{lookup this.fields 'System.Title'}}]({{replace this.url "_apis/wit/workItems" "_workitems/edit"}})
@DaveRuijter
DaveRuijter / gitversion.yml
Created October 3, 2021 20:27
My GitVersion configuration for automatic SemVer
next-version: 1.0
assembly-versioning-scheme: MajorMinorPatch
assembly-file-versioning-scheme: MajorMinorPatchTag
assembly-informational-format: '{InformationalVersion}'
mode: ContinuousDelivery
increment: Inherit
continuous-delivery-fallback-tag: ci
tag-prefix: '[vV]'
major-version-bump-message: '\+semver:\s?(breaking|major)'
minor-version-bump-message: '\+semver:\s?(feature|minor)'
@DaveRuijter
DaveRuijter / vacuum_all_databricks_delta_tables.py
Created October 21, 2021 19:21
Script that iterates all tables in all databases that match the given database name filter, and performs the VACUUM command on each table.
database_names_filter = "20_silver*"
try:
dbs = spark.sql(f"SHOW DATABASES LIKE '{database_names_filter}'").select("databaseName").collect()
dbs = [(row.databaseName) for row in dbs]
for database_name in dbs:
print(f"Found database: {database_name}, performing actions on all its tables..")
tables = spark.sql(f"SHOW TABLES FROM `{database_name}`").select("tableName").collect()
tables = [(row.tableName) for row in tables]
for table_name in tables:
#spark.sql(f"ALTER TABLE `{database_name}`.`{table_name}` SET TBLPROPERTIES ('delta.logRetentionDuration'='interval 30 days', 'delta.deletedFileRetentionDuration'='interval 7 days')")
@DaveRuijter
DaveRuijter / backup_sta_lifecycle_policy_rules.json
Created October 21, 2021 19:27
This policy is used on the storage account that contains the backup copies of the Data Lake. This will apply retention of 60 days to the weekly backups, and retention of 30 days to the daily (incremental) backups.
{
"rules": [
{
"enabled": true,
"name": "weeklybackupsrule",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
@DaveRuijter
DaveRuijter / data_lake_sta_lifecycle_policy_rules.json
Created October 21, 2021 19:28
This policy is used on the storage account of the Data Lake. This will ensure new data in the dls/05_store/_archive folder of the lake is automatically assigned to the cool access tier.
{
"rules": [
{
"enabled": true,
"name": "daily-moving-data-lake-store-archive-to-cool",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"tierToCool": {
@DaveRuijter
DaveRuijter / backup-dls.ps1
Created October 21, 2021 20:16
This PowerShell script performs a copy between two storage accounts using AzCopy.
param(
[String]$sourceStorageAccount,
[String]$targetStorageAccount,
[String]$sourceFolder,
[String]$targetFolder,
[String]$sourceSasToken,
[String]$targetSasToken,
[String]$triggerPeriod,
[Int32]$azCopyConcurrency
)
@DaveRuijter
DaveRuijter / job-backup-dls.yml
Last active November 23, 2021 19:32
This YAML file is part of the Backup Strategy
parameters:
- name: backups
displayName: 'Array of backups'
type: object
default: []
- name: serviceConnectionName
displayName: 'Name of the DevOps Service Connection'
type: string
- name: execute
displayName: 'Execute this Job'
@DaveRuijter
DaveRuijter / stage-backup-dls.yml
Created October 21, 2021 20:28
This YAML file is part of the Data Lake Backup Strategy
parameters:
- name: dependsOnStage
type: string
- name: triggerPeriod
type: string
- name: environment
type: string
- name: backupStore
type: boolean
- name: backupBronze