Skip to content

Instantly share code, notes, and snippets.

@ehrnst
Created January 25, 2022 15:03
Show Gist options
  • Save ehrnst/bb63af04541060af7bdde66f926ac849 to your computer and use it in GitHub Desktop.
Save ehrnst/bb63af04541060af7bdde66f926ac849 to your computer and use it in GitHub Desktop.
AzCopy to move files in directory structure in Azure blob storage
## connect to storage using SAS
$storageName = ""
$sasToken = ""
$container = ""
$ctx = New-AzureStorageContext -StorageAccountName $storageName -SasToken $sasToken
# get all the blobs
$blobs = Get-AzureStorageBlob -Container $container -Context $ctx
# a date to filter on
$date = (get-date -Date 20.01.2022 -AsUTC)
# filter the blobs for date and where name has /c:/..
$blobsToModify = $blobs | where { ($_.LastModified.DateTime -ge $date) -and ($_.LastModified.DateTime -le $date.AddHours(24)) -and ($_.Name -like "*/chunks/C:/Users/*") }
# loop through the blobs
# get the original folder name and the blob name with some splitting
foreach ($blob in $blobsToModify) {
$blobtoMove = $blob.name
$original = $blob.Name.split("/",2)[0] # trim to original name
$newBlob = $blob.Name.split("/")[-1] # trim to original chunk name
# actually copy
./azcopy.exe copy "https://$storageName.blob.core.windows.net/thanos/$blobToMove$sasToken" "https://$storageName.blob.core.windows.net/thanos/$original/chunks/$newBlob$sasToken" --overwrite=prompt --s2s-preserve-access-tier=false --include-directory-stub=false --recursive --log-level=INFO;
}
# antother loop to delete the whole c:/ folder after the chunks of data is moved
# i have a separate loop as there might be multiple chunks in the folder.
foreach ($blob in $blobsToModify) {
$original = $blob.Name.split("/",2)[0] # trim to original name
./azcopy.exe remove "https://$storageName.blob.core.windows.net/thanos/$original/chunks/C%3A/$sasToken" --from-to=BlobTrash --recursive --log-level=INFO;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment