Migrate Nearline Storage to Coldline storage for Arq Backup on Google Cloud

readme.md

###Change Storage class from nearline to coldline - Google Cloud for Arq Backup

• Upload the storage-class.json to your google cloud console command line (things older than 1 day)
• run gsutil defstorageclass get gs://{BUCKETNAME}
• run gsutil defstorageclass set COLDLINE gs://{BUCKETNAME}
• run gsutil lifecycle get gs://{BUCKETNAME}
• run gsutil lifecycle set storage-class.json gs://{BUCKETNAME}

bigdata-log-query.sql

SELECT 
  MIN(cast(time_micros as timestamp)) as mi,
  MAX(cast(time_micros as timestamp)) as ma,
  COUNT(*) as ct,
  cs_bucket,
  cs_user_agent
FROM [{PROJECTNAME}:storageanalysis.usage]
--WHERE cs_user_agent = 'GCS Lifecycle Management' -- if you only want Lifecycle events
GROUP BY cs_bucket,cs_user_agent
ORDER BY ma DESC

load-log-data.sh

#Setup instructions, create storageanalysis.usage
# https://cloud.google.com/storage/docs/access-logs
#Enable Logging:
#gsutil logging set on -b gs://{LOGBUCKET} gs://{ARQBUCKET}

# delete old bucket
bq rm -f storageanalysis.usage 
#load data for each bucket
bq load --skip_leading_rows=1 storageanalysis.usage \
          gs://{LOGBUCKET}/{ARQBUCKET}_usage* \
          ./cloud_storage_usage_schema_v0.json

storage-class.json

{
    "lifecycle": {
        "rule": [{
            "action": {
                "type": "SetStorageClass",
                "storageClass": "COLDLINE"
            },
            "condition": {
                "createdBefore": "2017-01-01",
                "matchesStorageClass": ["NEARLINE"]
            }
        }]
    }
}