Skip to content

Instantly share code, notes, and snippets.

@rkoster
Last active December 18, 2015 01:39
Show Gist options
  • Save rkoster/5705252 to your computer and use it in GitHub Desktop.
Save rkoster/5705252 to your computer and use it in GitHub Desktop.
Script for migrating blobs from one blob store to an other. This script expects a config/private.tmp.yml file which will be renamed to config/private.yml when blobs are downloaded. The contents for the new config/final.yml are hardcoded into the script. You should fill in your own target blobstore credentials
#!/bin/sh
# reset stuff
sudo rm -r .blobs
rm -r blobs
rm -r blobs_import
git checkout -- config/blobs.yml
git checkout -- config/final.yml
rm config/private.yml
# get blobs
bosh sync blobs
mv blobs blobs_import
# Format for private.tmp.yml:
# ---
# blobstore:
# s3:
# access_key_id: ACCESS
# secret_access_key: PRIVATE
# switch config to new blob store
cp config/private.tmp.yml config/private.yml
cat > config/final.yml <<EOF
---
final_name: cf-contrib
blobstore:
provider: s3
options:
bucket_name: cf-services-contrib-release
# AWS s3 user: blobstore-reader
access_key_id: AKIAJPWUHOHV42IMTNQQ
secret_access_key: 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
encryption_key: cloudfoundry-community
EOF
# clear blobs.yml
cat > config/blobs.yml <<EOF
---
{}
EOF
# re-add blobs
for blob in `cd blobs_import && find */*`
do
dir=`echo $blob | cut -d'/' -f1`
bosh add blob blobs_import/$blob $dir
done
# upload blobs
bosh upload blobs
@pmenglund
Copy link

The final.yml should not have the secrets, they should be in private.yml so they aren't committed to the repo. This is unfortunately not documented anywhere but in the code :(

@drnic
Copy link

drnic commented Jun 5, 2013

final.yml needs to have the read-access credentials though?

@drnic
Copy link

drnic commented Jun 5, 2013

The blobstore-reader credentials here only have read-access to the s3 account

@drnic
Copy link

drnic commented Jun 5, 2013

But it does mean that private.yml shouldn't need deleting & copying. You should be able to create a config/private.yml up front and just use that.

@grenzr
Copy link

grenzr commented Jul 29, 2013

Nice script - would be nice if you didnt have to type 'yes' for each blob to upload though.. there doesnt seem to be a switch to auto-yes each blob upload :/

Looks like TieWei commented that check out in his BOSH fork though :D
https://github.com/TieWei/bosh/commit/d5c3aa93f918da38a52094993d2ff37d7184d91f

@drnic
Copy link

drnic commented Nov 13, 2015

bosh -n upload blob stops the yes prompt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment