Skip to content

Instantly share code, notes, and snippets.

@peterdietz

peterdietz/s3-migrate.log

Created October 13, 2015 20:30
Show Gist options
  • Save peterdietz/585ba3a6859a5d2bd8d8 to your computer and use it in GitHub Desktop.
Save peterdietz/585ba3a6859a5d2bd8d8 to your computer and use it in GitHub Desktop.
Download ZIP
DSpace bitstore-migrate from local filesystem to AWS S3
Raw
s3-migrate.log
peterdietz:assetstore-20151013-again peterdietz$ /dspace/bin/dspace bitstore-migrate -h
usage: Migrate
-a,--source <arg> Source assetstore (to lose content)
-b,--destination <arg> Destination assetstore (to gain content)
-h,--help Help
-k,--keep Keep assets in source assetstore (don't delete
it).
-p,--print Print out current assetstore information
-s,--size <arg> Batch commit size
peterdietz:assetstore-20151013-again peterdietz$ /dspace/bin/dspace bitstore-migrate -p
store[0] == DSBitStore, which has 284 # of bitstreams.
store[1] == S3BitStore, which has 0 # of bitstreams.
Incoming assetstore is store[1]
peterdietz:assetstore-20151013-again peterdietz$ /dspace/bin/dspace bitstore-migrate -a 0 -b 1
peterdietz:assetstore-20151013-again peterdietz$ /dspace/bin/dspace bitstore-migrate -p
store[0] == DSBitStore, which has 0 # of bitstreams.
store[1] == S3BitStore, which has 284 # of bitstreams.
Incoming assetstore is store[1]
@peterdietz
Copy link
Author

Behind the scenes were logs such as:
2015-10-13 16:15:04,457 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=323
2015-10-13 16:15:05,294 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=386
2015-10-13 16:15:12,828 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=324
2015-10-13 16:15:21,266 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=428
2015-10-13 16:15:21,427 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=325
2015-10-13 16:15:21,792 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=387
2015-10-13 16:15:21,794 INFO org.dspace.storage.bitstore.BitstreamStorageManager @ Migration Commit Checkpoint: 200
2015-10-13 16:15:34,012 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=326
2015-10-13 16:15:34,204 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=327
2015-10-13 16:15:34,729 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=388
2015-10-13 16:15:58,995 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=328
2015-10-13 16:15:59,217 INFO org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=429

@peterdietz
Copy link
Author

Now, better logs. You can follow along, if you get bored during migration.

2015-10-14 01:00:14,690 INFO  org.dspace.storage.bitstore.BitstreamStorageManager @ Copying bitstream:324 from assetstore[2] to assetstore[1] Name:Dickmann2004.pdf, SizeBytes:1558508
2015-10-14 01:00:21,936 INFO  org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=324
2015-10-14 01:00:21,937 INFO  org.dspace.storage.bitstore.BitstreamStorageManager @ Removing bitstream:324 from assetstore[2]
2015-10-14 01:00:21,938 INFO  org.dspace.storage.bitstore.BitstreamStorageManager @ Migration Commit Checkpoint: 283
2015-10-14 01:00:21,939 INFO  org.dspace.storage.bitstore.BitstreamStorageManager @ Copying bitstream:428 from assetstore[2] to assetstore[1] Name:HonorsThesis.pdf, SizeBytes:1789281
2015-10-14 01:00:30,363 INFO  org.dspace.content.Bitstream @ anonymous::update_bitstream:bitstream_id=428
2015-10-14 01:00:30,364 INFO  org.dspace.storage.bitstore.BitstreamStorageManager @ Removing bitstream:428 from assetstore[2]
2015-10-14 01:00:30,365 INFO  org.dspace.storage.bitstore.BitstreamStorageManager @ Migration Commit Checkpoint: 284

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment