As I am an avid photographer & travelling quite a lot, I wanted to sync my Adobe Lightroom library from Macbook to my Microsoft Windows desktop at home with very little/no human intervention. It is also a good thing that both Lightroom versions (mac & windows) share a common file system/structure, this helped alot in synchronizing data from one environment to another.
This HOWTO assumes you already have an AWS account created and running with an IAM user configured.
Put together, the solution would look as such (pardon my poor diagram skills):
_
macbook pro (` ). windows desktop
----------- --> internet --> ( amazon'`. --> internet --> ---------------
remote ( s3 ) home
` __.:'-'
For this tutorial/use-case, I am using:
- AWS Command Line Interface (CLI)
- AWS Identity and Access Management (IAM)
- Amazon Simple Storage Service (S3)
- Adobe Lightroom 6
- my trusty Macbook pro running Apple MacOs Yosemite (10.10.5)
- a dell optiplex 390 running Microsoft Windows 2012 Server R2
As of today, my small Adobe Lightroom library runs about 140gb using exclusively RAW Nikon format photos (.NEF). A quick check on my S3 Bucket:
~:nico bucketsize my-bucket
total size:
139.298 GB
total count:
37460 objects
You can find the source code for my bucketsize alias on my gist here: https://gist.github.com/anamorph/699dbae01ff0f4942acc#gistcomment-1484907
Open your Terminal App then paste these lines:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install python
sudo pip install --upgrade pip
sudo pip install awscli
aws configure
Follow the steps of this command to fill in the following information from your already existing IAM user:
- Access Key / Secret Key combination
- Default region setting
- Default output format setting
This will create the following configuration files in ~/.aws/ :
~:nico ls ~/.aws/
config credentials
This file contains the configuration details per profile as configured using step 2 as follows:
~:nico cat ~/.aws/config
[default]
output = json
region = eu-west-1
This file contains the default and additional profiles you have configured using using step 2, modulo a few switches as explained in the documentation:
~:nico cat ~/.aws/credentials
[default]
aws_access_key_id = YOURACCESSKEY
aws_secret_access_key = YOURSECRETKEY
By creating separate IAM accounts for the purpose of synchronizing our files from the mac to s3 and from s3 to the Microsoft Windows desktop, we limit the impact on our infrastructure in the event of these accounts being compromised to strictly access s3.
| Account | Function | Additional info |
|---|---|---|
| s3sync-mac | Synchronizes my photos from Mac to Amazon S3. | Secured with Virtual MFA. |
| s3sync-pc | Synchronizes my photos from Amazon S3 to PC. | Limited to a single IP using a /32 CIDR within a policy. |
To add an extra security layer, we attach a virtual MFA device to the s3sync-mac user account thus allowing secure authentication from any internet-enabled location.
For this section, references are as follows
- IAM Create User
- IAM Create Access Key
- IAM Virtual MFA Device
- IAM Enable Virtual MFA Device
- IAM List Virtual MFA Device
aws iam create-user --user-name s3sync-mac
aws iam create-user --user-name s3sync-pc
For each of these commands you should be getting answers in the following format:
{
"User": {
"UserName": "s3sync-mac",
"Path": "/",
"CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
"UserId": "SOMEUSERID",
"Arn": "arn:aws:iam::YOURACCOUNTID:user/s3sync-mac"
}
}
Copy the output of both commands and store it somewhere while we continue with the creation of the accounts.
We'll now create then enable the MFA device to the s3sync-mac user account:
aws iam create-virtual-mfa-device --virtual-mfa-device-name s3sync-mac --outfile /Users/nico/Desktop/s3sync-mac-QRCode.png --bootstrap-method QRCodePNG
Which will return:
{
"VirtualMFADevice": {
"SerialNumber": "arn:aws:iam::YOURACCOUNTID:mfa/s3sync-mac"
}
}
Let's enable this MFA to our s3sync-mac user account. We first need to scan the newly generated QRcode on our desktop with our TOTP security token application on our mobile phone. I use Google Authenticator. Once scanned, we'll get a first code, then a second one: write both of them down, we'll use them to validate our virtual MFA from the command line:
aws iam enable-mfa-device --user-name s3sync-mac --serial-number arn:aws:iam::YOURACCOUNTID:mfa/s3sync-mac --authentication-code-1 123456 --authentication-code-2 789012
No output for this one, but you can check your MFA device status using this command and easily check that everything is working accordingly:
{
"VirtualMFADevices": [
{
"SerialNumber": "arn:aws:iam::YOURACCOUNTID:mfa/s3sync-mac",
"EnableDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
"User": {
"UserName": "s3sync-mac",
"Path": "/",
"CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
"UserId": "SOMEUSERID",
"Arn": "arn:aws:iam::YOURACCOUNTID:user/s3sync-mac"
}
}
]
}
We'll now generate the Access Key / Secret Key pair, then configure a new profile on the CLI.
aws iam create-access-key --user-name s3sync-mac
Which will return:
{
"AccessKey": {
"UserName": "s3sync-mac",
"Status": "Active",
"CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
"SecretAccessKey": "MYSECRETKEY",
"AccessKeyId": "MYACCESSKEY"
}
}
Take the output of this command to create your new CLI profile:
~:nico aws configure --profile s3sync-mac
AWS Access Key ID [None]: MYSECRETKEY
AWS Secret Access Key [None]: MYACCESSKEY
Default region name [None]: eu-west-1
Default output format [None]: json
We now have a functional user account for our photos synchronization from the mac to our Amazon S3 Bucket, a corresponding profile set up with the CLI and added security via the use of a Virtual MFA token.
Our s3sync-pc user account has been created at the very beginning of this step 3.; we'll now generate a pair of Access Key/Secret Key and secure this account with an IAM policy.
aws iam create-access-key --user-name s3sync-pc
Which will return:
{
"AccessKey": {
"UserName": "s3sync-pc",
"Status": "Active",
"CreateDate": "YYYY-MM-DDTHH:MM:SS.XXXZ",
"SecretAccessKey": "MYSECRETKEY",
"AccessKeyId": "MYACCESSKEY"
}
}
Now, let's setup the CLI on our Microsoft Windows desktop:
Download then run the installation package on your Microsoft Windows desktop:
Once this is done, we can take the output of the previous command to create your new CLI profile on the Microsoft Windows desktop:
c:\aws configure --profile s3sync-pc
AWS Access Key ID [None]: MYSECRETKEY
AWS Secret Access Key [None]: MYACCESSKEY
Default region name [None]: eu-west-1
Default output format [None]: json
Now that our users are created and configured using profiles in the CLI, let's create our s3 bucket to store our photos. A quick reminder about Amazon s3:
- Amazon s3 will allow users to connect to your bucket if you give them access (via Amazon Identity and Access Management, Bucket Policies, etc ...)
- Since Amazon s3 is a online object storage service, your files will be (if given access to) accessible via the internet, using HTTP/HTTPS standards.
- Therefore your bucket name must be UNIVERSALLY UNIQUE; in short, pick a bucket name that isn't something amongst the lines of photos or myphotos, these names are most likely already taken.
The CLI will tell you so:
~:nico aws s3 mb s3://photos
make_bucket failed: s3://photos/ A client error (BucketAlreadyExists) occurred when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
Let's be creative:
~:nico aws s3 mb s3://my-lightroom-bucket
make_bucket: s3://my-lightroom-bucket/
Be sure to adapt the commands and the following bucket policy to reflect your bucket name.
Amazon Identity and Access Management (IAM) will help us in limiting the access to s3 for both our IAM users:
- we will only be able to use the s3sync-pc user to downoad photos from the s3 bucket from the Microsoft Windows desktop location, my home ip address.
- We will only be able to use the s3sync-mac user to upload photos to the s3 bucket.
For this purpose we will use the following policy:
{
"Id": "Policy1442860910448",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1442860826662",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-lightroom-bucket",
"Condition": {
"IpAddress": {
"aws:SourceIp": "123.123.123.123/32"
}
},
"Principal": {
"AWS": [
"arn:aws:iam::MYACCOUNTID:user/s3sync-pc"
]
}
},
{
"Sid": "Stmt1442860864410",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-lightroom-bucket/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "123.123.123.123/32"
}
},
"Principal": {
"AWS": [
"arn:aws:iam::MYACCOUNTID:user/s3sync-pc"
]
}
},
{
"Sid": "Stmt1442860885410",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-lightroom-bucket",
"Principal": {
"AWS": [
"arn:aws:iam::MYACCOUNTID:user/s3sync-mac"
]
}
},
{
"Sid": "Stmt1442860908897",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-lightroom-bucket/*",
"Principal": {
"AWS": [
"arn:aws:iam::MYACCOUNTID:user/s3sync-mac"
]
}
}
]
}
Why the policy has 2 parts for each user, with different bucket orthographs ? Well, let's explain this quickly:
s3:GetObject applies to the objects in the bucket:
"Resource": "arn:aws:s3:::my-lightroom-bucket/*"
while s3:ListBucket applies to the Bucket itself:
"Resource": "arn:aws:s3:::my-lightroom-bucket"
Now that we have that out of our way, we'll attach the policy to the bucket and test it (attach + put file + list bucket):
~:nico aws s3api put-bucket-policy --bucket my-lightroom-bucket --policy file://my-lightroom-policy.json
~:nico aws s3 cp my-lightroom-policy.json s3://my-lightroom-bucket --profile s3sync-mac
upload: ./my-lightroom-policy.json to s3://my-lightroom-bucket/my-lightroom-policy.json
~:nico aws s3 ls my-lightroom-bucket --profile s3sync-mac
2015-09-21 20:53:17 1500 my-lightroom-policy.json
For this section, references are as follows
Using the s3sync-mac account, we will now sync our Adobe Lightroom photo library to Amazon s3:
aws s3 sync /Volumes/Lightroom/lightroom/ s3://my-lightroom-bucket --profile s3sync-mac
Using Amazon s3's sync command, we'll push only the new files that don't exist on our Amazon s3 bucket. The new files will then be synched by my Microsoft Windows desktop at home.
For this section, references are as follows
To automate things, we'll use the following bach script to then create a scheduled task that will run every hour and synchronize the Adobe Lightroom library back on its hard drive. As a precaution, we'll log everything that happens into a unique logfile; so first things first, let's create our filetree:
c:\> mkdir c:\my-lightroom-sync\logs c:\my-lightroom-sync\bin\
Now that this is done, let's batch things up !
@echo off
goto :sync
***********************************************************************
id: my-lightroom-sync.bat
author: nicolas@nuage.ninja
version: 1.0
desc: This batch synchronizes from Amazon s3 downwards to the local disk
usage: my-lightroom-sync.bat (no arguments needed)
***********************************************************************
:sync
set AWS_CLI_PROFILE=s3sync-pc
set SYNC_REMOTE_BUCKET=my-lightroom-bucket
set SYNC_LOCAL_DIR=c:\Lightroom
set SYNC_LOGS=c:\my-lightroom-sync\logs
set SYNC_TIMESTAMP_DATE=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%
set SYNC_TIMESTAMP_TIME=%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
set SYNC_ERRLOG=%SYNC_LOGS%\my-lightroom-sync-%SYNC_TIMESTAMP_DATE%-%SYNC_TIMESTAMP_TIME%.log
@echo ### %SYNC_TIMESTAMP_TIME% - starting synchronization >> %SYNC_ERRLOG%
aws s3 sync s3://%SYNC_REMOTE_BUCKET% %SYNC_LOCAL_DIR% --profile %AWS_CLI_PROFILE% >> %SYNC_ERRLOG%
set SYNC_TIMESTAMP_END=%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
@echo ### %SYNC_TIMESTAMP_END% - ending synchronization >> %SYNC_ERRLOG%
Batch needs to be saved as c:\my-lightroom-sync\bin\my-lightroom-sync.bat using your favourite editor. Once this is done, you will need to run the first occurence manually in order to synchronize the initial library stored in Amazon s3. After that, you should be able to use the scheduled task:
C:\> schtasks /create /tn "my-lightroom-sync" /tr c:\my-lightroom-sync\bin\my-lightroom-sync.bat" /sc HOURLY /np
SUCCESS: The scheduled task "my-lightroom-sync" has successfully been created.
C:\> schtasks /query /TN my-lightroom-sync
Folder: \
TaskName Next Run Time Status
======================================== ====================== ===============
my-lightroom-sync 9/22/2015 1:06:00 AM Ready
The only thing you now need to worry about is to run the sync command from your mac when you have new content in your Adobe Lightroom library and let the scheduled task run the synchronization for you. Additionally, you might want to adapt this to your usage.
/Volumes/lightroom/Lightroom:nico aws s3 sync . s3://my-lightroom-bucket --profile s3sync-mac
upload: Photos/2015/2015-07-26/DSC_0517.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0517.NEF
upload: Photos/2015/2015-07-26/DSC_0515.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0515.NEF
upload: Photos/2015/2015-07-26/DSC_0516.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0516.NEF
upload: Photos/2015/2015-07-26/DSC_0514.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0514.NEF
upload: Photos/2015/2015-07-26/DSC_0518.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0518.NEF
upload: Photos/2015/2015-07-26/DSC_0520.NEF to s3://my-lightroom-bucket/Photos/2015/2015-07-26/DSC_0520.NEF
...
upload: Photos/2015/2015-08-09/DSC_0118.NEF to s3://my-lightroom-bucket/Photos/2015/2015-08-09/DSC_0118.NEF
Completed 493 part(s) with ... file(s) remaining
and this will also sync modified photos, right? (I'm not an expert of Lightroom)