How to mirror ubuntu ESM protected repository

mirroring-ubuntu-ESM-repo.md

Via https://ubuntu.com/esm#faq

"We're mirroring the repository on our internal Landscape server. Can we still get Ubuntu ESM if using Landscape? ESM is just a regular Ubuntu archive, but authenticated and served over HTTPS. Archive mirroring is already available in Landscape and is the only supported mechanism for mirroring the ESM archive."

TL;DR

It seems its just possible to use URI with login:pass

# Extract login/pass
> cat /etc/apt/auth.conf.d/90ubuntu-advantage
> /usr/bin/aptly -config /etc/aptly.conf mirror create -architectures="amd64" -with-sources=false -with-udebs=false ubuntu-esm-trusty https://<login>:<pass>@esm.ubuntu.com/ubuntu trusty-updates
# Security repo
> aptly mirror show esm-ubuntu-trusty-security|grep packages
Number of packages: 841

# OR
> curl -v https://<login>:<pass>@esm.ubuntu.com/ubuntu

can you explain the login:pass?
how do you know the login name from the output of cat /etc/apt/auth.conf.d/90ubuntu-advantage

nm.. I see... different login:pass depending on the app repo. You and the other guys rock. Thanks for working this out... it allowed me to get my copy working.

For anyone who needs it spelled out, your 90ubuntu-advantage file will have something like:
machine esm.ubuntu.com/apps/ubuntu/ login bearer password mAgJOEW...[long URL-encoded string continues...]
so your username will be bearer and password will be that long URL-encoded string

try this

sudo pro attach yourtoken

go to https://ubuntu.com/pro/dashboard for get your personal token

ubuntuadvantagelong="$(cat /etc/apt/auth.conf.d/90ubuntu-advantage | grep esm.ubuntu.com/apps| sed 's|machine esm.ubuntu.com/apps/ubuntu/ login bearer password ||' | sed 's| # ubuntu-advantage-tools||')"

ubuntuadvantage=$(echo bearer:${ubuntuadvantagelong:0:146})

encode result $ubuntuadvantage to base64 in variable

use header

Authorization: Basic $ubuntuadvantageinbase64encode

work

We created a script exactly for this cause: https://github.com/APIVersa/rSyncOverHTTP

You can use it along with the following instructions to do exactly this and mirror the esm repos:

1. make sure you have a computer with esm enabled and the modules you want to mirror enabled
   (use the command pro status --all to check)
2. cd /etc/apt/auth.conf.d
3. sudo su
4. cat 90* | awk -F '#' '{print $1}' > ~/keys.txt
5. cd ~/
6. open the keys.txt file (maybe nano?)
7. you'll need to find the very last word from each line, that will be the password
8. the username is always bearer, so you can start to make a list with username:password with bearer:PASSWORD_HERE
9. in the same keys.txt file you'll see a "machine" line. That is the URL you need to mirror. You'll use it as follows: https://bearer:PASSWORD_HERE@machine_line_here/

• BE SURE TO INCLUDE THE FOLDER PATH AFTER THE DOMAIN!!!

10. Use our mirror script as follows now: rsyncoverhttp -s the_url_from_step_9 -d /path/to/main/directory/ -nr

Congrats, you should now have a working mirror. Be sure in step 10, supply for -d the MAIN DIRECTORY URL OF SERVER, NOT THE SUBDIRECTORY FOR THE SPECIFIC ESM MODULE.

Here is an example run after installing following instructions and using above instructions:
rsyncoverhttp -s https://bearer:INFRA_PASSWORD_HERE@esm.ubuntu.com/infra/ -d /var/www/esm/ -nr
rsyncoverhttp -s https://bearer:APP_PASSWORD_HERE@esm.ubuntu.com/apps/ -d /var/www/esm/ -nr

** NOTE ** YOU WILL HAVE A DIFFERENT PASSWORD FOR EACH MODULE / DIRECTORY ON esm.ubuntu.com

Thanks :)