Skip to content

Instantly share code, notes, and snippets.

@gurdiga

gurdiga/Makefile

Last active Jun 7, 2020
Embed
What would you like to do?
PhotoBucket export

PhotoBucket export

The other day I set out to export photos from a friend’s PhotoBucket account, and this is the code that came out.

Why a Makefile? — Whenever I start any coding project, Makefile is probably the first file that I create. Especially for a project like this, I run a lot of command-line commands, and I don’t want to remember them all, and so a Makefile comes in quite handy. This time, bit by bit, it ended up gathering a bunch of shell functions.

Summary

Here are the steps that got me here:

  • Opened the PhotoBucket account in the browser and inspected the network requests until I found the ones that load the album list.
  • From there, I just did Copy as cURL on the ones that were interesting, pasted them into my Makefile and tweaked to fit.
  • Once I got a way to get the list of albums, the next step is to get the list of photos for every of them. I put them all in a plain text file, one URL per line.
  • From there it’s easy: I just xargs them to wget and let it run.

Enjoy!

.ONESHELL:
MAKEFLAGS=
export CURL_PARAMS=--silent --connect-timeout 5 --retry-delay 0 --retry 5
start: list-albums # download-photos
list-albums:
@function main() {
list_albums "/albums/c111/SandraDodd"
}
function list_albums() {
local album_path="$$1"
if finished_listing "$$album_path"; then
stderr "Album already listed: $$album_path, skipping."
else
get_photos_for_album "$$album_path" >> photo-list.txt
echo ":: Finished $$album_path." >> photo-list.txt
fi
list_sub_albums "$$album_path" \
| while read count path; do
# escaped_album_path=`js_escape "$${path}"`
# url_count=`grep -P "$$escaped_album_path/[^/]+[^\.]$$" photo-list.txt | wc -l`
# if (( $$count != $$url_count )); then
# echo "========= Media count mismatch in $$path: expected $$count not $$url_count"
# fi
list_albums "$$path"
done
}
function get_photos_for_album() {
local album_path=$$1
stderr "Listing photos for album $$album_path:"
stderr "- page 1:"
retry curl $(CURL_PARAMS) \
'https://s26.photobucket.com/component/Common-PageCollection-Album-AlbumPageCollection?filters%5Balbum%5D='`js_escape "$${album_path}"`'&filters%5Balbum_content%5D=2&sort=2&limit=23&page=1&&linkerMode=true&json=1&hash=b85d3adadbb25a247969cbd262625379&_=1590247575048' \
-XGET \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'Pragma: no-cache' \
-H 'Cookie: _sfkcs2_6uljjjp7smmhu694vd1vk40rf2=TQaeWewbBVqOOW%2FKIpVpHzhghsWgEYcj9n%2BK2iVetiEd0OpwQ5VEunt7H4cqko%2Fr85zEV66fSxYpOSE%2BIME3ChgXNU6BtqEGacmglQhU1t0yfoeEZUPCGEjywxg8yNHzYRzq5Th%2FdqrAxot2vWsR0M7S7%2F3%2Bhyvzo4JplcP%2BNpU3ZtOF9YreMJh6CR8okWAzopPzAUMds9S27IcgKIiYkkt260%2FyHlwuWadNYcMhUv5ZZmC%2FHSwf8PxA9BL3BtiqYrdlaeABtDe26F%2BDx4M%2BKURMOI4zdl4FmEC3IC%2FKdXN5JByN9xAmrYSTFIIUaX1V3VgOpOmrlivUEuUBmOOFOjyf2ec%3D; pbauth=1_qRoONoJzEcndh1h3YMYz2WGJ979f3TWdz5nuw1DWoOJOBYFDlNs1mWjhDx%2F4USDzDu9Zr4%2BvqY1wIhj9%2BtCay3Ip40j9HbKI4Vs8whMiJZ%2FDl90MNztV%2F6%2FJQVFl8JNVNb6G05t%2F9Uc2bXPFl%2F8e6R%2FtwcVtq090ayRZKXSD3O2GyQRSyZ4LpvUcjq6xM2k%2FQ%2BALR9Qw1v%2BkxJQR%2BxC5K1%2FvlmDp%2BoyTsLAH3eJ7Iq2MfBTy8iXJJqWNhK0PWo4Bu7duipztez2LXYCCf%2Bcorp76nUwr57%2B6q%2FuEYuFnRPDAj9YI8Bpfh%2BE68xHcl3flA5LtPhqxpvrwnpDVRrQRYE0v1oCVzyh1ep3nZH7PeZcs4UEc8EIHbpp459YyEQq6CXS5Xc1fomM%2B2ESEllDexlaUU4M4YjqNXGJ8Hvk5iRzbTBpUBV1UiTEVUvsfDeFNaWYPwkKGwfm7P15Dq2Na1emicmFO1sJdwci83cAHnoKfVUVfzT4bV%2B%2BxKUyJW0aOCW9EwmuzhDDsIbq6c6kbARQ7HhB5Cv4%2B%2B2WSCf7fzYwreO78G%2BuYIdWXA34r1kouydMlqMczPXHtqxuudygq7YGSike0bnei01Dl1NYHA%2B0NJEK6h2z7Alj8oOtOdtU2B3L%2BNRzUwz44W91woPzo9%2FdK%2BV3t7%2BbT8%2BU3TTiv9uXPUD5nuFSSBF9Z6EOCay%2FfQGlf%2F5mzd3%2F%2FhsIe%2FCI4bOBW06yjHL%2FzwXWRYWGG8Q0%3D; pbtr=%7B%22logged_in_page%22%3A1%2C%22page_view%22%3A1%2C%22legacy%22%3A%7B%22page_view%22%3A1%7D%7D; adParams=%7B%22age%22%3A66%2C%22gender%22%3A%22F%22%2C%22reg_zip%22%3A%2287112%22%2C%22username%22%3A%22SandraDodd%22%2C%22login%22%3A%22Y%22%2C%22utype%22%3A%22plus%22%2C%22ba%22%3Anull%2C%22cam_make%22%3A%22canon%22%2C%22cam_type%22%3A%22SLR%22%2C%22m_carrier%22%3A%22T-Mobile_USA%22%7D; pb_beta=rh%2BgTOWy5jDeLatFu5PJnAHoHb8%2FIA1tfLJO0HoVW2gKEaImHcQuZuZeCXKfEnU40rrZpYgwKqSGc9cLkIoko1hk6uti9NgRxXD1tLTchsnu8sbtZOCOpgQSchLH%2Fdp5EzCnj484ka6LrRSm7orh2xDfiNt%2BQWy76J1hR%2F2Uh2JbOoEZ%2FT74RiRAA8o501A5; pb_rt=YzFkR2Z2aW5mcmNTRTFFcGxJendoRE45LzNhMk9lV1grNG9yQUgzMDRrN1ZBMG1jcmdSV3lIa2p1elh1b2pQTDM1enF6am01aDNHaUQ5MTVqUWx4WHNwdjlNUkZjSjMraEVlMFlaK2ppMkMwUGxVSlhGUnQrMEVVemNnYWk4bWRHMVB6bW51NDJEMXI3TlgxZEs1eGlqMGF3enBmQm8xaA%3D%3D; pb_t=emVwUW9sZXJoOHZRZnJiYm5DQ3FxRlR1ZUZmNGp3MVlKclY1Vld3MDM4VmF5VG9VYXREM2F2NVdMTmJMd0ZHZEhqdTAwNHg0QU94Z1BnWHVCQ1pqb0kyaUlkOFl1WkZUL1ZqSnlYQWRvRW0rRFZqczlaa2VmTzR4UERHcUlRTytzeVkxdEdXNllFR3d6REhHRDg0cFM0VEZ0MzhXWFB1dA%3D%3D; pb_un=U2FuZHJhRG9kZA%3D%3D; pbmp=%7B%22userid%22%3A10909473%2C%22userproperties%22%3A%7B%22username%22%3A%22SandraDodd%22%2C%22email%22%3A%22Sandra%40SandraDodd.com%22%2C%22gender%22%3A%22F%22%2C%22birth+date%22%3A%221953-07-24%22%2C%22zip+code%22%3A%2287112%22%2C%22account+created%22%3A%222005-12-15%22%2C%22last+login%22%3A%222020-05-23%22%2C%22storage+used%22%3A%2211.01%22%7D%7D; PHPSESSID=6uljjjp7smmhu694vd1vk40rf2; navToggleCookie=%22show%22; pb_mobile=false; PB_JSENABLED=1; pb_features=1976263543; pb_userid=OTU1OTZhNzkzYWU4NDJkZTQ2ZjRiMzQzN2IzNDk5OTkkYToxOntzOjc6InRyYWNraWQiO3M6MjU6IjE1ODgzMzQ2NzMuNjM5MjEyNjQxOTQ3MTciO30%3D' \
-H 'Referer: https://s26.photobucket.com/user/SandraDodd/library/2014?sort=2&page=0' \
-H 'Cache-Control: no-cache' \
-H 'Host: s26.photobucket.com' \
-H 'Accept-Language: en-us' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Connection: keep-alive' \
-H 'X-Requested-With: XMLHttpRequest' \
| tee photo-list-page.json \
| jq -r '.body.objects[].fullsizeUrl'
# DEBUG: | jq . | tee /dev/fd/2 \
local total_photo_count=`jq -r '.body.total' photo-list-page.json`
rm photo-list-page.json
stderr "total_photo_count: $$total_photo_count"
local page_count=$$(( $$total_photo_count / 23 + 1))
stderr "page_count: $$page_count"
if [[ $$page_count -gt 1 ]]; then
for i in `seq 2 $$page_count`; do \
stderr "- page $$i:"
retry curl $(CURL_PARAMS) \
'https://s26.photobucket.com/component/Common-PageCollection-Album-AlbumPageCollection?filters%5Balbum%5D='`js_escape "$${album_path}"`'&filters%5Balbum_content%5D=2&sort=2&limit=23&page='$${i}'&&linkerMode=true&json=1&hash=b85d3adadbb25a247969cbd262625379&_=1590247575048' \
-XGET \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'Pragma: no-cache' \
-H 'Cookie: _sfkcs2_6uljjjp7smmhu694vd1vk40rf2=TQaeWewbBVqOOW%2FKIpVpHzhghsWgEYcj9n%2BK2iVetiEd0OpwQ5VEunt7H4cqko%2Fr85zEV66fSxYpOSE%2BIME3ChgXNU6BtqEGacmglQhU1t0yfoeEZUPCGEjywxg8yNHzYRzq5Th%2FdqrAxot2vWsR0M7S7%2F3%2Bhyvzo4JplcP%2BNpU3ZtOF9YreMJh6CR8okWAzopPzAUMds9S27IcgKIiYkkt260%2FyHlwuWadNYcMhUv5ZZmC%2FHSwf8PxA9BL3BtiqYrdlaeABtDe26F%2BDx4M%2BKURMOI4zdl4FmEC3IC%2FKdXN5JByN9xAmrYSTFIIUaX1V3VgOpOmrlivUEuUBmOOFOjyf2ec%3D; pbauth=1_qRoONoJzEcndh1h3YMYz2WGJ979f3TWdz5nuw1DWoOJOBYFDlNs1mWjhDx%2F4USDzDu9Zr4%2BvqY1wIhj9%2BtCay3Ip40j9HbKI4Vs8whMiJZ%2FDl90MNztV%2F6%2FJQVFl8JNVNb6G05t%2F9Uc2bXPFl%2F8e6R%2FtwcVtq090ayRZKXSD3O2GyQRSyZ4LpvUcjq6xM2k%2FQ%2BALR9Qw1v%2BkxJQR%2BxC5K1%2FvlmDp%2BoyTsLAH3eJ7Iq2MfBTy8iXJJqWNhK0PWo4Bu7duipztez2LXYCCf%2Bcorp76nUwr57%2B6q%2FuEYuFnRPDAj9YI8Bpfh%2BE68xHcl3flA5LtPhqxpvrwnpDVRrQRYE0v1oCVzyh1ep3nZH7PeZcs4UEc8EIHbpp459YyEQq6CXS5Xc1fomM%2B2ESEllDexlaUU4M4YjqNXGJ8Hvk5iRzbTBpUBV1UiTEVUvsfDeFNaWYPwkKGwfm7P15Dq2Na1emicmFO1sJdwci83cAHnoKfVUVfzT4bV%2B%2BxKUyJW0aOCW9EwmuzhDDsIbq6c6kbARQ7HhB5Cv4%2B%2B2WSCf7fzYwreO78G%2BuYIdWXA34r1kouydMlqMczPXHtqxuudygq7YGSike0bnei01Dl1NYHA%2B0NJEK6h2z7Alj8oOtOdtU2B3L%2BNRzUwz44W91woPzo9%2FdK%2BV3t7%2BbT8%2BU3TTiv9uXPUD5nuFSSBF9Z6EOCay%2FfQGlf%2F5mzd3%2F%2FhsIe%2FCI4bOBW06yjHL%2FzwXWRYWGG8Q0%3D; pbtr=%7B%22logged_in_page%22%3A1%2C%22page_view%22%3A1%2C%22legacy%22%3A%7B%22page_view%22%3A1%7D%7D; adParams=%7B%22age%22%3A66%2C%22gender%22%3A%22F%22%2C%22reg_zip%22%3A%2287112%22%2C%22username%22%3A%22SandraDodd%22%2C%22login%22%3A%22Y%22%2C%22utype%22%3A%22plus%22%2C%22ba%22%3Anull%2C%22cam_make%22%3A%22canon%22%2C%22cam_type%22%3A%22SLR%22%2C%22m_carrier%22%3A%22T-Mobile_USA%22%7D; pb_beta=rh%2BgTOWy5jDeLatFu5PJnAHoHb8%2FIA1tfLJO0HoVW2gKEaImHcQuZuZeCXKfEnU40rrZpYgwKqSGc9cLkIoko1hk6uti9NgRxXD1tLTchsnu8sbtZOCOpgQSchLH%2Fdp5EzCnj484ka6LrRSm7orh2xDfiNt%2BQWy76J1hR%2F2Uh2JbOoEZ%2FT74RiRAA8o501A5; pb_rt=YzFkR2Z2aW5mcmNTRTFFcGxJendoRE45LzNhMk9lV1grNG9yQUgzMDRrN1ZBMG1jcmdSV3lIa2p1elh1b2pQTDM1enF6am01aDNHaUQ5MTVqUWx4WHNwdjlNUkZjSjMraEVlMFlaK2ppMkMwUGxVSlhGUnQrMEVVemNnYWk4bWRHMVB6bW51NDJEMXI3TlgxZEs1eGlqMGF3enBmQm8xaA%3D%3D; pb_t=emVwUW9sZXJoOHZRZnJiYm5DQ3FxRlR1ZUZmNGp3MVlKclY1Vld3MDM4VmF5VG9VYXREM2F2NVdMTmJMd0ZHZEhqdTAwNHg0QU94Z1BnWHVCQ1pqb0kyaUlkOFl1WkZUL1ZqSnlYQWRvRW0rRFZqczlaa2VmTzR4UERHcUlRTytzeVkxdEdXNllFR3d6REhHRDg0cFM0VEZ0MzhXWFB1dA%3D%3D; pb_un=U2FuZHJhRG9kZA%3D%3D; pbmp=%7B%22userid%22%3A10909473%2C%22userproperties%22%3A%7B%22username%22%3A%22SandraDodd%22%2C%22email%22%3A%22Sandra%40SandraDodd.com%22%2C%22gender%22%3A%22F%22%2C%22birth+date%22%3A%221953-07-24%22%2C%22zip+code%22%3A%2287112%22%2C%22account+created%22%3A%222005-12-15%22%2C%22last+login%22%3A%222020-05-23%22%2C%22storage+used%22%3A%2211.01%22%7D%7D; PHPSESSID=6uljjjp7smmhu694vd1vk40rf2; navToggleCookie=%22show%22; pb_mobile=false; PB_JSENABLED=1; pb_features=1976263543; pb_userid=OTU1OTZhNzkzYWU4NDJkZTQ2ZjRiMzQzN2IzNDk5OTkkYToxOntzOjc6InRyYWNraWQiO3M6MjU6IjE1ODgzMzQ2NzMuNjM5MjEyNjQxOTQ3MTciO30%3D' \
-H 'Referer: https://s26.photobucket.com/user/SandraDodd/library/2014?sort=2&page=0' \
-H 'Cache-Control: no-cache' \
-H 'Host: s26.photobucket.com' \
-H 'Accept-Language: en-us' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Connection: keep-alive' \
-H 'X-Requested-With: XMLHttpRequest' \
| jq -r '.body.objects[].fullsizeUrl'
done
fi
stderr "Done."
}
function stderr() {
echo "+++ $$@" > /dev/fd/2
}
function finished_listing() {
grep --quiet ":: Finished $$1." photo-list.txt
}
function list_sub_albums() {
local album_path=`encode_uri_component "$$1"`
retry curl $(CURL_PARAMS) \
'https://s26.photobucket.com/component/Albums-SubalbumList?albumPath='$${album_path}'&fetchSubAlbumsOnly=true&deferCollapsed=true&json=1&hash=7b9a1264c8f2b1fcd0576c5e7b6dd6fd&_=1590250908618' \
-XGET \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'Pragma: no-cache' \
-H 'Cookie: _sfkcs2_6uljjjp7smmhu694vd1vk40rf2=TQaeWewbBVqOOW%2FKIpVpHzhghsWgEYcj9n%2BK2iVetiEd0OpwQ5VEunt7H4cqko%2Fr85zEV66fSxYpOSE%2BIME3ChgXNU6BtqEGacmglQhU1t0yfoeEZUPCGEjywxg8yNHzYRzq5Th%2FdqrAxot2vWsR0M7S7%2F3%2Bhyvzo4JplcP%2BNpU3ZtOF9YreMJh6CR8okWAzopPzAUMds9S27IcgKIiYkkt260%2FyHlwuWadNYcMhUv5ZZmC%2FHSwf8PxA9BL3BtiuaLBgaeABtDe26F%2BDx4M%2BKURMOI4zdl4FmEC3IC%2FKdXN5JByN9xAmrYSTFIIUaX1V3VsJpOmrlivUEuUBmOOFOk2f18KaTw0oG8Ctzq40BJkp; pbauth=1_qRoONoJzEcndh1h3YMYz2WGJ979f3TWdz5nuw1DWoOJOBYFDlNs1mWjhDx%2F4USDzDu9Zr4%2BvqY1wIhj9%2BtCay3Ip40j9HbKI4Vs8whMiJZ%2FDl90MNztV%2F6%2FJQVFl8JNVNb6G05t%2F9Uc2bXPFl%2F8e6YD4uDVQU19BLfvo5GkTlYWGyQRSyZ4LpvFi2kUA%2F8aIz1zSw8aI5Gp4L%2BGlA8GOHPlTvPPnQ%2BX0I%2F2tnFafr7vpDoDh5aaMhLrni6pKipTAv%2FydiDvIoMgYtlQaRol%2FCfELqn3dhb6xAtL2GwEde%2BqmZ4wo4Oy%2FMHC%2BQpTE3SvhsDwPWTbYLyADQXYq81T8S%2BDCGD%2FRTUyzjGUru032JDWjWQ05JjVNdfVVfTbhzjbymWinYbhvGtHaTEq1i5%2Bz3VaUU4M4YjqNeqymi7fdN8%2FKJGgZn1j691HCXz7SlFrtlV%2BrkWtphZe9JzcqX7b3NISin93QX3fYvyqVLoevpvVZj%2Fu8LfTJBwWfCLsT5H3BxS%2BGxAvOt1%2Bzgwa7tt%2FtoTlCQWrDSTGff%2FDLhlvsaLAqJkbRlxBkQG4CublivlW3AJ5tLvRoP%2B6Ec15B4pHYj971xwCChqS3Hp%2BEQOfjHbuxIFYbPB0fH0hpWMw%2BuLc7B3L%2BNRzUwz44W91woPzo9%2FdK%2BV3t7%2BbT8%2BU3TTiv9uXPUD5nuFSSBF9Z6EOCay%2FfQGlf%2F5mzd3%2F%2FhsIe%2FCI4bOBW06yjHL%2FzpWgRDbDMsN4%3D; adParams=%7B%22age%22%3A66%2C%22gender%22%3A%22F%22%2C%22reg_zip%22%3A%2287112%22%2C%22username%22%3A%22SandraDodd%22%2C%22login%22%3A%22Y%22%2C%22utype%22%3A%22plus%22%2C%22ba%22%3Anull%2C%22cam_make%22%3A%22canon%22%2C%22cam_type%22%3A%22SLR%22%2C%22m_carrier%22%3A%22T-Mobile_USA%22%7D; pb_beta=rh%2BgTOWy5jDeLatFu5PJnAHoHb8%2FIA1tfLJO0HoVW2gKEaImHcQuZuZeCXKfEnU40rrZpYgwKqSGc9cLkIoko1hk6uti9NgRxXD1tLTchsnu8sbtZOCOpgQSchLH%2Fdp5EzCnj484ka6LrRSm7orh2xDfiNt%2BQWy76J1hR%2F2Uh2JbOoEZ%2FT74RiRAA8o501A5; pb_rt=YzFkR2Z2aW5mcmNTRTFFcGxJendoRE45LzNhMk9lV1grNG9yQUgzMDRrN1ZBMG1jcmdSV3lIa2p1elh1b2pQTDM1enF6am01aDNHaUQ5MTVqUWx4WHNwdjlNUkZjSjMraEVlMFlaK2ppMkMwUGxVSlhGUnQrMEVVemNnYWk4bWRHMVB6bW51NDJEMXI3TlgxZEs1eGlqMGF3enBmQm8xaA%3D%3D; pb_t=emVwUW9sZXJoOHZRZnJiYm5DQ3FxRlR1ZUZmNGp3MVlKclY1Vld3MDM4VmF5VG9VYXREM2F2NVdMTmJMd0ZHZEhqdTAwNHg0QU94Z1BnWHVCQ1pqb0kyaUlkOFl1WkZUL1ZqSnlYQWRvRW0rRFZqczlaa2VmTzR4UERHcUlRTytzeVkxdEdXNllFR3d6REhHRDg0cFM0VEZ0MzhXWFB1dA%3D%3D; pb_un=U2FuZHJhRG9kZA%3D%3D; pbmp=%7B%22userid%22%3A10909473%2C%22userproperties%22%3A%7B%22username%22%3A%22SandraDodd%22%2C%22email%22%3A%22Sandra%40SandraDodd.com%22%2C%22gender%22%3A%22F%22%2C%22birth+date%22%3A%221953-07-24%22%2C%22zip+code%22%3A%2287112%22%2C%22account+created%22%3A%222005-12-15%22%2C%22last+login%22%3A%222020-05-23%22%2C%22storage+used%22%3A%2211.01%22%7D%7D; PHPSESSID=6uljjjp7smmhu694vd1vk40rf2; navToggleCookie=%22show%22; pb_mobile=false; PB_JSENABLED=1; pb_features=1976263543; pb_userid=OTU1OTZhNzkzYWU4NDJkZTQ2ZjRiMzQzN2IzNDk5OTkkYToxOntzOjc6InRyYWNraWQiO3M6MjU6IjE1ODgzMzQ2NzMuNjM5MjEyNjQxOTQ3MTciO30%3D' \
-H 'Referer: https://s26.photobucket.com/user/SandraDodd/library/?sort=3&page=1' \
-H 'Cache-Control: no-cache' \
-H 'Host: s26.photobucket.com' \
-H 'Accept-Language: en-us' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Connection: keep-alive' \
-H 'X-Requested-With: XMLHttpRequest' \
| jq -r '.body.subAlbums[] | [.mediaCount, .path] | @tsv'
}
function encode_uri_component() {
node -e "console.log(encodeURIComponent('$$1'))"
}
function js_escape() {
node -e "console.log(escape('$$1'))"
}
function retry() {
local retry_count=0
while true; do
((retry_count++))
timeout 5 "$$@" && break
stderr "Timed out; retrying ($$retry_count)."
if [[ $$retry_count > 4 ]]; then
stderr "Command failed 5 times.\n $$@"
break
fi
done
}
main
download-photos:
mkdir -p photos
grep -v '^:: Finished ' photo-list.txt \
| grep -vi '\.pbw$$' \
| grep -vi '\.mp4$$' \
| xargs wget --no-verbose --continue --mirror --no-host-directories -P photos
download-videos:
grep -v '^:: Finished ' photo-list.txt \
| grep -i '\.mp4$$' \
| xargs wget --no-verbose --header='Referer: https://opic2.pbsrc.com/flash/oldplayer/Player.swf' --continue --mirror --no-host-directories -P photos
download-flash-videos:
grep -v '^:: Finished ' photo-list.txt \
| grep -i '\.flv$$' \
| xargs wget --no-verbose --header='Referer: https://opic2.pbsrc.com/flash/oldplayer/Player.swf' --continue --mirror --no-host-directories -P photos
download-pbw:
grep -v '^:: Finished ' photo-list.txt \
| grep -i '\.pbw$$' \
| xargs wget --no-verbose --header='Referer: https://opic2.pbsrc.com/flash/oldplayer/Player.swf' --continue --mirror --no-host-directories -P photos
stats:
@echo "Enlisted photos: `grep -v '^:: Finished ' photo-list.txt | sort -fu | wc -l`"
echo "Downloaded photos: `find photos/albums/c111/SandraDodd -type f | wc -l`"
echo "Total photo size: `du -sh photos/albums/c111/SandraDodd`"
edit:
code -n .
e: edit
upload:
rsync -azv photos/albums/ root@ssh.sandradodd.com:/var/www/site/albums
// Just manually walk the album list and click them all so that they are expanded.
var areAllAlbumsExpanded = $$('#albumsTargetList .icon-caret-right').length === 0;
// Count all the photos, given all of the albums are expanded.
var totalPhotoCount = $$("#albumsTargetList a.icon-all-photos")
.map((a) => a.title)
.filter((t) => t.match(/ \(\d+\)$/))
.map((t) => parseInt(t.match(/\((\d+)\)$/)[1]))
.reduce((a, x) => a + x, 0);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.