Create a gist now

Instantly share code, notes, and snippets.

@riyad /.gitignore
Last active Mar 29, 2018

What would you like to do?
Backup and Restore your Android Phone With ADB
.DS_Store
/tmp/

Android-Backup

Backup and restore your Android phone with ADB (and rsync)

It will backup and restore all of your /sdcard directory. Assuming you're using also something like Titanium Backup you'll be able to backup and restore all your apps, settings and data.

It uses ADB for setup and rsync to do the copying since the Android File Transfer Tool for Mac has a laughable quality for Google’s standards.

It's based on a pure ADB version by Riyad Preukschas and has been improved with ideas and methods from Simon Josefsson and pts.

Installation

Assuming you've installed and setup ADB from Android's SDK tools package. Download and extract the current version.

wget -O android-backup-master.tar.gz https://github.com/riyad/android-backup/archive/master.tar.gz
tar xzf android-backup-master.tar.gz
cd android-backup-master/

Move the files to their necessary locations.

# move scripts to a directory on $PATH
mv android-backup android-restore /usr/local/bin/
# move backup rsync to a location where it can be found
mkdir /usr/local/lib/android-backup
mv rsync.bkp /usr/local/lib/android-backup/

Make sure that the directory you move android-backup and android-restore to is in $PATH.

Usage

Backup

Connect your phone via USB. Make sure you have set up ADB debugging already. Then run:

android-backup [<backup-dir>]

If you haven't given a backup directory it will ask you if it should generate one for you.

If you have an older backup already you can use it to speed up the backup process. Rsync will automatically find out what files to download leaving out those that have not changed in the mean time.

cp -r previous-backup/ current-backup/
android-backup current-backup/

Restore

Connect your phone via USB. Make sure you have set up ADB debugging already. Then run:

android-restore <backup-dir>

Caveats

  • Currently restoring will not be able to be able to restore the correct timestamps of files. They'll be set to the current date and time when you're restoring them.

Extract Rsync for Android Yourself

The provided rsync.bkp file was extracted from LineageOS 14.1-20170131-NIGHTLY for OnePlus X via

adb shell which rsync
adb pull /system/xbin/rsync rsync.bkp

Build Rsync for Android Yourself

NOTE: The following steps assume you already have the Android NDK installed.

Clone the rsync for android source (e.g. from @CyanogenMod) ...

git clone https://github.com/LineageOS/android_external_rsync.git
cd android_external_rsync
# checkout the most recent branch
git checkout cm-14.1

... create the missing Application.mk build file ...

mkdir jni
# create missing build script (e.g. from https://gist.github.com/riyad/59c17ce7a1ade6cfc3c6)
wget -O jni/Application.mk https://gist.githubusercontent.com/riyad/59c17ce7a1ade6cfc3c6/raw/b56679f6188d9f56315dcd5e904fad0f9bd1439d/Application.mk

... and start the build with:

export NDK_PROJECT_PATH=`pwd`
ndk-build -d rsync

Find your self-build rsync in obj/local/*/rsync.

Contact and Issues

Please, report all issues on our issue tracker on GitHub: https://github.com/riyad/android-backup/issues

#!/bin/bash
#
# Riyad Preukschas <riyad@informatik.uni-bremen.de>
#
# Backs up your Android phone's storage.
# It'll also back up your current TitaniumBackup apps for faster restoration.
# on a running Android /sdcard is symlinked to /storage/emulated/[legacy|0]
# but in TWRP it's directly mounted to /sdcard (so the other dirs aren't there)
readonly ANDROID_STORAGE="/sdcard"
readonly REMOTE_TEMP_DIR="/data/local/tmp" # you can't chmod +x files in /sdcard
readonly REMOTE_RSYNC_CONFIG="${REMOTE_TEMP_DIR}/rsync.conf"
readonly REMOTE_RSYNC_PORT=1873 # > 1024 so we don't need to be root
readonly LOCAL_RSYNC_PORT=6010
REMOTE_RSYNC_BIN="rsync"
readonly REMOTE_TEMP_RSYNC_BIN="${REMOTE_TEMP_DIR}/rsync"
readonly LOCAL_RSYNC_FILTER_FILE=`mktemp -t "$(basename $0).XXXXXXXXXX"`
# log everything using syslog
exec 2>&1 | logger
alias adb="adb -d"
adb_package_path() {
adb shell pm path "$1" | tr -d '\r' | sed "s/^package://"
}
adb_pull_app() {
if [[ -z "$(adb_package_path \"$1\")" ]]; then
echo "Can't find app $1"
return 1
else
rsync_pull "$(adb_package_path \"$1\")" "$2"
fi
}
adb_push() {
adb push -p "$1" "$2"
}
adb_rm_dir() {
adb shell "[[ -d \"$1\" ]] && rm -r \"$1\""
}
rsync_cleanup() {
adb forward --remove tcp:${LOCAL_RSYNC_PORT}
adb shell rm -f "${REMOTE_RSYNC_CONFIG}"
if [[ -f "${REMOTE_TEMP_RSYNC_BIN}" ]]; then
echo "Cleaning up temporary rsync"
adb shell rm -f "${REMOTE_TEMP_RSYNC_BIN}"
fi
[[ -e "${LOCAL_RSYNC_FILTER_FILE}" ]] && rm "${LOCAL_RSYNC_FILTER_FILE}"
}
rsync_pull() {
rsync --partial --progress --archive --human-readable --delete \
--filter=". ${LOCAL_RSYNC_FILTER_FILE}" --prune-empty-dirs \
"rsync://localhost:${LOCAL_RSYNC_PORT}/root/$1" "$2"
: rc $?
}
rsync_setup() {
if [[ -z "$(adb shell which \"${REMOTE_RSYNC_BIN}\")" ]]; then
echo "Device doesn't have rsync."
# try different locations for backup rsync
local SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
for rsync_src_dir in "${SCRIPT_DIR}" "/usr/local/lib/android-backup"; do
local RSYNC_SRC="${rsync_src_dir}/rsync.bkp"
[[ -e "${RSYNC_SRC}" ]] && break
done
# have we found a backup rsync?
if [[ -e "${RSYNC_SRC}" ]]; then
echo "Providing temporary rsync from ${RSYNC_SRC} ..."
adb_push "${RSYNC_SRC}" "${REMOTE_TEMP_RSYNC_BIN}"
adb shell chmod 755 "${REMOTE_TEMP_RSYNC_BIN}"
REMOTE_RSYNC_BIN="${REMOTE_TEMP_RSYNC_BIN}"
else
echo "Can't provide temprorary rsync."
exit 1
fi
fi
# setup rsync daemon on the device
# from http://blog.josefsson.org/2015/11/28/automatic-android-replicant-backup-over-usb-using-rsync/
# and http://ptspts.blogspot.de/2015/03/how-to-use-rsync-over-adb-on-android.html
adb shell "echo \"address = 127.0.0.1\nport = ${REMOTE_RSYNC_PORT}\n\n[root]\npath = /\ngid = sdcard_rw\nread only = true\"" \> "${REMOTE_RSYNC_CONFIG}"
touch "${LOCAL_RSYNC_FILTER_FILE}"
adb shell "${REMOTE_RSYNC_BIN} --daemon --no-detach --config=\"${REMOTE_RSYNC_CONFIG}\" &" # for debugging --log-file=/proc/self/fd/2
# forward a local port to the rsync daemon on the device
adb forward tcp:${LOCAL_RSYNC_PORT} tcp:${REMOTE_RSYNC_PORT}
sleep 2
}
# make sure we get a path to backup stuff into
if [[ -z "$1" ]]; then
backup_dir_name="./bkp `date '+%Y-%m-%dT%H-%M'`"
echo "You haven't specified a backup directory."
read -p "Use '${backup_dir_name}'? [y/n] " yn
case $yn in
[Yy]* ) ;; # just get through
[Nn]* ) exit;;
* ) exit 1;;
esac
readonly BACKUP_DIR="${backup_dir_name}"
else
readonly BACKUP_DIR="$1"
fi
# make sure we don't overwrite anything by accident
if [[ -d "${BACKUP_DIR}" ]]; then
echo "WARNING: '${BACKUP_DIR}' exists already!"
read -p "Are you sure you want to overwrite files in there? [Yes/No] " yn
case $yn in
'Yes'* ) ;; # just get through
'No'* ) exit;;
* ) exit 1;;
esac
fi
mkdir -p "${BACKUP_DIR}"
rsync_setup
echo "Backing up user data ..."
# ignore known cache files
echo 'H .DS_Store' >> "${LOCAL_RSYNC_FILTER_FILE}"
# don't let rsync delete these (see below)
echo 'P /TitaniumBackup.apk' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'P /TitaniumBackupPro.apk' >> "${LOCAL_RSYNC_FILTER_FILE}"
# don't sync certain files and directories
echo 'H .cache/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H .thumbs/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H .thumbnails/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /Android/data/com.android.providers.media/albumthumbs/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /Android/data/*/cache' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /Applidium Image Cache/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /CameraZOOM/.packs/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /CameraZOOM/.temp/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /osmand/*.obf' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /osmand/tiles/**/*.tile' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /quran_android/audio/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# don't sync large directories that can easily be restored
echo 'H /Music/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /Movies/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /Podcasts/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# start the backup
rsync_pull "${ANDROID_STORAGE}/" "${BACKUP_DIR}/"
# backup TitaniumBackup app separately
echo "Backing up TitaniumBackup apps ..."
adb_pull_app com.keramidas.TitaniumBackup "${BACKUP_DIR}/TitaniumBackup.apk"
adb_pull_app com.keramidas.TitaniumBackupPro "${BACKUP_DIR}/TitaniumBackupPro.apk"
rsync_cleanup
echo "... Done!"
#!/bin/bash
#
# Riyad Preukschas <riyad@informatik.uni-bremen.de>
#
# Restores previously backed-up files to your Android phone.
# You still need to restore your apps and config by running TitaniumBackup.
# for easy listing of directory contents starting with '.'
GLOBIGNORE=.:..
# on a running Android /sdcard is symlinked to /storage/emulated/[legacy|0]
# but in TWRP it's directly mounted to /sdcard (so the other dirs aren't there)
readonly ANDROID_STORAGE="/sdcard"
readonly REMOTE_TEMP_DIR="/data/local/tmp" # you can't chmod +x files in /sdcard
readonly REMOTE_RSYNC_CONFIG="${REMOTE_TEMP_DIR}/rsync.conf"
readonly REMOTE_RSYNC_PORT=1873 # > 1024 so we don't need to be root
readonly LOCAL_RSYNC_PORT=6010
REMOTE_RSYNC_BIN="rsync"
readonly REMOTE_TEMP_RSYNC_BIN="${REMOTE_TEMP_DIR}/rsync"
readonly LOCAL_RSYNC_FILTER_FILE=`mktemp -t "$(basename $0).XXXXXXXXXX"`
# these are relative to BACKUP_DIR
readonly TB_DIR="TitaniumBackup"
readonly TB_APKS="TitaniumBackup.apk TitaniumBackupPro.apk"
readonly PRIO_DIRS="WhatsApp DCIM Pictures"
readonly SKIP_DIRS="${TB_DIR} ${TB_APKS} ${PRIO_DIRS}"
# log everything using syslog
exec 2>&1 | logger
alias adb="adb -d"
# thanks to http://stackoverflow.com/questions/8063228/how-do-i-check-if-a-variable-exists-in-a-list-in-bash
contains() {
local list="$1"
local item="$2"
[[ ${list} =~ (^|[[:space:]])${item}($|[[:space:]]) ]]
}
adb_install() {
adb install "$1"
}
adb_push() {
adb push -p "$1" "${ANDROID_STORAGE}/$1"
}
rsync_cleanup() {
adb forward --remove tcp:${LOCAL_RSYNC_PORT}
adb shell rm -f "${REMOTE_RSYNC_CONFIG}"
if [[ -f "${REMOTE_TEMP_RSYNC_BIN}" ]]; then
echo "Cleaning up temporary rsync"
adb shell rm -f "${REMOTE_TEMP_RSYNC_BIN}"
fi
[[ -e "${LOCAL_RSYNC_FILTER_FILE}" ]] && rm "${LOCAL_RSYNC_FILTER_FILE}"
}
rsync_push() {
# needs extra --temp-dir option against 'mkstemp "..." (in root) failed: Operation not permitted (1)' errors
# needs extra --no-times option against 'failed to set times on "..." (in root): Operation not permitted (1)' errors
rsync --partial --progress --archive --human-readable --delete \
--filter=". ${LOCAL_RSYNC_FILTER_FILE}" --temp-dir="${REMOTE_TEMP_DIR}" --no-times \
"$1" "rsync://localhost:${LOCAL_RSYNC_PORT}/root/$2"
: rc $?
}
rsync_setup() {
if [[ -z "$(adb shell which \"${REMOTE_RSYNC_BIN}\")" ]]; then
echo "Device doesn't have rsync."
# try different locations for backup rsync
local SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
for rsync_src_dir in "${SCRIPT_DIR}" "/usr/local/lib/android-backup"; do
local RSYNC_SRC="${rsync_src_dir}/rsync.bkp"
[[ -e "${RSYNC_SRC}" ]] && break
done
# have we found a backup rsync?
if [[ -e "${RSYNC_SRC}" ]]; then
echo "Providing temporary rsync from ${RSYNC_SRC} ..."
adb_push "${RSYNC_SRC}" "${REMOTE_TEMP_RSYNC_BIN}"
adb shell chmod 755 "${REMOTE_TEMP_RSYNC_BIN}"
REMOTE_RSYNC_BIN="${REMOTE_TEMP_RSYNC_BIN}"
else
echo "Can't provide temprorary rsync."
exit 1
fi
fi
# setup rsync daemon on the device
# from http://blog.josefsson.org/2015/11/28/automatic-android-replicant-backup-over-usb-using-rsync/
# and http://ptspts.blogspot.de/2015/03/how-to-use-rsync-over-adb-on-android.html
adb shell "echo \"address = 127.0.0.1\nport = ${REMOTE_RSYNC_PORT}\n\n[root]\npath = /\nuse chroot = false\nread only = false\"" \> "${REMOTE_RSYNC_CONFIG}"
touch "${LOCAL_RSYNC_FILTER_FILE}"
adb shell "${REMOTE_RSYNC_BIN} --daemon --no-detach --config=\"${REMOTE_RSYNC_CONFIG}\" &" # for debugging --log-file=/proc/self/fd/2
# forward a local port to the rsync daemon on the device
adb forward tcp:${LOCAL_RSYNC_PORT} tcp:${REMOTE_RSYNC_PORT}
sleep 2
}
# make sure we get a path to restore stuff from
if [[ -z "$1" ]]; then
echo "Please specify the directory your backups are in!"
exit 1
else
readonly BACKUP_DIR="$1"
fi
if [[ ! -d "${BACKUP_DIR}" ]]; then
echo "'${BACKUP_DIR}' is not a directory."
exit 1
fi
rsync_setup
# ignore known cache files
echo 'H .DS_Store' >> "${LOCAL_RSYNC_FILTER_FILE}"
# don't sync custom files
echo 'H /TitaniumBackup.apk' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo 'H /TitaniumBackupPro.apk' >> "${LOCAL_RSYNC_FILTER_FILE}"
# preserve standard Andoid directories
# echo 'P Alarms/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Android/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P DCIM/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Download/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Movies/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Music/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Notifications/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Pictures/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Podcasts/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P Ringtones/' >> "${LOCAL_RSYNC_FILTER_FILE}"
# echo 'P storage/' >> "${LOCAL_RSYNC_FILTER_FILE}"
echo "Restoring TitaniumBackup data first ..."
if [ ! -d "${BACKUP_DIR}/${TB_DIR}" ]; then
echo "Can't find TitaniumBackup data directory"
exit 1
fi
rsync_push "${BACKUP_DIR}/${TB_DIR}/" "${ANDROID_STORAGE}/${TB_DIR}/"
# install TitaniumBackup so we can start restoring stuff on the device itself
echo "Restoring TitaniumBackup apps ..."
for apk in ${TB_APKS}; do
if [[ -f "${BACKUP_DIR}/${apk}" ]]; then
adb_install "${BACKUP_DIR}/${apk}"
else
echo "Can't find TitaniumBackup app ${apk}."
exit 1
fi
done
echo "Restoring some other things with priority ..."
for dir in ${PRIO_DIRS}; do
[[ ! -d "${BACKUP_DIR}/${dir}" ]] && continue
rsync_push "${BACKUP_DIR}/${dir}/" "${ANDROID_STORAGE}/${dir}/"
done
echo "Restoring all the rest ..."
rsync_push "${BACKUP_DIR}/" "${ANDROID_STORAGE}/"
rsync_cleanup
echo "... Done!"
The MIT License (MIT)
Copyright (c) 2015-2016 Riyad Preukschas
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
* Add flag to force using the backup rsync
* Try to get root on the device if possible
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment