Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Google Cloud Bucket: create explicit dirs
#!/bin/bash
##
## Creates explicit dirs on the bucket
##
function print_help() {
echo "Usage:" $(basename $0) bucket_mounted_dir
exit 1
}
if [[ -z $1 || ! -d $1 ]]; then
print_help
fi
# Expect mount dir is /*/*
path=$(echo $1|sed -rn 's|^/[^/]+/[^/]+/?||p')
mnt=${1%$path}
mnt=${mnt%/}
path=${path%/}
bucket=$(mount | grep -F "$mnt")
bucket=${bucket%% *}
if [[ -z "$bucket" || "$bucket" == /* ]]; then
echo No or wrong mount point for $mnt
exit 1
fi
bucket=gs://"$bucket"
echo Creating dirs for $bucket/$path in $mnt/$path ...
echo Press ENTER to continue ...
read
declare -A cdirs # created dirs
gsutil ls -r "$bucket/$path" | while IFS='' read -r line; do
if [[ -z "$line" || "$line" == */ || "$line" == *: ]]; then continue; fi
#echo Line $line
dir=${line#$bucket}
if [[ -z $dir ]]; then continue; fi
dir=${dir%/*} # dirname
#echo Examining $dir
if [[ -z $dir || ${cdirs[$dir]+1} ]]; then continue; fi
cdirs[$dir]=1
diskdir="$mnt$dir"
if [[ -e "$diskdir" ]]; then
echo Dir already exists $diskdir
continue
fi
echo Creating $diskdir
mkdir -p "$diskdir"
done
@yalaudah

This comment has been minimized.

Copy link

@yalaudah yalaudah commented May 30, 2018

Hi Ondrej, can you please explain how this script should be used? Should it be run in the directory where the bucket is mounted?

@bintt

This comment has been minimized.

Copy link

@bintt bintt commented Dec 13, 2018

thank you very much, your code is a life saver.

Hi Ondrej, can you please explain how this script should be used? Should it be run in the directory where the bucket is mounted?
you can run this code anywhere, it comes with help how to use it.

@xmedeko

This comment has been minimized.

Copy link
Owner Author

@xmedeko xmedeko commented Jul 22, 2019

@yalaudah The script needs a mounted by gcsfuse and working gsutil command. The mounted gcsfuse dir, where you want to created dirs is a first argument of the script. (The gs:// uri for gsutil is recognized from the output of mount command.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.