Skip to content

Instantly share code, notes, and snippets.

@tavinus
Last active September 12, 2024 12:57
Show Gist options
  • Save tavinus/93bdbc051728748787dc22a58dfe58d8 to your computer and use it in GitHub Desktop.
Save tavinus/93bdbc051728748787dc22a58dfe58d8 to your computer and use it in GitHub Desktop.
Send files to Nextcloud/Owncloud shared folder using curl
#!/usr/bin/env bash
############################################################
# MIGRATED TO REPOSITORY
# https://github.com/tavinus/cloudsend.sh
#
# This gist will NOT be updated anymore
############################################################
############################################################
#
# cloudsend.sh
#
# Uses curl to send files to a shared
# Nextcloud/Owncloud folder
#
# Usage: ./cloudsend.sh <file> <folderLink>
# Help: ./cloudsend.sh -h
#
# Gustavo Arnosti Neves
# https://github.com/tavinus
#
# Contributors:
# @MG2R @gessel
#
# Get this script to current folder with:
# curl -O 'https://raw.githubusercontent.com/tavinus/cloudsend.sh/master/cloudsend.sh' && chmod +x cloudsend.sh
#
############################################################
CS_VERSION="0.1.6"
CLOUDURL=""
FOLDERTOKEN=""
PUBSUFFIX="public.php/webdav"
HEADER='X-Requested-With: XMLHttpRequest'
INSECURE=''
# https://cloud.mydomain.net/s/fLDzToZF4MLvG28
# curl -k -T myFile.ext -u "fLDzToZF4MLvG28:" -H 'X-Requested-With: XMLHttpRequest' https://cloud.mydomain.net/public.php/webdav/myFile.ext
log() {
[ "$VERBOSE" == " -s" ] || printf "%s\n" "$1"
}
printVersion() {
printf "%s\n" "CloudSender v$CS_VERSION"
}
initError() {
printVersion >&2
printf "%s\n" "Init Error! $1" >&2
printf "%s\n" "Try: $0 --help" >&2
exit 1
}
usage() {
printVersion
printf "\n%s%s\n" "Parameters:" "
-h | --help Print this help and exits
-q | --quiet Be quiet
-V | --version Prints version and exits
-k | --insecure Uses curl with -k option (https insecure)
-p | --password Uses env var \$CLOUDSEND_PASSWORD as share password
You can 'export CLOUDSEND_PASSWORD' at your system, or set it at the call.
Please remeber to also call -p to use the password set."
printf "\n%s\n%s\n%s\n" "Use:" " $0 <filepath> <folderLink>" " CLOUDSEND_PASSWORD='MySecretPass' $0 -p <filepath> <folderLink>"
printf "\n%s\n%s\n%s\n" "Example:" " $0 './myfile.txt' 'https://cloud.mydomain.net/s/fLDzToZF4MLvG28'" " CLOUDSEND_PASSWORD='MySecretPass' $0 -p './myfile.txt' 'https://cloud.mydomain.net/s/fLDzToZF4MLvG28'"
}
##########################
# Process parameters
if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
usage
exit 0
fi
if [ "$1" = "-V" ] || [ "$1" = "--version" ]; then
printVersion
exit 0
fi
if [ "$1" = "-q" ] || [ "$1" = "--quiet" ]; then
VERBOSE=" -s"
shift
fi
if [ "$1" = "-k" ] || [ "$1" = "--insecure" ]; then
INSECURE=' -k'
log " > Insecure mode ON"
shift
fi
if [ "$1" = "-p" ] || [ "$1" = "--password" ]; then
PASSWORD=${CLOUDSEND_PASSWORD}
log " > Using password from env"
shift
fi
##########################
# Validate input
FILENAME="$1"
CLOUDURL=''
# if we have index.php in the URL, process accordingly
if [[ $2 == *"index.php"* ]]; then
CLOUDURL="${2%/index.php/s/*}"
else
CLOUDURL="${2%/s/*}"
fi
FOLDERTOKEN="${2##*/s/}"
if [ ! -f "$FILENAME" ]; then
initError "Invalid input file: $FILENAME"
fi
if [ -z "$CLOUDURL" ]; then
initError "Empty URL! Nowhere to send..."
fi
if [ -z "$FOLDERTOKEN" ]; then
initError "Empty Folder Token! Nowhere to send..."
fi
##########################
# Check for curl
CURLBIN='/usr/bin/curl'
if [ ! -x "$CURLBIN" ]; then
CURLBIN="$(which curl 2>/dev/null)"
if [ ! -x "$CURLBIN" ]; then
initError "No curl found on system!"
fi
fi
##########################
# Extract base filename
BFILENAME=$(/usr/bin/basename $FILENAME)
##########################
# Send file
#echo "$CURLBIN"$INSECURE$VERBOSE -T "$FILENAME" -u "$FOLDERTOKEN":"$PASSWORD" -H "$HEADER" "$CLOUDURL/$PUBSUFFIX/$BFILENAME"
"$CURLBIN"$INSECURE$VERBOSE -T "$FILENAME" -u "$FOLDERTOKEN":"$PASSWORD" -H "$HEADER" "$CLOUDURL/$PUBSUFFIX/$BFILENAME"
@MG2R
Copy link

MG2R commented Mar 27, 2019

I've created a fork to add a more quiet mode and to allow working with passwords. Pretty new to actually working with gists... I'm assuming there's no PRs here?

@tavinus
Copy link
Author

tavinus commented Apr 17, 2019

Hi there! Sorry for the delay. This gives no notifications of messages. Also, no PRs on gists ...

I like your changes @MG2R , I will try to add them here.
I guess I will import your fork to a full repo (next week).
Just need to add some parameter parsing so we can pass them in any order (that is easy, I have it ready from other scripts).

Apart from this, I have another script called cloudmanager that can do pretty much anything that webdav allows.
But it is in Brazilian Portuguese only for now. I will see about making it multilanguage or translating it.

I will post here when I do that.
Here is the help in Portuguese (the commands are in English, so you can guess it):

$ cloudmanager --help

Arnosti Cloud Manager v0.0.5

OPCOES

 -V, --version  Imprime versao na tela e termina execucao
 -h, --help     Imprime esta ajuda e termina execucao

COMANDOS

> ls,list <destino>
    Lista uma pasta ou arquivo
    Ex: cloudmanager list

> mkdir,makeFolder <nomedapasta>
    Cria uma pasta no cloud
    Ex: cloudmanager mkdir PastaTeste

> send,upload <pasta/arquivo.local> <pasta/arquivo.cloud>
    Envia um arquivo local para o cloud
    Ex: cloudmanager send '/user/nfe/meuarquivo.xml' 'ARCO/nfe/meuarquivo.xml'

> mv,move <origem> <destino>
    Move um arquivo ou pasta dentro do cloud
    Ex: cloudmanager move 'ARCO/meuarquivo.txt' 'ATL/meuarquivo.txt'

> cp,copy <origem> <destino>
    Copia um arquivo ou pasta dentro do cloud
    Ex: cloudmanager copy 'ARCO/meuarquivo.txt' 'ATL/meuarquivo.txt'

> del,delete <destino>
    Apaga um arquivo ou pasta dentro do cloud (move para a lixeira)
    Ex: cloudmanager del 'ARCO/meuarquivo.txt'

> get,download <origem> <destino>
    Faz download de um arquivo do cloud para um disco local
    Ex: cloudmanager get 'ARCO/meuarquivo.txt' '/user/nfe/meuarquivo.txt'

> shares,listShares
    Imprime tabela com compartilhamentos (183 cols)
    Especifique uma pasta ou deixe em branco para listar tudo.
    Ex: cloudmanager shares

> links,sharedLinks
    Lista os compartilhamentos por links em formato <URL>,<NOME ARQUIVO>
    Ex: cloudmanager listShares

This one needs a config file for authentication though (for obvious reasons).
Cheers!

@didierm
Copy link

didierm commented Aug 21, 2019

Hi,
any inclination to release your cloudmanager script ?

The following one-liner downloads all files from a password-protected shared folder :
(xml_grep dependency ; could also be done with xmllint or xpath)

SRV={full server URL}
USR={folder token}
PW={password}

for url in $(curl -s -X PROPFIND -u ${USR}:${PW} ${SRV}/public.php/webdav/ | xml_grep --text_only "//d:multistatus/d:response/d:href" | egrep -v '/$') ; do echo "Downloading : $url ..." ; curl -sJO -X GET -u ${USR}:${PW} ${SRV}/$url ; done

@tavinus
Copy link
Author

tavinus commented Sep 5, 2019

I've been so busy mate..

x_x

@xtrasimplicity
Copy link

Any way to make this use a password protected share?

I'm not sure if you discovered this already, but the : in the -u flag is a separator between the username and password.
i.e.

 -u "$FOLDERTOKEN":"MyPassword"

@eudoxos
Copy link

eudoxos commented Mar 13, 2020

is it working with nc15? how to debug sending problems?
Anyone can confirm? I am testing with nc17, there is no error after the upload, but no file appears in the cloud...

@tavinus
Copy link
Author

tavinus commented Mar 16, 2020

Hi all.
I have updated the app with changes from others and mine.

v0.1.5

Just tested sending files to a Nextcloud 18.0.1 instance share with and without password and both worked fine.

Shares with passwords

To upload to a share with password set you need to

  • Set environment variable $CLOUDSEND_PASSWORD
    AND
  • Use the -p | --password flag on call

To set the env var password

To set the variable you can either do it system-wide (not too secure)

export CLOUDSEND_PASSWORD='MySecretPass'

Or you can define at the time of the call, appending as in the --help examples.

There are many ways to implement it this way, but the idea is that the password var is defined only in the context of the bash instance that it is running.

Also, be aware that the password will probably end up at your bash history file if you call it from a terminal, but will not show up on the process listings, like on ps, top, htop.

Help info updated

I have appended the password information to the --help info and also added examples on how to set password at the call time.

$ ./cloudsend.sh --help
CloudSender v0.1.5

Parameters:
  -h | --help      Print this help and exits
  -q | --quiet     Be quiet
  -V | --version   Prints version and exits
  -k | --insecure  Uses curl with -k option (https insecure)
  -p | --password  Uses env var $CLOUDSEND_PASSWORD as share password
                   You can 'export CLOUDSEND_PASSWORD' at your system, or set it at the call.
                   Please remeber to also call -p to use the password set.

Use:
  ./cloudsend.sh <filepath> <folderLink>
  CLOUDSEND_PASSWORD='MySecretPass' ./cloudsend.sh -p <filepath> <folderLink>

Example:
  ./cloudsend.sh './myfile.txt' 'https://cloud.mydomain.net/s/fLDzToZF4MLvG28'
   CLOUDSEND_PASSWORD='MySecretPass' ./cloudsend.sh -p './myfile.txt' 'https://cloud.mydomain.net/s/fLDzToZF4MLvG28'

Thanks @MG2R for the password implementation and to @gessel for the late fixes.

Cheers! 🍺
Gus

@tavinus
Copy link
Author

tavinus commented Mar 16, 2020

Hi,
any inclination to release your cloudmanager script ?

Hi!
I have rebuilt my CloudManager App and released the code as public AGPLv3.
https://github.com/tavinus/cloudmanager

Please note that the CloudManager App is NOT a replacement for this CloudSend script.
The tools are actually complimentary.

  • CloudManager is a fully capable webdav client; able to access Nextcloud using a Nextcloud account / password
  • CloudSend is a simple client designed to send files to public shared folders using the shared folder link / password

I will probably eventually include CloudSend into CloudManager, or else I will end up migrating this gist to a full repository.


CloudManager Features

(as of 2020/03/16)

Tries to be feature-complete with the Nextcloud Webdav's capabilities.

Webservice Tasks

  • list file/folder
  • create folder
  • send file
  • get file
  • delete file/folder
  • move file/folder
  • copy file/folder
  • list shares
  • list links

Internal App Goals

  • target folder detection redirection (ends with /)
  • external config files
  • reset config files
  • multiple users
  • multi-language support
  • auto-detect / auto-load system language
  • force language from param

@michimau
Copy link

Ola Gustavo, https://gist.github.com/tavinus/93bdbc051728748787dc22a58dfe58d8#file-cloudsend-sh-L103 breaks up nc 18.0.2.
Reverting to L102, does the trick:

With L103: /usr/bin/curl -T testfile.txt -u 3RZB8ywFcLAaYX8: -H X-Requested-With: XMLHttpRequest https://myserver/s/3RZB8ywFcLAaYX8/public.php/webdav/testfile.txt <- no good
With L102: /usr/bin/curl -T testfile.txt -u 3RZB8ywFcLAaYX8: -H X-Requested-With: XMLHttpRequest https://myserver/public.php/webdav/testfile.txt <- good call

@tavinus
Copy link
Author

tavinus commented Mar 24, 2020

Hi @michimau,

Seems like the problem is not really with the version of Nextcloud, but with how it generates the shared links.

There seems to be 2 different possibilities on how these links are created. I tested on 2 different instances, one running NC18 and another NC16. Funny thing is that the NC16 is the one that created links equal to yours, while the NC18 created different links.

The two possibilities

  1. https://cloud.domain.tld/index.php/s/4TxCwFD7ForMDot
  2. https://cloud.domain.tld/s/CQK4AJt2io4wWmH

So one has /index.php/ and the other one does not.
The ones with index.php require L103 while the others require L102.
Having nextcloud running in a folder seems to be ok in either case (eg. domain.tld/nextcloud)

I am not sure why this is not consistent (why some times it has index.php and some times not).

In any case, I have added a check for the index.php part in the URL and then it will be processed either with what was Line 102 or with what was in Line 103 automagically.

Hopefully this will make it work for everyone and in every version.

I have updated the script to the new version with the fixes ( v0.1.6 ).

Edit: The only thing I can think about is that one server is running on nginx and the other one on Apache (even though it is routed through an nginx proxy as well). So maybe the difference in the link comes from some apache .htaccess or something.

Cheers ☕
Gus

@michimau
Copy link

michimau commented Mar 25, 2020

Thank you for the patch, simple and effective.

I have tested the following setups in docker environments:
haproxy -> nc-apache
haproxy -> nginx -> nc-fpm
and the behavior has been consistent within 18.0.x versions.

@tavinus
Copy link
Author

tavinus commented Mar 25, 2020

I see... who knows, maybe there is a config for that and we are missing it.
Thanks for reporting back.

@Noschvie
Copy link

Hi @tavinus
any way to to create a folder named "my Sub Folder" (filenames with blanks) ?

pi@baumgarten:~/cloudmanager $ ./cloudmanager.sh makeFolder 'my sub folder'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.38 (Debian) Server at 172.18.0.3 Port 80</address>
</body></html>

@tavinus
Copy link
Author

tavinus commented Mar 30, 2020

Hi.
I will answer it here for completeness, but please let us continue this on the cloudmanager issue.


This has actually always been possible, but there is no recursive folder creation.
The parent folder must exist. So you need one call for each folder level to be created.

It should work for files and folders with spaces on both source and target locations.

Names must be quoted or escaped, eg:

  • "my file name"
  • my\ file\ name

Thanks for letting me know of this issue.

Please let me know if this works for you.

Cheers! 🍺
Gus

@C0rn3j
Copy link

C0rn3j commented Apr 12, 2020

I would like to alias or otherwise hide the URL to the shared folder away from the command, so I could simply execute cloudsend file.

Do you think it'd be possible to put the URL behind a flag (-u --url ?) instead of a positional argument so that I could do this in a clean way?

@tavinus
Copy link
Author

tavinus commented Apr 13, 2020

Hi, there are many ways to do that.

You could create a separate script that calls cloudsend.sh $1 <hardcodedUrl>.
So <file> would come from $1 (first parameter) and the URL could be a fixed variable or the URL string.

Example (untested)

#!/bin/bash
CLOUDSENDBIN='/path/to/cloudsend.sh'            # Using full path can avoid errors
SHAREURL='https://cloud.domain.tld/s/xdFvws'    # Your folder share URL

"$CLOUSENDBIN" "$1" "$SHAREURL"

If the share has a password that should also be added.

Example (untested)

#!/bin/bash
CLOUDSENDBIN='/path/to/cloudsend.sh'            # Using full path can avoid errors
SHAREURL='https://cloud.domain.tld/s/xdFvws'    # Your folder share URL

CLOUDSEND_PASSWORD='MySecretPass' "$CLOUSENDBIN" -p "$1" "$SHAREURL"

Or you could modify this gist and hard code the URL inside it as well (around line 103).
https://gist.github.com/tavinus/93bdbc051728748787dc22a58dfe58d8#file-cloudsend-sh-L103
Will have to add a new var and use it in place of $2.

Example mod (untested)

SHAREURL='https://cloud.domain.tld/s/xdFvws'    # Your folder share URL
CLOUDURL=''
# if we have index.php in the URL, process accordingly
if [[ $SHAREURL == *"index.php"* ]]; then
        CLOUDURL="${SHAREURL%/index.php/s/*}"
else
        CLOUDURL="${SHAREURL%/s/*}"
fi

FOLDERTOKEN="${SHAREURL##*/s/}"

I would argue that it is cleaner the way it is now though.
It behaves like copy A B, which makes more sense for me than using a --url parameter.
So I don't think I will change the main script to behave in a different way.

Cheers! 🍺

@C0rn3j
Copy link

C0rn3j commented Apr 13, 2020

Thanks for the lengthy response with examples! I'd prefer to avoid hardcoding modifications or creating secondary scripts though.

I think there's a way that'd satisfy both use cases - how about adding support for both positional parameters and flags?

The script would still be convenient to use by real people, but you could use it in scripts/aliases without hardcoding modifications or ugly workarounds.

I actually went ahead and did just that(along with some other small changes), please take a look - https://haste.rys.pw/raw/afisuwosoz

You can either use the positional arguments like until now, or you can use --file together with --url (have to use both or nothing) and positioning is then ignored.

All the code is there, what's left is to modify usage() to reflect the new changes.

@tavinus
Copy link
Author

tavinus commented Apr 13, 2020

Hi again.
I guess that if you already have a fork that works the way you want, you are good to go.
Like I said, I see no reason to add that complexity and I prefer it the way it is ( copy <source> <destination> ).
Cheers!

@C0rn3j
Copy link

C0rn3j commented Apr 13, 2020

I'm not sure if we're on the same page - copy <source> <destination> is still available with my changes, my point was to not break existing usage, it's just that you can ALSO use parameters for easy use in scripts.

There's no real 'complexity' added, I simply put the Process parameters stage into a case statement (which is cleaner) and added a condition if it encounters an unknown(positional) parameter.

I'd prefer to have my changes merged rather than have to maintain my own fork.

I've updated the URL to a file with the usage() changes already present.

@nerrixde
Copy link

@tavinus You may want to move everything here into a github repo, it's maybe getting too large for a single gist ^^

@tavinus
Copy link
Author

tavinus commented Apr 15, 2020

Migrated to repository

https://github.com/tavinus/cloudsend.sh

This gist will not be updated anymore.

@tavinus
Copy link
Author

tavinus commented May 15, 2020

Rebuild

I have just rebuilt most of the script and added many improvements.
But it is only available at the new repository.
This gist will be kept as is for all eternity.

The only 'downside' is that the old -p option is now -e, because -p now receives the password as a parameter, as in -p <myPass>.

From here to there, this is the current log (Aug 2020)

  • v0.1.7 - Rename
    • Adds option -r / --rename to change the destination (uploaded) file name.
  • v2.0.0 - Rebuild
    • Script rebuild; many internal changes
    • Changes how options and parameters are parsed
    • Added many validation and checking functions
    • Options can now come in any order and after the filename and URL Link
    • Updated and added verbose messages
    • Updated quiet mode
    • Updated --help and readme.md
    • Password parsing has changed and is NOT compatible with prior versions
      • -p now asks for a password as a parameter, as in -p <pass>
      • new option -e uses an envrionment variable password (substitutes the old -p option)
  • v2.1.0 - Pipes and Globs
    • Adds stdin as input option (piped content)
      • uses either - or . as input file name
      • requires -r <filename>
    • Allows to use globbing at the input
      • can send multiple files by using globs
      • requires -g option
      • not compatible with -r <name> option (rename)
    • Adds new checks and parsing
    • Cleaned up unused code
    • Updated readme and --help
  • v2.1.1 - Pipes and Globs (fix)
    • Small fix to file name parsing VS globbing
    • Updated readme globbing link
  • v2.1.2 - Pipes and Globs (readme-help)
    • Better help and readme text and examples.
  • v2.1.3 - Pipes and Globs (fix2)
    • Disables curl globbing when not globbing.

More info on globbing:


CLICK HERE to see how to send an entire folder (and subfolders?) with the new script and find

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment