Strange git-annex behaviour

runme

#!/bin/bash

#=======================================================================
# runme
# File ID: 36b615f8-f051-11e4-a442-fefdb24f8e10
#
# Demonstration of strange behaviour in git-annex, README is deleted 
# after a "git annex get" when --file is used together with "git annex 
# addurl". It's also unable to check the existence of the file on the 
# FTP server, so I have to use --force to drop it. This does not happen 
# when using an http url. Have tested with versions back to v5.20131221, 
# they all behave the same.
#
# Using newest git-annex version from 
# <https://downloads.kitenet.net/.git/> in directory 
# git-annex/linux/current/ .
#
# $ git annex version
# git-annex version: 5.20150420-gb0ebb23
# build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 
# WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA
# key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E 
# SKEIN512E MD5E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 MD5 
# WORM URL
# remote types: git gcrypt S3 bup directory rsync web bittorrent webdav 
# tahoe glacier ddar hook external
# local repository version: unknown
# supported repository version: 5
# upgrade supported from repository versions: 0 1 2 4
#
# $ git --version
# git version 2.4.0
#
# $ arch
# x86_64
#
# $ lsb_release -a
# No LSB modules are available.
# Distributor ID: Debian
# Description:    Debian GNU/Linux 7.8 (wheezy)
# Release:        7.8
# Codename:       wheezy
#
# Author: Øyvind A. Holm <sunny@sunbase.org>
# License: GNU General Public License version 2 or later.
#=======================================================================

repo=tmpdirawedsfkn
unset debugstr filestr use_http_instead

# Uncomment for debug info
# debugstr="--debug"

# Comment out to disable use of --file
filestr="--file=README"

# Uncomment to test with http instead
# use_http_instead=true

msg() {
    echo
    echo ================== $* ==================
}

test -d $repo && chmod -R +w $repo && rm -rf $repo

msg git init &&
git init $repo &&
cd $repo &&
msg git annex init &&
git annex init &&
msg git commit, empty start commit &&
git commit --allow-empty -m "Empty startcommit" &&
{
        msg git annex addurl &&
        test "$use_http_instead" = "true" &&
        git annex addurl http://sunbase.org/ga-test/README $filestr ||
        git annex addurl ftp://ftp.funet.fi/pub/Linux/mirrors/debian/README $filestr
} &&
msg git commit, add README &&
git commit -m "Add README" &&
msg git annex drop --force &&
git annex drop --force &&
msg git annex get $debugstr &&
git annex get $debugstr && # This is where the mystery happens
msg ls -l &&
ls -l &&
echo &&
{
    ls | grep -q README &&
    echo README exists, life is good ||
    echo README is gone, should not happen
} &&
echo &&
echo Reached the end

wiki-bugreport.md

Please describe the problem.

When using "git annex addurl --file" with an ftp url, the committed file is deleted after dropping the contents with --force (because git-annex can't determine if the ftp server contains a valid copy) and executing "git annex get". It's the "git annex get" command that deletes the file.

This does not happen when using an http url.

What steps will reproduce the problem?

git clone https://gist.github.com/sunny256/24f6c29645efd0aab4d9

and execute the bash script runme. There's more info in a long comment there, plus various flags you can enable/disable to test under different conditions.

What version of git-annex are you using? On what operating system?

Using the newest git-annex from https://downloads.kitenet.net/.git/ in directory git-annex/linux/current/, 5.20150420-gb0ebb23.

Have tested with versions way back to v5.20131221, they all behave the same.

Using Debian GNU/Linux 7.8 (wheezy) on x86_64 with brand new git 2.4.0.

Please provide any additional information below.

[[!format sh """

If you can, paste a complete transcript of the problem occurring here.

If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

$ ./runme

================== git init ================== Initialized empty Git repository in /home/sunny/src/git/ga-bug/tmpdirawedsfkn/.git/

================== git annex init ================== init ok (recording state in git...)

================== git commit, empty start commit ================== [master (root-commit) 6d5d623] Empty startcommit

================== git annex addurl ================== addurl README (downloading ftp://ftp.funet.fi/pub/Linux/mirrors/debian/README ...) --2015-05-02 03:28:59-- ftp://ftp.funet.fi/pub/Linux/mirrors/debian/README => '.git/annex/tmp/URL--ftp&c%%ftp.funet.fi%pub%Linux%mirrors%debian%README' Resolving ftp.funet.fi (ftp.funet.fi)... 193.166.3.2, 2001:708:10:9::20:2 Connecting to ftp.funet.fi (ftp.funet.fi)|193.166.3.2|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pub/Linux/mirrors/debian ... done. ==> SIZE README ... 1495 ==> PASV ... done. ==> RETR README ... done. Length: 1495 (1.5K) (unauthoritative)

100%[================================================>] 1,495 --.-K/s in 0.01s

2015-05-02 03:29:00 (125 KB/s) - '.git/annex/tmp/URL--ftp&c%%ftp.funet.fi%pub%Linux%mirrors%debian%README' saved [1495]

ok (recording state in git...)

================== git commit, add README ================== [master 264d597] Add README 1 file changed, 1 insertion(+) create mode 120000 README

================== git annex drop --force ================== drop README ok (recording state in git...)

================== git annex get ================== get README (from web...) --2015-05-02 03:29:00-- ftp://ftp.funet.fi/pub/Linux/mirrors/debian/README => '.git/annex/tmp/SHA256-s1495--8822780b87a880ca9956ac108812557044618859cecb07df488df57e8134e34f' Resolving ftp.funet.fi (ftp.funet.fi)... 193.166.3.2, 2001:708:10:9::20:2 Connecting to ftp.funet.fi (ftp.funet.fi)|193.166.3.2|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pub/Linux/mirrors/debian ... done. ==> SIZE README ... 1495 ==> PASV ... done. ==> RETR README ... done. Length: 1495 (1.5K) (unauthoritative)

100%[================================================>] 1,495 --.-K/s in 0s

2015-05-02 03:29:02 (73.1 MB/s) - '.git/annex/tmp/SHA256-s1495--8822780b87a880ca9956ac108812557044618859cecb07df488df57e8134e34f' saved [1495]

ok (recording state in git...)

================== ls -l ================== total 0

README is gone, should not happen

Reached the end $

End of transcript or log.

"""]]