Linux df and du mismatching

du-df-mismatch.txt

The df -h on the partition /data was showing full usage of 20GB but du -sh /data was showing a usage of only 581MB.

ubuntu@ip-xxx-xx-x-xx:/data$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/xvdb        20G   20G  0G  100% /data

ubuntu@ip-xxx-xx-x-xx:/data$ du -sh .
581M	.

This huge difference means that files are deleted from filesystem but are still referenced from some open applications.

There were no results of deleted files after running the following command.
ubuntu@ip-xxx-xx-x-xx:/data$ lsof -nP | grep '(deleted)'
ubuntu@ip-xxx-xx-x-xx:/data$

After few minutes of confusion I tried running the same command with sudo and there it was.

ubuntu@ip-xxx-xx-x-xx:/data$ sudo lsof -nP | grep deleted
sh         5267             root  cwd       DIR             202,16          0     393217 /data/xxxxxxxxxxxxx (deleted)
sh         5267             root    2w      REG             202,16 3742909615     397337 /data/xxxxxxxxxxxxx/nohup.out (deleted)
tail       5270             root  cwd       DIR             202,16          0     393217 /data/xxxxxxxxxxxxx (deleted)
tail       5270             root    2w      REG             202,16 3742909615     397337 /data/xxxxxxxxxxxxx/nohup.out (deleted)
tail       5270             root    3r      REG             202,16 1196962556     397656 /data/xxxxxxxxxxxxx/log/production.log (deleted)
tail      14934             root    2w      REG             202,16    1424371     398138 /data/xxxxxxxxxxxxx/nohup.out~ (deleted)
tail      14934             root    3r      REG             202,16 9281997539     659099 /data/xxxxxxxxxxxxx/log/production.log.1 (deleted)
tail      19409             root    2w      REG             202,16    1424371     398138 /data/xxxxxxxxxxxxx/nohup.out~ (deleted)
tail      19409             root    3r      REG             202,16 4805248107     655668 /data/xxxxxxxxxxxxx/log/production.log.9 (deleted)

Looking at the corresponding processes, I found that there were open processes that were holding references to the old files. 
ubuntu@ip-xxx-xx-x-xx:/data$ ps -ef | grep 5267
ubuntu    2712  1616  0 13:13 pts/3    00:00:00 grep --color=auto 5267
root      5267     1  0 Jan10 ?        00:00:00 sh -c tail -f -n 0 "/data/xxxxxxxxxxxxx/log/production.log"
root      5270  5267  0 Jan10 ?        00:00:00 tail -f -n 0 /data/xxxxxxxxxxxxx/log/production.log

After killing all of such process, df and du matched completely and 18GB of disk space got freed up.
ubuntu@ip-xxx-xx-x-xx:/data$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/xvdb        20G  625M   18G   4% /data

Hurray!