Skip to content

Instantly share code, notes, and snippets.

@guycalledseven
Last active March 28, 2022 19:49
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save guycalledseven/dbe3900fe04fd6104abf09f078f0a015 to your computer and use it in GitHub Desktop.
bizarre nginx issues caused by incorrect file permissions

issues experienced

nginx error.log shows permission denied

nginx log in /var/log/nginx/error.log shows intereseting Error 13: Permission denied:

2017/11/14 00:33:30 [crit] 5117#5117: *224 open() "/var/lib/nginx/proxy/3/02/0000000023" failed (13: Permission denied) while reading upstream, client: 93.136.115.2, server: git.domain.tld, request: "GET /css/semantic-2.2.10.min.css HTTP/1.1", upstream: "http://127.0.0.1:3000/css/semantic-2.2.10.min.css", host: "git.domain.tld"

random images/css styles not loading

Browser reports broken link. curl reports 200 ok. (!!!?) after 10 force reloads of web page, maybe everything will load correctly. expirienced via ssl/non-ssl/http1 or http2.

upstream git server ssl issues on older libssl (ubuntu/windows)

rpc failed error 56

$ git clone https://git.domain.tld/seven/test.git .
Cloning into '.'...
Username for 'https://git.domain.tld': seven
Password for 'https://seven@git.domain.tld':
remote: Counting objects: 1359, done.
remote: Compressing objects: 100% (1332/1332), done.
error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

exit status 141 - {error: git-upload-pack died of signal 13

2017/11/13 21:02:58 http: multiple response.WriteHeader calls
2017/11/13 21:02:58 [ERROR] [.../routes/repo/http.go:284 serviceRPC()] HTTP.serviceRPC: fail to serve RPC 'upload-pack': exit status 141 - {error: git-upload-pack died of signal 13
 %!s(int=0)  %!s(bytes.readOp=0)}
[Macaron] 2017-11-13 21:02:58: Completed POST /seven/test.git/git-upload-pack 500 Internal Server Error in 126.261691ms

Stack overflow did not help much, since for this error - there is 100 possible reasons listed: https://stackoverflow.com/questions/15240815/git-fatal-the-remote-end-hung-up-unexpectedly

"error: git-upload-pack died of signal 13" https://confluence.atlassian.com/bitbucketserverkb/error-git-upload-pack-died-of-signal-13-779171770.html The error above does not represent an issue with Server. It means that a Git client has closed its connection prematurely.

Solution

  1. Find out the owner of nginx process:
ps aux | grep "nginx: worker process"
nginx     5117  0.0  0.1 125644  6676 ?        S    00:30   0:00 nginx: worker process
  1. Check if owner of process is the same as owner of folders/files:
sudo ls -l /var/lib/nginx/proxy/
total 40
drwx------  4 nginx root 4096 Nov 14 00:51 0
drwx------  5 nginx root 4096 Jan 19  2015 1
drwx------  5 nginx root 4096 Jan 19  2015 2
drwx------  5 nginx root 4096 Jan 19  2015 3
drwx------  5 nginx root 4096 Nov 15 10:33 4
drwx------  5 nginx root 4096 Nov 15 10:33 5
drwx------  5 nginx root 4096 Nov 15 12:19 6
drwx------  5 nginx root 4096 Nov 15 12:19 7
drwx------  5 nginx root 4096 Nov 15 12:19 8
drwx------ 49 nginx root 4096 Nov 13 20:09 9
  1. Depending of step 1, reset file permissions:
sudo chown -R nginx:root /var/lib/nginx/proxy/

or:

sudo chown -R www-data:root /var/lib/nginx/proxy/

and restart service:

sudo systemctl restart nginx

Cause

nginx log in /var/log/nginx/error.log shows intereseting Error 13: Permission denied:

2017/11/14 00:33:30 [crit] 5117#5117: *224 open() "/var/lib/nginx/proxy/3/02/0000000023" failed (13: Permission denied) while reading upstream, client: 93.136.115.2, server: git.domain.tld, request: "GET /css/semantic-2.2.10.min.css HTTP/1.1", upstream: "http://127.0.0.1:3000/css/semantic-2.2.10.min.css", host: "git.domain.tld"

While upgrading from Ubuntu 14.04 LTS to 16.04, nginx migration did not reset file permissions correctly in nginx temp folder /var/lib/nginx/proxy/. This cause some content to be available, and some not - all extremely hard to debug.

Notice folder 5 with owner root:root:

sudo ls -l /var/lib/nginx/proxy/
total 40
drwx------  4 nginx root 4096 Nov 14 00:51 0
drwx------  5 nginx root 4096 Jan 19  2015 1
drwx------  5 nginx root 4096 Jan 19  2015 2
drwx------  5 nginx root 4096 Jan 19  2015 3
drwx------  5 nginx root 4096 Nov 15 10:33 4
drwx------  5 root root 4096 Nov 15 10:33 5
drwx------  5 nginx root 4096 Nov 15 12:19 6
drwx------  5 nginx root 4096 Nov 15 12:19 7
drwx------  5 nginx root 4096 Nov 15 12:19 8
drwx------ 49 nginx root 4096 Nov 13 20:09 9

debug git

Run once to get full debug:

export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1

Add GIT_CURL_VERBOSE=1 before your git command:

GIT_CURL_VERBOSE=1 git clone https://git.domain.tkd/seven/test.git .
@pfrayer
Copy link

pfrayer commented Jun 22, 2018

I had the same issue and your solution fixed it 😃 Thanks for the root cause explanation

@fraschbi
Copy link

Very helpful. Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment