Skip to content

Instantly share code, notes, and snippets.

@trinitronx

trinitronx/docker-registry-issue-540-reproduction-2.md

Created April 16, 2015 00:21
Show Gist options
  • Save trinitronx/15078746b31d310c1a35 to your computer and use it in GitHub Desktop.
Save trinitronx/15078746b31d310c1a35 to your computer and use it in GitHub Desktop.
Download ZIP
Another Reproduction of docker-archive/docker-registry#540
Raw
docker-registry-issue-540-reproduction-2.md

docker pull Output:

Running on docker daemon/client host with IP 12.34.56.78

2015/04/15 17:01:01 Get https://registry.example.com:443/v1/repositories/library/ansible-base/images: read tcp 33.33.33.33:443: i/o timeout

docker daemon Logs (/var/log/docker):

Running on docker daemon/client host with IP 12.34.56.78

[003b8841] -job image_delete(cb5f50a9c417) = OK (0)
[info] POST /v1.15/images/create?fromImage=registry.example.com%3A443%2Flibrary%2Fansible-base%3Av1.7.2
[003b8841] +job pull(registry.example.com:443/library/ansible-base, v1.7.2)
Get https://registry.example.com:443/v1/repositories/library/ansible-base/images: read tcp 33.33.33.33:443: i/o timeout
[003b8841] -job pull(registry.example.com:443/library/ansible-base, v1.7.2) = ERR (1)
[info] POST /v1.15/images/create?fromImage=registry.example.com%3A443%2Flibrary%2Fansible-base%3Av1.7.2
[003b8841] +job pull(registry.example.com:443/library/ansible-base, v1.7.2)
[info] GET /v1.15/containers/json
[003b8841] +job containers()
[003b8841] -job containers() = OK (0)

Manual curl Output for 2nd Request

I tried to duplicate the same request on another host just using cURL just after the first one failed on the docker daemon / client host. Here is the output of 2 successive cURL commands. The first was a failure, with same 504 Gateway Timeout error. The second was a successful request. I didn't do anything in between requests except wait a couple seconds.

Ran these cURL commands on Laptop Host with IP 22.22.22.22:

# 1st cURL Command run at approximately: 15/Apr/2015:23:41:02 +0000

$ curl -v -s https://registry.example.com/v1/repositories/library/ansible-base/images
* Adding handle: conn: 0x7fe5cc00aa00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fe5cc00aa00) send_pipe: 1, recv_pipe: 0
* About to connect() to registry.example.com port 443 (#0)
*   Trying 33.33.33.33...
* Connected to registry.example.com (33.33.33.33) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: *.example.com
* Server certificate: DigiCert High Assurance CA-3
* Server certificate: DigiCert High Assurance EV Root CA
* Server certificate: Baltimore CyberTrust Root
> GET /v1/repositories/library/ansible-base/images HTTP/1.1
> User-Agent: curl/7.30.0
> Host: registry.example.com
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
* Server nginx/1.7.8 is not blacklisted
< Server: nginx/1.7.8
< Date: Wed, 15 Apr 2015 23:41:07 GMT
< Content-Type: text/html
< Content-Length: 182
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.7.8</center>
</body>
</html>
* Connection #0 to host registry.example.com left intact

# 2nd cURL Command run at: 15/Apr/2015:23:41:45 +0000

$ curl -v -s https://registry.example.com/v1/repositories/library/ansible-base/images
* Adding handle: conn: 0x7fa984004000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fa984004000) send_pipe: 1, recv_pipe: 0
* About to connect() to registry.example.com port 443 (#0)
*   Trying 33.33.33.33...
* Connected to registry.example.com (33.33.33.33) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: *.example.com
* Server certificate: DigiCert High Assurance CA-3
* Server certificate: DigiCert High Assurance EV Root CA
* Server certificate: Baltimore CyberTrust Root
> GET /v1/repositories/library/ansible-base/images HTTP/1.1
> User-Agent: curl/7.30.0
> Host: registry.example.com
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx/1.7.8 is not blacklisted
< Server: nginx/1.7.8
< Date: Wed, 15 Apr 2015 23:41:45 GMT
< Content-Type: application/json
< Content-Length: 988
< Connection: keep-alive
< X-Docker-Token: Token signature=03FEYD473D225FQ1,repository="library/ansible-base",access=read
< X-Docker-Endpoints: registry.example.com
< Pragma: no-cache
< Cache-Control: no-cache
< Expires: -1
< WWW-Authenticate: Token signature=03FEYD473D225FQ1,repository="library/ansible-base",access=read
< Strict-Transport-Security: max-age=31536000; includeSubdomains
<
* Connection #0 to host registry.example.com left intact
[{"id": "494842a095dc42bdfc504628b69fd8b9f7a4ec1c19304bf97c4ac9a50d98d235"}, {"id": "7a4981346fd5e1eded40b59bb923143a27bde5b4bd4a6158cd116870ef45d650"}, {"id": "c536bbcc73a8b5c4c75512fa71fad27f2335737b4237c07aa2bcefc891351e3a"}, {"id": "cab29f700de3074279341eb99a101feee76d62fbd4e0b6b80c8e8d7250cac61e"}, {"id": "7d4f8ebd21d70394b5b5682be65a66745862ab02b1d7a67f4a61f71908bb958e"}, {"id": "89cebd3de749c9654fdfa413f668eef0d979f9f2ca90eea8882ca431a0ab60e1"}, {"id": "898d4e5477072c559d8ac537ee5c4351b417f538d3fa4d3bfb98813087086842"}, {"id": "8b2c46d59b1320ad46025f24b4ab6db2551a49f0c9d3a6b35e90d2bfbdc842f2"}, {"id": "9181048b41d21e1c9d6276add6bca40fb8f841027717ff492ada109c32642750"}, {"id": "2b4210d6a4eb49fda54d3967a829572384072b50cd435503c6d577ef4c9a8763"}, {"id": "21769647d5a97c884e59b94b523b11612162410054a702f59a77db5fe148ae57"}, {"id": "1ff992c29d25a28361748361897e4cc395e84b43563951db64c9754f099b560a"}, {"id": "710ac9edc9db4774dd4d318fcd488c89effb58a3e45736eb9b8f0146e859fa75"}]

nginx-proxy Logs:

Running on registry.example.com host with IP 33.33.33.33

nginx.1    | registry.example.com 12.34.56.78 - - [15/Apr/2015:23:00:01 +0000] "GET /v1/_ping HTTP/1.1" 200 1603 "-" "Go 1.1 package http"
nginx.1    | 12.34.56.78 - - [15/Apr/2015:23:00:01 +0000] "GET /v1/_ping HTTP/1.1" 200 1603 "-" "Go 1.1 package http" "-"
nginx.1    | registry.example.com 12.34.56.78 - - [15/Apr/2015:23:00:01 +0000] "GET /v1/_ping HTTP/1.1" 200 1603 "-" "Go 1.1 package http"
nginx.1    | 2015/04/15 23:01:01 [error] 188#0: *3881 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 12.34.56.78, server: registry.example.com, request: "GET /v1/repositories/library/ansible-base/images HTTP/1.1", upstream: "http://172.17.0.18:5000/v1/repositories/library/ansible-base/images", host: "registry.example.com:443"
nginx.1    | 12.34.56.78 - - [15/Apr/2015:23:01:01 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 504 182 "-" "docker/1.3.3 go/go1.3.3 git-commit/d344625 kernel/2.6.32-431.el6.x86_64 os/linux arch/amd64" "-"
nginx.1    | registry.example.com 12.34.56.78 - - [15/Apr/2015:23:01:01 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 504 182 "-" "docker/1.3.3 go/go1.3.3 git-commit/d344625 kernel/2.6.32-431.el6.x86_64 os/linux arch/amd64"


nginx.1    | 2015/04/15 23:41:07 [error] 188#0: *3883 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 22.22.22.22, server: registry.example.com, request: "GET /v1/repositories/library/ansible-base/images HTTP/1.1", upstream: "http://172.17.0.18:5000/v1/repositories/library/ansible-base/images", host: "registry.example.com"
nginx.1    | 22.22.22.22 - - [15/Apr/2015:23:41:07 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 504 182 "-" "curl/7.30.0" "-"
nginx.1    | registry.example.com 22.22.22.22 - - [15/Apr/2015:23:41:07 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 504 182 "-" "curl/7.30.0"
nginx.1    | 22.22.22.22 - - [15/Apr/2015:23:41:45 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 200 988 "-" "curl/7.30.0" "-"
nginx.1    | registry.example.com 22.22.22.22 - - [15/Apr/2015:23:41:45 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 200 988 "-" "curl/7.30.0"

docker-registry Logs:

Running on registry.example.com host with IP 33.33.33.33

172.17.0.27 - - [15/Apr/2015:23:00:01 +0000] "GET /v1/_ping HTTP/1.1" 200 1603 "-" "Go 1.1 package http"
172.17.0.27 - - [15/Apr/2015:23:00:01 +0000] "GET /v1/_ping HTTP/1.1" 200 1603 "-" "Go 1.1 package http"


2015-04-15 23:41:44,994 ERROR: Exception on /v1/repositories/library/ansible-base/images [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 306, in wrapper
    return f(namespace=namespace, repository=repository, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 280, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/index.py", line 112, in get_repository_images
    data = store.get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/s3.py", line 165, in get_content
    return super(Storage, self).get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/boto.py", line 220, in get_content
    if not key.exists():
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 537, in exists
    return bool(self.bucket.lookup(self.name, headers=headers))
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 142, in lookup
    return self.get_key(key_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
S3ResponseError: S3ResponseError: 403 Forbidden

15/Apr/2015:23:41:44 +0000 ERROR: Exception on /v1/repositories/library/ansible-base/images [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 306, in wrapper
    return f(namespace=namespace, repository=repository, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 280, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/index.py", line 112, in get_repository_images
    data = store.get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/s3.py", line 165, in get_content
    return super(Storage, self).get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/boto.py", line 220, in get_content
    if not key.exists():
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 537, in exists
    return bool(self.bucket.lookup(self.name, headers=headers))
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 142, in lookup
    return self.get_key(key_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
S3ResponseError: S3ResponseError: 403 Forbidden

[2015-04-15 23:41:45 +0000] [91] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 108, in handle_request
    resp.write(item)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 344, in write
    util.write(self.sock, arg, self.chunked)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 301, in write
    sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 458, in sendall
    data_sent += self.send(_get_memory(data, data_sent), flags)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 435, in send
    return sock.send(data, flags)
error: [Errno 32] Broken pipe
[2015-04-15 23:41:45 +0000] [91] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 108, in handle_request
    resp.write(item)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 344, in write
    util.write(self.sock, arg, self.chunked)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 301, in write
    sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 458, in sendall
    data_sent += self.send(_get_memory(data, data_sent), flags)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 435, in send
    return sock.send(data, flags)
error: [Errno 32] Broken pipe
172.17.0.27 - - [15/Apr/2015:23:41:45 +0000] "GET /v1/repositories/library/ansible-base/images HTTP/1.1" 200 988 "-" "curl/7.30.0"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment