Missing image layer in a docker registry #docker #registry

Missing image layer in a docker registry.md

Missing image layer in a docker registry

$ docker push registry.com/image/name:master
The push refers to repository [registry.com/image/name]
9be3d32c4bdc: Preparing
a17a1099bdbb: Preparing
896990f929fd: Preparing
74cfeaa3d1d8: Preparing
dd2b13d45ffc: Preparing
ff64ee97d76a: Preparing
480f61641fa1: Preparing
b3eaed7a085d: Preparing
1a058d5342cc: Preparing
ff64ee97d76a: Waiting
b3eaed7a085d: Waiting
1a058d5342cc: Waiting
480f61641fa1: Waiting
896990f929fd: Layer already exists
9be3d32c4bdc: Layer already exists
a17a1099bdbb: Layer already exists
dd2b13d45ffc: Layer already exists
74cfeaa3d1d8: Layer already exists
480f61641fa1: Layer already exists
ff64ee97d76a: Layer already exists
1a058d5342cc: Layer already exists
b3eaed7a085d: Layer already exists
master: digest: sha256:5b784cb8080660d7af7222a6b5b0630faa6942b39f0c533354500e946bf92c7c size: 2201

$ docker pull registry.com/image/name:master
master: Pulling from image/name
97518928ae5f: Already exists
784cd1fd612b: Pull complete
0ec5d186b713: Pull complete
98dc27ad6276: Pull complete
94d536c3c847: Downloading [===============================================>   ]  116.2MB/123.2MB
56a018b1cd83: Download complete
89370282bef7: Download complete
eaf17779ae5d: Downloading [==================================================>]   2.59MB/2.59MB
e031dab02d72: Download complete
unexpected EOF

$ journalctl -eu docker
...
Dec 02 07:57:19 yuri2 dockerd[377]: time="2021-12-02T07:57:19.373399892+02:00" level=info msg="Download failed, retrying (1/5): unexpected EOF"
Dec 02 07:57:24 yuri2 dockerd[377]: time="2021-12-02T07:57:24.790823039+02:00" level=info msg="Download failed, retrying (2/5): unexpected EOF"
Dec 02 07:57:35 yuri2 dockerd[377]: time="2021-12-02T07:57:35.195908802+02:00" level=info msg="Download failed, retrying (3/5): unexpected EOF"
Dec 02 07:57:50 yuri2 dockerd[377]: time="2021-12-02T07:57:50.804338597+02:00" level=info msg="Download failed, retrying (4/5): unexpected EOF"
Dec 02 07:58:11 yuri2 dockerd[377]: time="2021-12-02T07:58:11.223387541+02:00" level=info msg="Download failed, retrying (5/5): unexpected EOF"
Dec 02 07:58:36 yuri2 dockerd[377]: time="2021-12-02T07:58:36.990099990+02:00" level=error msg="Download failed after 6 attempts: unexpected EOF"
Dec 02 07:58:36 yuri2 dockerd[377]: time="2021-12-02T07:58:36.990392346+02:00" level=info msg="Attempting next endpoint for pull after error: unexpected EOF"
Dec 02 07:58:36 yuri2 dockerd[377]: time="2021-12-02T07:58:36.999236138+02:00" level=info msg="Layer sha256:a05eb5e84c6df5c43b46a2644bb0f98f2d8b671ad2f122087d2cbb5ec8528c11 cleaned up"
Dec 02 07:58:36 yuri2 dockerd[377]: time="2021-12-02T07:58:36.999305071+02:00" level=info msg="Layer sha256:b0f4cb46c82c41cac44c05bcfb9a2f1ba2e17edeedaccc4607a80008d2ce6eab cleaned up"
Dec 02 07:58:37 yuri2 dockerd[377]: time="2021-12-02T07:58:37.109161852+02:00" level=info msg="Layer sha256:335bf1b2785623ce882dba849ea5ad9227bd50c1a3e94488599a749c0a22ebfb cleaned up"

$ curl -sSL https://registry.com/v2/_catalog | jq -r .repositories[]
...
image/name
...

$ curl -sSL https://registry.com/v2/image/name/tags/list | jq
{
  "name": "image/name",
  "tags": [
    "master",
    "develop"
  ]
}

$ curl -sSL https://registry.com/v2/image/name/manifests/master | jq -r '.fsLayers[] | .blobSum'
...
sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d
sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca
...

$ curl -sSLv https://registry.com/v2/image/name/blobs/sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d -o /dev/null
...
> GET /v2/image/name/blobs/sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d HTTP/2
> Host: registry.com
> authorization: Basic ...
> user-agent: curl/7.79.1
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
< HTTP/2 200 
< accept-ranges: bytes
< cache-control: max-age=31536000
< content-type: application/octet-stream
< date: Thu, 02 Dec 2021 06:16:51 GMT
< docker-content-digest: sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d
< docker-distribution-api-version: registry/2.0
< etag: "sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d"
< server: nginx
< x-content-type-options: nosniff
< content-length: 553
< 
{ [5 bytes data]
* Connection #0 to host registry.com left intact

$ curl -sSL https://registry.com/v2/image/name/blobs/sha256:e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d | tar tzf -
some-dir/
some-dir/some-file

$ ssh root@registry.com docker exec 0728b0432c2d cat /var/lib/registry/docker/registry/v2/blobs/sha256/e0/e031dab02d727ce644ab4ea42a26e0f1661c01049a2197b9d58e22c44fecb03d/data | tar tzf -
some-dir/
some-dir/some-file

$ curl -sSLv https://registry.com/v2/image/name/blobs/sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca -o /dev/null
...
> GET /v2/image/name/blobs/sha256:eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca HTTP/2
> Host: registry.com
> authorization: Basic ...
> user-agent: curl/7.79.1
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host registry.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

$ ssh root@registry.com docker exec 0728b0432c2d ls /var/lib/registry/docker/registry/v2/blobs/sha256/ea/eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca
ls: /var/lib/registry/docker/registry/v2/blobs/sha256/ea/eaf17779ae5dc4e74e2aba3dcefc629f5b5717adb20facc93c6e726664637aca: No such file or directory