Skip to content

Instantly share code, notes, and snippets.

@abdennour
Last active June 19, 2024 10:38
Show Gist options
  • Save abdennour/74c5de79e57a47f3351217d674238da8 to your computer and use it in GitHub Desktop.
Save abdennour/74c5de79e57a47f3351217d674238da8 to your computer and use it in GitHub Desktop.
Nginx Reverse Proxy for Nexus Docker Registries

Overview

This is a solution of a common problem with Nexus Docker repositories. The administrator has to expose port for "pull", another port for "push", other ports for each hosted repository. This solution is about leveraging Nginx reverse proxy to avoid using these ports.

How it works ?

Given :

  • Nexus hostname is "nexus.example.com"
  • Nexus web port is 8081
  • A hosted repository is named "docker-hosted"
  • A group repository is named "docker-group"
  • Your nginx (with the nginx.conf of this gist) will run for example under cregistry.example.com

The following Nginx configuration file is for a reverse proxy without the need to expose connector ports from nexus :

  • docker pull cregistry.example.com/myimage lets Nginx forward the request to "docker-group"
  • docker push cregistry.example.com/myimage lets Nginx forward the request to "docker-hosted"

Notes

  • If you have more than one hosted repository, create another Nginx reverse proxy for it, then aggregate them using a parent Nginx reverse proxy that forwards the request according to certain criteria (.i.e: Host header).

  • All Nexus repositories must have consistent configuration of authentication: Either all require authentication, or all don't.

  • If TLS is enabled with Nexus, change proxy_set_header X-Forwarded-Proto "http"; by proxy_set_header X-Forwarded-Proto "https";

version: "3"
services:
web:
image: nginx:1.15
hostname: cregistry.example.com
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
ports:
- "80:80"
nexus:
image: sonatype/nexus3
hostname: nexus.example.com
volumes:
- "nexus-data:/nexus-data"
ports:
- "8081:8081"
volumes:
nexus-data: {}
events {
worker_connections 1024;
}
http {
proxy_send_timeout 120;
proxy_read_timeout 300;
proxy_buffering off;
keepalive_timeout 5 5;
tcp_nodelay on;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
server {
listen *:80;
location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
}
rewrite ^/(.*)$ /repository/docker-group/$1 last;
}
location ~ ^/(v1|v2)/ {
if ($request_method ~* (POST|PUT|DELETE|PATCH) ) {
rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
}
rewrite ^/(.*)$ /repository/docker-group/$1 last;
}
location / {
proxy_pass http://nexus.example.com:8081/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "http";
}
}
}
@AlexGluck
Copy link

AlexGluck commented Jun 3, 2022

@raducanuoctavian
UPD: Add feature like in hub.docker.com. For pull image from nexus hosted docker repository without set docker repository name.
Create required docker hosted repository library.
Feature in next block:

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
      proxy_pass http://nexus-node/repository/library/$1/library/library/$2/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_valid      404 10m;
      proxy_cache_use_stale  http_500;
    }

Example: docker pull nexus.example.org/image:latest

If image not found fallback search in docker-group.

@Wadzio
Copy link

Wadzio commented Sep 5, 2022

I don't like the "main" folder with the same name as the docker repo name (like in the screen below):
2022-08-29_22h33_44

This is my version to prevent this "main" folder:

...
...

map $upstream_http_location $upstream_docker_version {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(?<version>v1|v2)/([-_0-9a-z\.]+)/(.*)$" $version;
}
map $upstream_http_location $upstream_docker_repo_name {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(v1|v2)/(?<repo_name>[-_0-9a-z\.]+)/(.*)$" $repo_name;
}
map $upstream_http_location $upstream_docker_rest_uri {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(v1|v2)/([-_0-9a-z\.]+)/(?<rest_uri>.*)$" $rest_uri;
}

map $uri $docker_repo_name_in {
    "~^/(v1|v2)/(?<repo_name>[-_0-9a-z\.]+)/(.*)$" $repo_name;
}

map $upstream_docker_repo_name:$docker_repo_name_in $response_header_location {
    "~^(.*):\1$" $upstream_http_location;
    default /$upstream_docker_version/$docker_repo_name_in/$upstream_docker_repo_name/$upstream_docker_rest_uri;
}

upstream nexus-node {
    server localhost:8081 max_fails=0;
    keepalive 150;
    keepalive_timeout 60s;
    keepalive_requests 1000;
}

server {
    listen 80;

    ...
    ...

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/(blobs/sha256.*|manifests/.*)$ {
        proxy_pass http://nexus-node/repository/$2/$1/$3/$4$is_args$args;
        proxy_hide_header Location;
        add_header Location $response_header_location always;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z]+)/(.*) {
        proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
        proxy_hide_header Location;
        add_header Location $response_header_location always;
    }

    location ~ ^/(v1|v2)/?$ {
        proxy_pass http://nexus-node/repository/docker-login/$1/$2$is_args$args;
    }

    ...
    ...
}

...
...

@AlexGluck
Copy link

@Wadzio This awesome, thanks ! 👍

@pubyun
Copy link

pubyun commented Sep 12, 2022

@Wadzio great! would you like to paste complete nginx config?

@AlexGluck
Copy link

@pubyun

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
events {
  worker_connections  1024;
}
http {

  proxy_send_timeout 120;
  proxy_read_timeout 300;
  proxy_buffering    off;
  keepalive_timeout  5 5;
  tcp_nodelay        on;
  client_max_body_size 0;
  chunked_transfer_encoding on;
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=nexus:100m inactive=30d  max_size=2g;

  upstream nexus-node {
    server nexus:8081 max_fails=0;
    keepalive 150;
    keepalive_timeout 60s;
    keepalive_time 1h;
    keepalive_requests 1000;
  }

map $upstream_http_location $upstream_docker_version {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(?<version>v1|v2)/([-_0-9a-z\.]+)/(.*)$" $version;
}
map $upstream_http_location $upstream_docker_repo_name {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(v1|v2)/(?<repo_name>[-_0-9a-z\.]+)/(.*)$" $repo_name;
}
map $upstream_http_location $upstream_docker_rest_uri {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(v1|v2)/([-_0-9a-z\.]+)/(?<rest_uri>.*)$" $rest_uri;
}

map $uri $docker_repo_name_in {
    "~^/(v1|v2)/(?<repo_name>[-_0-9a-z\.]+)/(.*)$" $repo_name;
}

map $upstream_docker_repo_name:$docker_repo_name_in $response_header_location {
    "~^(.*):\1$" $upstream_http_location;
    default /$upstream_docker_version/$docker_repo_name_in/$upstream_docker_repo_name/$upstream_docker_rest_uri;
}

  server {
    listen  80;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    location ~ ^/api/(.*) {
      proxy_pass http://nexus-node/$1$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
      proxy_pass http://nexus-node/repository/library/$1/library/library/$2/$3$is_args$args;
      proxy_hide_header Location;
      add_header Location $response_header_location always;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location ~ ^/(v1|v2)/?$ {
      proxy_pass http://nexus-node/repository/docker-login/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/(_catalog|search)$ {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      proxy_hide_header Location;
      add_header Location $response_header_location always;
      error_page 400 404 500 = @fallback;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      400 500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback2 {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location / {
      proxy_pass http://nexus-node/;
    }
  }
}

@pubyun
Copy link

pubyun commented Sep 12, 2022

@AlexGluck thanks.

docker push docker.example.com/repo1/ubuntu <<<
denied: Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

@AlexGluck
Copy link

@pubyun Please read my comment, you need create required repos.

@a-langer
Copy link

@AlexGluck, @Wadzio thank you for your work. I released patch for Nexus OSS with authorization via SSO and tokens https://github.com/a-langer/nexus-sso, which includes these fixes for Docker registries.

@Alceatraz
Copy link

Alceatraz commented Nov 27, 2022

https://issues.sonatype.org/plugins/servlet/mobile#issue/NEXUS-28247 look like old bug.

@AlexGluck I think I found the reason of the unknown blob

If, You:

  • group repo using both hosted and proxy
  • proxy contains some layer, Lets say layer:aaaaaaa
  • then build your image FROM image:xxxx(contains layer:aaaaaaa)
  • push your image
org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/xxxxxxxx/manifests/xxxxxxxx: 400 - org.sonatype.nexus.repository.docker.internal.V2Exception: Invalid Manifest
org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxfor: /v2/xxxxxxxx/manifests/xxxxxxxx in repository RepositoryImpl$$EnhancerByGuice$$xxxxxxxxxxxxxx{type=hosted, format=docker, name='docker-hosted'}

Because pushed layers contains layer:aaaaaaa, But when push layers nginx will route HEAD to group, Response tell docker-client layer:aaaa exist but actualy not exist in hosted, Then at last step PUT manifest, There must be some parse steps, Nexus can't find layer:aaaa in hosted repo, Then throw an error.

If remove all proxy repo from group, Push will success.

I think its need some application layer proxy, For example: capture POST ^/(v1|v2)/(.*?)/blobs/uploads/$ then parse it. Use the parsed date to redirect next HEAD ^/(v1|v2)/(.*?)/blobs/uploads/.*?$ to hosted. But it can't be done: docker client will send HEAD request directly, There is no prerequest.

So, I think its a countermeasures for avoid this hack behiver.

@AlexGluck
Copy link

@Alceatraz For workaround this unwanted behavior i added this error_page 400 404 500 = @fallback;

@Alceatraz
Copy link

@Alceatraz For workaround this unwanted behavior i added this error_page 400 404 500 = @fallback;

Wired, This config not working for me. docker will show raw html content to stdout

@AlexGluck
Copy link

@Alceatraz i tested many clients and docker include, may be you do something wrong?

@Alceatraz
Copy link

@Alceatraz i tested many clients and docker include, may be you do something wrong?

root@debian:/opt/nexus# cat /etc/nginx/nginx.conf 
user www-data;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
events {
  worker_connections  1024;
}
http {

  proxy_send_timeout 120;
  proxy_read_timeout 300;
  proxy_buffering    off;
  tcp_nodelay        on;
  client_max_body_size 0;
  chunked_transfer_encoding on;
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=nexus:100m inactive=30d  max_size=2g;

  upstream nexus-node {
    server 127.0.0.1:8081 max_fails=0;
    keepalive 150;
    keepalive_timeout 60s;
    keepalive_requests 1000;
  }

map $upstream_http_location $upstream_docker_version {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(?<version>v1|v2)/([-_0-9a-z\.]+)/(.*)$" $version;
}
map $upstream_http_location $upstream_docker_repo_name {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(v1|v2)/(?<repo_name>[-_0-9a-z\.]+)/(.*)$" $repo_name;
}
map $upstream_http_location $upstream_docker_rest_uri {
    "~^(http(s)?:/)?(/[-_:0-9a-z\.]+)?/(v1|v2)/([-_0-9a-z\.]+)/(?<rest_uri>.*)$" $rest_uri;
}

map $uri $docker_repo_name_in {
    "~^/(v1|v2)/(?<repo_name>[-_0-9a-z\.]+)/(.*)$" $repo_name;
}

map $upstream_docker_repo_name:$docker_repo_name_in $response_header_location {
    "~^(.*):\1$" $upstream_http_location;
    default /$upstream_docker_version/$docker_repo_name_in/$upstream_docker_repo_name/$upstream_docker_rest_uri;
}

  server {
    listen  80;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    location ~ ^/api/(.*) {
      proxy_pass http://nexus-node/$1$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
      proxy_pass http://nexus-node/repository/library/$1/library/library/$2/$3$is_args$args;
      proxy_hide_header Location;
      add_header Location $response_header_location always;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location ~ ^/(v1|v2)/?$ {
      proxy_pass http://nexus-node/repository/docker-login/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/(_catalog|search)$ {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      proxy_hide_header Location;
      add_header Location $response_header_location always;
      error_page 400 404 500 = @fallback;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      400 500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback2 {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location / {
      proxy_pass http://nexus-node/;
    }
  }
}

root@debian:/opt/nexus# docker pull 127.0.0.1/library/alpine
Using default tag: latest
Error response from daemon: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "\n<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n  <title>404 - Nexus Repository Manager</title>\n  <meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"/>\n\n\n  <!--[if lt IE 9]>\n  <script>(new Image).src=\"../../../favicon.ico?3.43.0-01\"</script>\n  <![endif]-->\n  <link rel=\"icon\" type=\"image/png\" href=\"../../../favicon-32x32.png?3.43.0-01\" sizes=\"32x32\">\n  <link rel=\"mask-icon\" href=\"../../../safari-pinned-tab.svg?3.43.0-01\" color=\"#5bbad5\">\n  <link rel=\"icon\" type=\"image/png\" href=\"../../../favicon-16x16.png?3.43.0-01\" sizes=\"16x16\">\n  <link rel=\"shortcut icon\" href=\"../../../favicon.ico?3.43.0-01\">\n  <meta name=\"msapplication-TileImage\" content=\"../../../mstile-144x144.png?3.43.0-01\">\n  <meta name=\"msapplication-TileColor\" content=\"#00a300\">\n\n  <link rel=\"stylesheet\" type=\"text/css\" href=\"../../../static/css/nexus-content.css?3.43.0-01\"/>\n</head>\n<body>\n<div class=\"nexus-header\">\n  <a href=\"../../..\">\n    <div class=\"product-logo\">\n      <img src=\"../../../static/rapture/resources/icons/x32/nexus-white.png?3.43.0-01\" alt=\"Product logo\"/>\n    </div>\n    <div class=\"product-id\">\n      <div class=\"product-id__line-1\">\n        <span class=\"product-name\">Nexus Repository Manager</span>\n      </div>\n      <div class=\"product-id__line-2\">\n        <span class=\"product-spec\">OSS 3.43.0-01</span>\n      </div>\n    </div>\n  </a>\n</div>\n\n<div class=\"nexus-body\">\n  <div class=\"content-header\">\n    <img src=\"../../../static/rapture/resources/icons/x32/exclamation.png?3.43.0-01\" alt=\"Exclamation point\" aria-role=\"presentation\"/>\n    <span class=\"title\">Error 404</span>\n    <span class=\"description\">Not Found</span>\n  </div>\n  <div class=\"content-body\">\n    <div class=\"content-section\">\n      Not Found\n    </div>\n  </div>\n</div>\n</body>\n</html>\n\n"
{

	"insecure-registry":[
		"127.0.0.1",
		"127.0.0.1:5000",
		"127.0.0.1:8081"

	]
}

1

@AlexGluck
Copy link

@Alceatraz in your logs i see you don't have container image and screenshot shows nexus docker repo library not created. Everything work correctly.

@phonographLP
Copy link

@AlexGluck I thought "proxy_buffering off" is incompatible with nginx caching as per
https://trac.nginx.org/nginx/ticket/2048
Or am I mistaken? Or this requires a specific nginx version to work?

@AlexGluck
Copy link

@phonographLP Perhaps this is a legacy piece, I will try to test the behavior of the cache.

@simao-silva
Copy link

Is this Nginx configuration still valid? I am running Nexus OSS 3.47.1-01 and I have added the groups as required but I always get a 404 when doing docker login.

Screenshot from 2023-03-18 19-05-13

@AlexGluck
Copy link

@simao-silva i don't see repo docker-login.

@simao-silva
Copy link

For correct registry API version check and check authorization require docker repository docker-login (create required), recommends type group and contain docker proxy for hub.docker.com.

After reading this I am still confused. What type of repository should docker-login be ?

@AlexGluck
Copy link

@simao-silva type group.

@Alceatraz
Copy link

Is this Nginx configuration still valid? I am running Nexus OSS 3.47.1-01 and I have added the groups as required but I always get a 404 when doing docker login.

Screenshot from 2023-03-18 19-05-13

I'm using debian, Seems like new version docker-client will request another URL, For me I just give up use nginx for that bypass. I just wrote a server proxy for nexus OSS. All source uploaded.

@a-langer
Copy link

@Alceatraz In my version of the rules docker_location.conf, everything still works, at least on "docker/23.0.3".

@magicJie
Copy link

magicJie commented May 9, 2023

@Alceatraz In my version of the rules docker_location.conf, everything still works, at least on "docker/23.0.3".

Have you tested the containerd client? I used this configuration, but cannot fetch images from the proxy repository.

@a-langer
Copy link

a-langer commented May 9, 2023

Tell me, how did you test the containerd client? I always thought containerd was part of Docker, at least it works under the hood for me:

~$ systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor prese>
...

However, I tested fetch image through the "ctr" utility, and it works for me:

$ sudo ctr image pull -u "LOGIN:PASSWORD" nexus_host/myrepo/alpine:latest
nexus_host/myrepo/alpine:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1304f174557314a7ed9ebbd6eab12fed11cb0cd9809e4c27f29af86979a3b870: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:213ec9aee27d8be045c6a82b7eac22c9a64b44558183764a1a7f626352392a49:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:9c6f0724472873bb40a2ae67a9a8adcb57673a193rty8b06eb778cda855171b5:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.0 s                                                                    total:  2.0 Mi (2.0 MiB/s)                                       
unpacking linux/amd64 sha256:1304f174557314a7ed9ebbd6eab12fed11cb0cd9809e4c27f29af86979a3b870...
done: 117.010985ms

First, I would test if this configuration works for you through the Docker client to make sure the access system is set up correctly, and only then through "ctr".

@magicJie
Copy link

magicJie commented May 10, 2023

Here's the configuration for my repository:
image
aiminjie type:hosted,privite repository
docker-root type:hosted,
docker-proxy-docker-ip:proxy for docker.io.

I hope to automatically download docker.io repository images through the group repository, so that I can easily download docker.io images. Similarly, I may add many other public repositories' images. By aggregating them and using my domain name, I can download images from multiple websites. However, when I use the command "crictl pull swr1.aiminjie.com/nginx", an error occurs:

E0510 13:59:48.844167 252623 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "swr1.lan.aiminjie.com/nginx:latest": failed to resolve reference "swr1.lan.aiminjie.com/nginx:latest": swr1.lan.aiminjie.com/nginx:latest: not found" image="swr1.lan.aiminjie.com/nginx" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "swr1.lan.aiminjie.com/nginx:latest": failed to resolve reference "swr1.lan.aiminjie.com/nginx:latest": swr1.lan.aiminjie.com/nginx:latest: not found.

Obviously, it did not try to get the image through the proxy. This problem has bothered me for a long time. The aggregation and fetching function of nexus3 is really great, but now I don't know how to use it, which is regrettable!

@a-langer
Copy link

To get started, use docker and try to pull the image directly from the repository "docker-proxy-docker-io":

docker pull swr1.lan.aiminjie.com/docker-proxy-docker-io/nginx:latest 

Then try to pull from the group repository, note that you need to specify "docker-group" in the image name:

docker pull swr1.lan.aiminjie.com/docker-group/nginx:latest 

If you want to pull images from the Nexus root without specifying a repository name, then you need to use the group repository "docker-root" (add your proxy repository to it) , then you can do the following:

docker pull swr1.lan.aiminjie.com/nginx:latest 

See paragraph "Docker Repository Reverse Proxy" in README.md.

@magicJie
Copy link

magicJie commented May 10, 2023

image
I changed the default repository to docker-group and it solved my problem, but now I still need to complete the following command:

nerdctl -n k8s.io push swr1.lan.aiminjie.com/nginx:test

[root@devops-0001 opsu]# nerd push swr1.lan.aiminjie.com/alpine:test
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:820a7dab799cc10b4c1a752fcbd7898b382a3126f5f54b4458f698805f820ec4)
index-sha256:820a7dab799cc10b4c1a752fcbd7898b382a3126f5f54b4458f698805f820ec4:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:c059bfaa849c4d8e4aecaeb3a10c2d9b3d85f5165c66ad3a4d937758128c4d18:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s                                                                    total:  2.3 Ki (22.6 KiB/s)
FATA[0000] failed commit on ref "index-sha256:820a7dab799cc10b4c1a752fcbd7898b382a3126f5f54b4458f698805f820ec4": unexpected status from PUT request to https://swr1.lan.aiminjie.com/v2/alpine/manifests/test: 403 Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

but i can push image to a created Repository
nerdctl -n k8s.io push swr1.lan.aiminjie.com/nginx:test
image

@magicJie
Copy link

To get started, use docker and try to pull the image directly from the repository "docker-proxy-docker-io":

docker pull swr1.lan.aiminjie.com/docker-proxy-docker-io/nginx:latest 

Then try to pull from the group repository, note that you need to specify "docker-group" in the image name:

docker pull swr1.lan.aiminjie.com/docker-group/nginx:latest 

If you want to pull images from the Nexus root without specifying a repository name, then you need to use the group repository "docker-root" (add your proxy repository to it) , then you can do the following:

docker pull swr1.lan.aiminjie.com/nginx:latest 

See paragraph "Docker Repository Reverse Proxy" in README.md.

Okay, thank you for your help

@a-langer
Copy link

Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

Deploying to group repositories is not supported in the open source version of Nexus.
Glad it was helpful.

@EsDmitrii
Copy link

EsDmitrii commented May 25, 2023

Hi team!
I faced some issues with nexus and I don't know what happened.
I use this default nginx config
Errors I started to get is:

2023/05/24 09:00:22 [notice] 29323#29323: *20830115 "(POST|PUT|DELETE|PATCH)" does not match "HEAD", client: client_ip, server: my.awesome.server, request: "HEAD /v2/path/to/manifest HTTP/1.1", host: "my.awesome.server"

2023/05/24 09:00:19 [notice] 29323#29323: *20830104 "(POST|PUT|DELETE|PATCH|HEAD)" does not match "GET", client: client_ip, server: my.awesome.server, request: "GET /v2/path/to/blob/sha256 HTTP/1.1", host: "my.awesome.server:443"

can someone explain or assist what I need to do to fix this? I'm getting a lot 4xx in my Nginx
Screenshot 2023-05-25 at 15 18 57

@a-langer
Copy link

@EsDmitrii can be used completed solution https://github.com/a-langer/nexus-sso, it docker compose + preconfigured nginx reverse proxy:

git clone https://github.com/a-langer/nexus-sso.git
cd ./nexus-sso
docker compose up -d 

If needs use standart docker image (without Single Sign-On patch), then set in .env file:

NEXUS_IMAGE="sonatype/nexus3:3.51.0"

@EsDmitrii
Copy link

@AlexGluck Hi, sorry for the late response. I dig deeper and found something interesting
Incorrect request without token goes first, I get 401

[25/May/2023:17:16:57 +0300] "GET /v2/ HTTP/1.1" 401 113 "-" "docker/20.10.11 go/go1.16.9 git-commit/847da18 kernel/3.10.0-1160.53.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" "-"

And at the same time goes correct request with token

[25/May/2023:17:16:57 +0300] "GET /v2/token?scope=token_masked HTTP/1.1" 200 60 "-" "docker/20.10.11 go/go1.16.9 git-commit/847da18 kernel/3.10.0-1160.53.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" "-"

This issue not connected with nexus or nginx, so I'm sorry to bother you!
Thank you for your help, I appreciate it!

@EsDmitrii
Copy link

@a-langer Thank you for your help!
Unfortunately we use existing production Nexus server and I'd like not to redeploy it:)
Weird issue started to reproduce a couple days ago, so I'm trying to investigate who broke something and fix it:)

@EsDmitrii
Copy link

EsDmitrii commented Jun 28, 2023

Hi guys!
Here we go again:)
So I need to setup specific proxy from Nginx load balancer.
I have this setup:
Nginx that balancing traffic between Nexus and Harbor.
For some reasons I need to redirect all POST|PUT|DELETE|PATCH|HEAD requests to Harbor, and keep GET requests from Nexus.
I prepared this confing:

      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
         rewrite ^/(.*)$ https://harbor.my.domain redirect;
        }
      }
      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (GET) ) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker/$1 last;
      }

when I run docker push to nexus e.g. docker push nexus.my.domain/projectname/nginx:stable-alpine-slim, it should redirect and push image to Harbor, but I see the error denied: Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

when I create additional upstream in nexus Nginx config that points to harbor server and port and modify proxy like this:

      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
         proxy_pass http://harbor;
        }
      }
      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (GET) ) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker/$1 last;
      }

And push the image like in previous step, I see
unauthorized: unauthorized to access repository: projectname/nginx, action: push: unauthorized to access repository: projectname/nginx, action: push

Repo in harbor exists, docker logged in both harbor and nexus reposytories.
My user has access rights to project in Harbor and Nexus
I don't know for who Harbor says it: for me, or for Nexus (may be for some reasons docker push goes from Nexus instance, I don't know)
Does Nexus support this feature or not? May be I'm doing wrong, I don't know.

May be someone tried to do something same?
I appreciate you for your help!
Thank you in advance!

@AlexGluck
Copy link

@EsDmitrii please look how work registry https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push or https://docs.docker.com/registry/spec/api/ and tell how auth on nexus can be work in harbor?
Our code implement transparent reverse proxy on nexus, and what are you show not equal our solution.

@EsDmitrii
Copy link

@AlexGluck thank you, didn't thought about it

@Li-Gru
Copy link

Li-Gru commented Aug 9, 2023

upstream nexus          { server 127.0.0.1:8081;   }
upstream docker-hosted  { server 127.0.0.1:5000;   }
upstream docker-group   { server 127.0.0.1:5001;   }

map $request_method $redirection {
    default                    "nexus";
    "~(GET|HEAD)"              "docker-group";
    "~(PATCH|PUT|POST|DELETE)" "docker-hosted";
}

...

    location /v2 {
        client_max_body_size 1G;
        proxy_pass http://$redirection;
        ...
    }

@mohammadmet
Copy link

mohammadmet commented Sep 12, 2023

Hi everyone,

Thank you for your configuration; it worked perfectly. However, I have encountered an issue. When I push my image, I receive the following error:

1

here is my Config:

ssl_certificate /opt/CA/fullchain.pem;
ssl_certificate_key /opt/CA/privkey.pem; 

proxy_http_version 1.1;
proxy_set_header Connection "";

location ~ ^/api/(.*) {
    proxy_pass http://nexus-repo.de:8081/$1$is_args$args;
}

location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
  proxy_pass http://nexus-repo.de:8081/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
}

location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
  proxy_pass http://nexus-repo.de:8081/repository/library/$1/library/library/$2/$3$is_args$args;
  error_page 404 500 = @fallback2;
  proxy_intercept_errors on;
  recursive_error_pages on;
  proxy_cache            nexus;
  proxy_cache_valid      500 10d;
  proxy_cache_min_uses   3;
  proxy_cache_valid      404 15m;
  proxy_cache_use_stale  http_500;
}

location ~ ^/(v1|v2)/?$ {
    proxy_pass http://nexus-repo.de:8081/repository/docker-snapshot-local/$1/$2$is_args$args;
}

location ~ ^/(v1|v2)/(_catalog|search)$ {
    proxy_pass http://nexus-repo.de:8081/repository/docker/$1/$2$is_args$args;
}

location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
    proxy_pass http://nexus-repo.de:8081/repository/$2/$1/$3$is_args$args;
    proxy_hide_header Location;
    error_page 400 404 500 = @fallback;
    proxy_intercept_errors on;
    recursive_error_pages on;
    proxy_cache            nexus;
    proxy_cache_valid      400 500 10d;
    proxy_cache_min_uses   3;
    proxy_cache_valid      404 15m;
    proxy_cache_use_stale  http_500;
}

location @fallback {
    proxy_pass http://nexus-repo.de:8081/repository/$2/$1/$3$is_args$args;
    error_page 400 404 500 = @fallback2;
    proxy_intercept_errors on;
    recursive_error_pages on;
    proxy_cache            nexus;
    proxy_cache_valid      500 10d;
    proxy_cache_min_uses   3;
    proxy_cache_valid      404 15m;
    proxy_cache_use_stale  http_500;
}

location @fallback2 {
    proxy_pass http://nexus-repo.de:8081/repository/docker/$1/$2$is_args$args;
  }


location / {
    proxy_pass http://nexus-repo.de:8081/;
}

}

@AlexGluck
Copy link

@mohammadmet Do you have docker hosted repository docker-snapshot-local?

@mohammadmet
Copy link

@AlexGluck yes i have, here is list of my Repository:

docker (Docker group),
docker-public-remote (Docker Proxy)
docker-release-local (Docker Host)
docker-snapshot-local (Docker Host)
library (Docker host)

@AlexGluck
Copy link

@mohammadmet In my config different location:

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$3$is_args$args;

Highlights: nexus-node/repository/$2/$1/$2/$3$is_args$args;

@mohammadmet
Copy link

@AlexGluck When I push my image using this config However, the main folder has the same name as the Docker repository. That's why I attempted to use this configuration.

@AlexGluck
Copy link

@mohammadmet I'm not tested configuration with fix and author don't support with him solution. Why are you need hide additional folder?

@mohammadmet
Copy link

When I attempt to pull my image from Nexus, I currently need to use the following command:
docker pull nexus-repo.de/docker/docker-snapshot-local/final:test

However, I would like to be able to use this command instead:
docker pull nexus-repo.de/docker/final:test

Unfortunately, I am encountering this error when I try:

Unbenannt

PS: Thank you for your quick response.

@AlexGluck
Copy link

@mohammadmet Try docker pull nexus-repo.de/docker-snapshot-local/final:test

@mohammadmet
Copy link

The following Docker pulls are working as expected:

docker pull nexus-repo.de/docker-snapshot-local/final:test ✔️
docker pull nexus-repo.de/docker/nginx:latest ✔️
docker pull nexus-repo.de/docker/docker-snapshot-local/final:test ✔️

However, I'm encountering an issue with:
docker pull nexus-repo.de/docker/final:test ❌

When I push something to the Docker repository hosted in Nexus, I can pull it from the Docker group repository using the full name, like: nexus-repo.de/docker/docker-snapshot-local/final:test.
However, I would like to simplify this naming convention when pulling my images, such as docker pull
nexus-repo.de/docker/final:test.

Do you have any ideas on how I can achieve this?

@AlexGluck
Copy link

You can write me in telegram to discuss possible solutions.

@AlexGluck
Copy link

New interesting solution https://github.com/fraima/nexus-public/blob/release-3.60.0-03/README-PG.md
Nexus OSS with support postgres backend database.
Support for single sign-on (SSO), replacement of hazelcast with redis for distributed (replicated) cache, and full support for high availability are expected.

@tuananh170489
Copy link

@abdennour thanks for your sharing!
I tested it with docker-compose, and it worked perfectly.
Do you have the k8s ingress version of it?

@tuananh170489
Copy link

I figured out the solution, it worked properly with ingress nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nexus-docker
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD)) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker-group/$1 last;
      }

      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH)) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker-group/$1 last;
      }
spec:
  ingressClassName: nginx
  rules:
    - host: nexus.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nexus
                port:
                  number: 8081

@chaitany010
Copy link

chaitany010 commented May 1, 2024

Hello everyone, I have never configured a Nexus before. I received a request to maintain our Docker images in Nexus by setting up a Nexus artifacts server.
My arrangement:
Operating System: Ubuntu 22.04
Nexus: Sonatype RepositoryOSS 3.63.0-01 for Nexus
Nginx in reverse proxy mode.
My Requirement: Docker-proxy is needed to pull the docker images, and Docket-host repository is needed to push the images.
I have set up Nexus and Nginx, and by using the nginx configuration below—which is available in the Sonatype documentation at https://help.sonatype.com/en/docker-repository-reverse-proxy-strategies.html—I can access both the Nexus and the URL. However, the Docker login does not function. Could you please help me obtain the correct nginx configuration file? I sincerely appreciate anyone's assistance with this. Thank you in advance.

server {
listen 80;
server_name nexus.example.com;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl;
server_name nexus.example.com;

ssl on;

ssl_certificate /etc/ssl/certs/appcert/example.com.crt;
ssl_certificate_key /etc/ssl/certs/appcert/example.com.key;

# Docker hosted repository /v2 requests
location /repository/nexus-docker-host/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/repository/nexus-docker-host/;
}

# Docker proxy repository /v2 requests
location /repository/nexus-docker-proxy/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/repository/nexus-docker-proxy/;
}

# Regular Nexus requests
location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/;
}

}

@chaitany010
Copy link

Hi @AlexGluck thank you for your quick response. I am trying the config which you shared, getting the below error while testing config by running sudo nginx -t command. am i missing some thing? thank you!
nexus@nexus-test:/etc/nginx/sites-enabled$ sudo nginx -t
nginx: [emerg] "user" directive is not allowed here in /etc/nginx/sites-enabled/nexus.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

@AlexGluck
Copy link

@chaitany010 required code for you (i hope):

  proxy_send_timeout 120;
  proxy_read_timeout 300;
  proxy_buffering    off;
  keepalive_timeout  5 5;
  tcp_nodelay        on;
  client_max_body_size 0;
  chunked_transfer_encoding on;
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=nexus:100m inactive=30d  max_size=2g;

  upstream nexus-node {
    server nexus:8081 max_fails=0;
    keepalive 150;
    keepalive_timeout 60s;
    keepalive_time 1h;
    keepalive_requests 1000;
  }

  server {
    listen  80;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    location ~ ^/api/(.*) {
      proxy_pass http://nexus-node/$1$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
      proxy_pass http://nexus-node/repository/library/$1/library/library/$2/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location ~ ^/(v1|v2)/$ {
      proxy_pass http://nexus-node/repository/docker-login/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/(_catalog|search)$ {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$3$is_args$args;
      error_page 400 404 500 = @fallback;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      400 500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback2 {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location / {
      proxy_pass http://nexus-node/;
    }
  }

@yokozu777
Copy link

Hi all.
If I activate cache (including proxy_buffering on; )
docker push stops working
"unauthorized: access to the requested resource is not authorized"

Log with cache off:
171.2.1.4 - admin [11/May/2024:20:19:57 +0100] "POST

Log with cache on:
171.2.1.4 - - [11/May/2024:20:22:16 +0100] "POST

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment