Skip to content

Instantly share code, notes, and snippets.

@abdennour
Last active June 19, 2024 10:38
Show Gist options
  • Save abdennour/74c5de79e57a47f3351217d674238da8 to your computer and use it in GitHub Desktop.
Save abdennour/74c5de79e57a47f3351217d674238da8 to your computer and use it in GitHub Desktop.
Nginx Reverse Proxy for Nexus Docker Registries

Overview

This is a solution of a common problem with Nexus Docker repositories. The administrator has to expose port for "pull", another port for "push", other ports for each hosted repository. This solution is about leveraging Nginx reverse proxy to avoid using these ports.

How it works ?

Given :

  • Nexus hostname is "nexus.example.com"
  • Nexus web port is 8081
  • A hosted repository is named "docker-hosted"
  • A group repository is named "docker-group"
  • Your nginx (with the nginx.conf of this gist) will run for example under cregistry.example.com

The following Nginx configuration file is for a reverse proxy without the need to expose connector ports from nexus :

  • docker pull cregistry.example.com/myimage lets Nginx forward the request to "docker-group"
  • docker push cregistry.example.com/myimage lets Nginx forward the request to "docker-hosted"

Notes

  • If you have more than one hosted repository, create another Nginx reverse proxy for it, then aggregate them using a parent Nginx reverse proxy that forwards the request according to certain criteria (.i.e: Host header).

  • All Nexus repositories must have consistent configuration of authentication: Either all require authentication, or all don't.

  • If TLS is enabled with Nexus, change proxy_set_header X-Forwarded-Proto "http"; by proxy_set_header X-Forwarded-Proto "https";

version: "3"
services:
web:
image: nginx:1.15
hostname: cregistry.example.com
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
ports:
- "80:80"
nexus:
image: sonatype/nexus3
hostname: nexus.example.com
volumes:
- "nexus-data:/nexus-data"
ports:
- "8081:8081"
volumes:
nexus-data: {}
events {
worker_connections 1024;
}
http {
proxy_send_timeout 120;
proxy_read_timeout 300;
proxy_buffering off;
keepalive_timeout 5 5;
tcp_nodelay on;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
server {
listen *:80;
location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
}
rewrite ^/(.*)$ /repository/docker-group/$1 last;
}
location ~ ^/(v1|v2)/ {
if ($request_method ~* (POST|PUT|DELETE|PATCH) ) {
rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
}
rewrite ^/(.*)$ /repository/docker-group/$1 last;
}
location / {
proxy_pass http://nexus.example.com:8081/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "http";
}
}
}
@magicJie
Copy link

magicJie commented May 10, 2023

image
I changed the default repository to docker-group and it solved my problem, but now I still need to complete the following command:

nerdctl -n k8s.io push swr1.lan.aiminjie.com/nginx:test

[root@devops-0001 opsu]# nerd push swr1.lan.aiminjie.com/alpine:test
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:820a7dab799cc10b4c1a752fcbd7898b382a3126f5f54b4458f698805f820ec4)
index-sha256:820a7dab799cc10b4c1a752fcbd7898b382a3126f5f54b4458f698805f820ec4:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:c059bfaa849c4d8e4aecaeb3a10c2d9b3d85f5165c66ad3a4d937758128c4d18:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s                                                                    total:  2.3 Ki (22.6 KiB/s)
FATA[0000] failed commit on ref "index-sha256:820a7dab799cc10b4c1a752fcbd7898b382a3126f5f54b4458f698805f820ec4": unexpected status from PUT request to https://swr1.lan.aiminjie.com/v2/alpine/manifests/test: 403 Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

but i can push image to a created Repository
nerdctl -n k8s.io push swr1.lan.aiminjie.com/nginx:test
image

@magicJie
Copy link

To get started, use docker and try to pull the image directly from the repository "docker-proxy-docker-io":

docker pull swr1.lan.aiminjie.com/docker-proxy-docker-io/nginx:latest 

Then try to pull from the group repository, note that you need to specify "docker-group" in the image name:

docker pull swr1.lan.aiminjie.com/docker-group/nginx:latest 

If you want to pull images from the Nexus root without specifying a repository name, then you need to use the group repository "docker-root" (add your proxy repository to it) , then you can do the following:

docker pull swr1.lan.aiminjie.com/nginx:latest 

See paragraph "Docker Repository Reverse Proxy" in README.md.

Okay, thank you for your help

@a-langer
Copy link

Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

Deploying to group repositories is not supported in the open source version of Nexus.
Glad it was helpful.

@EsDmitrii
Copy link

EsDmitrii commented May 25, 2023

Hi team!
I faced some issues with nexus and I don't know what happened.
I use this default nginx config
Errors I started to get is:

2023/05/24 09:00:22 [notice] 29323#29323: *20830115 "(POST|PUT|DELETE|PATCH)" does not match "HEAD", client: client_ip, server: my.awesome.server, request: "HEAD /v2/path/to/manifest HTTP/1.1", host: "my.awesome.server"

2023/05/24 09:00:19 [notice] 29323#29323: *20830104 "(POST|PUT|DELETE|PATCH|HEAD)" does not match "GET", client: client_ip, server: my.awesome.server, request: "GET /v2/path/to/blob/sha256 HTTP/1.1", host: "my.awesome.server:443"

can someone explain or assist what I need to do to fix this? I'm getting a lot 4xx in my Nginx
Screenshot 2023-05-25 at 15 18 57

@a-langer
Copy link

@EsDmitrii can be used completed solution https://github.com/a-langer/nexus-sso, it docker compose + preconfigured nginx reverse proxy:

git clone https://github.com/a-langer/nexus-sso.git
cd ./nexus-sso
docker compose up -d 

If needs use standart docker image (without Single Sign-On patch), then set in .env file:

NEXUS_IMAGE="sonatype/nexus3:3.51.0"

@EsDmitrii
Copy link

@AlexGluck Hi, sorry for the late response. I dig deeper and found something interesting
Incorrect request without token goes first, I get 401

[25/May/2023:17:16:57 +0300] "GET /v2/ HTTP/1.1" 401 113 "-" "docker/20.10.11 go/go1.16.9 git-commit/847da18 kernel/3.10.0-1160.53.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" "-"

And at the same time goes correct request with token

[25/May/2023:17:16:57 +0300] "GET /v2/token?scope=token_masked HTTP/1.1" 200 60 "-" "docker/20.10.11 go/go1.16.9 git-commit/847da18 kernel/3.10.0-1160.53.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" "-"

This issue not connected with nexus or nginx, so I'm sorry to bother you!
Thank you for your help, I appreciate it!

@EsDmitrii
Copy link

@a-langer Thank you for your help!
Unfortunately we use existing production Nexus server and I'd like not to redeploy it:)
Weird issue started to reproduce a couple days ago, so I'm trying to investigate who broke something and fix it:)

@EsDmitrii
Copy link

EsDmitrii commented Jun 28, 2023

Hi guys!
Here we go again:)
So I need to setup specific proxy from Nginx load balancer.
I have this setup:
Nginx that balancing traffic between Nexus and Harbor.
For some reasons I need to redirect all POST|PUT|DELETE|PATCH|HEAD requests to Harbor, and keep GET requests from Nexus.
I prepared this confing:

      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
         rewrite ^/(.*)$ https://harbor.my.domain redirect;
        }
      }
      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (GET) ) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker/$1 last;
      }

when I run docker push to nexus e.g. docker push nexus.my.domain/projectname/nginx:stable-alpine-slim, it should redirect and push image to Harbor, but I see the error denied: Deploying to groups is a PRO-licensed feature. See https://links.sonatype.com/product-nexus-repository

when I create additional upstream in nexus Nginx config that points to harbor server and port and modify proxy like this:

      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
         proxy_pass http://harbor;
        }
      }
      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (GET) ) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker/$1 last;
      }

And push the image like in previous step, I see
unauthorized: unauthorized to access repository: projectname/nginx, action: push: unauthorized to access repository: projectname/nginx, action: push

Repo in harbor exists, docker logged in both harbor and nexus reposytories.
My user has access rights to project in Harbor and Nexus
I don't know for who Harbor says it: for me, or for Nexus (may be for some reasons docker push goes from Nexus instance, I don't know)
Does Nexus support this feature or not? May be I'm doing wrong, I don't know.

May be someone tried to do something same?
I appreciate you for your help!
Thank you in advance!

@AlexGluck
Copy link

@EsDmitrii please look how work registry https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push or https://docs.docker.com/registry/spec/api/ and tell how auth on nexus can be work in harbor?
Our code implement transparent reverse proxy on nexus, and what are you show not equal our solution.

@EsDmitrii
Copy link

@AlexGluck thank you, didn't thought about it

@Li-Gru
Copy link

Li-Gru commented Aug 9, 2023

upstream nexus          { server 127.0.0.1:8081;   }
upstream docker-hosted  { server 127.0.0.1:5000;   }
upstream docker-group   { server 127.0.0.1:5001;   }

map $request_method $redirection {
    default                    "nexus";
    "~(GET|HEAD)"              "docker-group";
    "~(PATCH|PUT|POST|DELETE)" "docker-hosted";
}

...

    location /v2 {
        client_max_body_size 1G;
        proxy_pass http://$redirection;
        ...
    }

@mohammadmet
Copy link

mohammadmet commented Sep 12, 2023

Hi everyone,

Thank you for your configuration; it worked perfectly. However, I have encountered an issue. When I push my image, I receive the following error:

1

here is my Config:

ssl_certificate /opt/CA/fullchain.pem;
ssl_certificate_key /opt/CA/privkey.pem; 

proxy_http_version 1.1;
proxy_set_header Connection "";

location ~ ^/api/(.*) {
    proxy_pass http://nexus-repo.de:8081/$1$is_args$args;
}

location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
  proxy_pass http://nexus-repo.de:8081/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
}

location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
  proxy_pass http://nexus-repo.de:8081/repository/library/$1/library/library/$2/$3$is_args$args;
  error_page 404 500 = @fallback2;
  proxy_intercept_errors on;
  recursive_error_pages on;
  proxy_cache            nexus;
  proxy_cache_valid      500 10d;
  proxy_cache_min_uses   3;
  proxy_cache_valid      404 15m;
  proxy_cache_use_stale  http_500;
}

location ~ ^/(v1|v2)/?$ {
    proxy_pass http://nexus-repo.de:8081/repository/docker-snapshot-local/$1/$2$is_args$args;
}

location ~ ^/(v1|v2)/(_catalog|search)$ {
    proxy_pass http://nexus-repo.de:8081/repository/docker/$1/$2$is_args$args;
}

location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
    proxy_pass http://nexus-repo.de:8081/repository/$2/$1/$3$is_args$args;
    proxy_hide_header Location;
    error_page 400 404 500 = @fallback;
    proxy_intercept_errors on;
    recursive_error_pages on;
    proxy_cache            nexus;
    proxy_cache_valid      400 500 10d;
    proxy_cache_min_uses   3;
    proxy_cache_valid      404 15m;
    proxy_cache_use_stale  http_500;
}

location @fallback {
    proxy_pass http://nexus-repo.de:8081/repository/$2/$1/$3$is_args$args;
    error_page 400 404 500 = @fallback2;
    proxy_intercept_errors on;
    recursive_error_pages on;
    proxy_cache            nexus;
    proxy_cache_valid      500 10d;
    proxy_cache_min_uses   3;
    proxy_cache_valid      404 15m;
    proxy_cache_use_stale  http_500;
}

location @fallback2 {
    proxy_pass http://nexus-repo.de:8081/repository/docker/$1/$2$is_args$args;
  }


location / {
    proxy_pass http://nexus-repo.de:8081/;
}

}

@AlexGluck
Copy link

@mohammadmet Do you have docker hosted repository docker-snapshot-local?

@mohammadmet
Copy link

@AlexGluck yes i have, here is list of my Repository:

docker (Docker group),
docker-public-remote (Docker Proxy)
docker-release-local (Docker Host)
docker-snapshot-local (Docker Host)
library (Docker host)

@AlexGluck
Copy link

@mohammadmet In my config different location:

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$3$is_args$args;

Highlights: nexus-node/repository/$2/$1/$2/$3$is_args$args;

@mohammadmet
Copy link

@AlexGluck When I push my image using this config However, the main folder has the same name as the Docker repository. That's why I attempted to use this configuration.

@AlexGluck
Copy link

@mohammadmet I'm not tested configuration with fix and author don't support with him solution. Why are you need hide additional folder?

@mohammadmet
Copy link

When I attempt to pull my image from Nexus, I currently need to use the following command:
docker pull nexus-repo.de/docker/docker-snapshot-local/final:test

However, I would like to be able to use this command instead:
docker pull nexus-repo.de/docker/final:test

Unfortunately, I am encountering this error when I try:

Unbenannt

PS: Thank you for your quick response.

@AlexGluck
Copy link

@mohammadmet Try docker pull nexus-repo.de/docker-snapshot-local/final:test

@mohammadmet
Copy link

The following Docker pulls are working as expected:

docker pull nexus-repo.de/docker-snapshot-local/final:test ✔️
docker pull nexus-repo.de/docker/nginx:latest ✔️
docker pull nexus-repo.de/docker/docker-snapshot-local/final:test ✔️

However, I'm encountering an issue with:
docker pull nexus-repo.de/docker/final:test ❌

When I push something to the Docker repository hosted in Nexus, I can pull it from the Docker group repository using the full name, like: nexus-repo.de/docker/docker-snapshot-local/final:test.
However, I would like to simplify this naming convention when pulling my images, such as docker pull
nexus-repo.de/docker/final:test.

Do you have any ideas on how I can achieve this?

@AlexGluck
Copy link

You can write me in telegram to discuss possible solutions.

@AlexGluck
Copy link

New interesting solution https://github.com/fraima/nexus-public/blob/release-3.60.0-03/README-PG.md
Nexus OSS with support postgres backend database.
Support for single sign-on (SSO), replacement of hazelcast with redis for distributed (replicated) cache, and full support for high availability are expected.

@tuananh170489
Copy link

@abdennour thanks for your sharing!
I tested it with docker-compose, and it worked perfectly.
Do you have the k8s ingress version of it?

@tuananh170489
Copy link

I figured out the solution, it worked properly with ingress nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nexus-docker
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD)) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker-group/$1 last;
      }

      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH)) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker-group/$1 last;
      }
spec:
  ingressClassName: nginx
  rules:
    - host: nexus.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nexus
                port:
                  number: 8081

@chaitany010
Copy link

chaitany010 commented May 1, 2024

Hello everyone, I have never configured a Nexus before. I received a request to maintain our Docker images in Nexus by setting up a Nexus artifacts server.
My arrangement:
Operating System: Ubuntu 22.04
Nexus: Sonatype RepositoryOSS 3.63.0-01 for Nexus
Nginx in reverse proxy mode.
My Requirement: Docker-proxy is needed to pull the docker images, and Docket-host repository is needed to push the images.
I have set up Nexus and Nginx, and by using the nginx configuration below—which is available in the Sonatype documentation at https://help.sonatype.com/en/docker-repository-reverse-proxy-strategies.html—I can access both the Nexus and the URL. However, the Docker login does not function. Could you please help me obtain the correct nginx configuration file? I sincerely appreciate anyone's assistance with this. Thank you in advance.

server {
listen 80;
server_name nexus.example.com;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl;
server_name nexus.example.com;

ssl on;

ssl_certificate /etc/ssl/certs/appcert/example.com.crt;
ssl_certificate_key /etc/ssl/certs/appcert/example.com.key;

# Docker hosted repository /v2 requests
location /repository/nexus-docker-host/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/repository/nexus-docker-host/;
}

# Docker proxy repository /v2 requests
location /repository/nexus-docker-proxy/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/repository/nexus-docker-proxy/;
}

# Regular Nexus requests
location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/;
}

}

@chaitany010
Copy link

Hi @AlexGluck thank you for your quick response. I am trying the config which you shared, getting the below error while testing config by running sudo nginx -t command. am i missing some thing? thank you!
nexus@nexus-test:/etc/nginx/sites-enabled$ sudo nginx -t
nginx: [emerg] "user" directive is not allowed here in /etc/nginx/sites-enabled/nexus.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

@AlexGluck
Copy link

@chaitany010 required code for you (i hope):

  proxy_send_timeout 120;
  proxy_read_timeout 300;
  proxy_buffering    off;
  keepalive_timeout  5 5;
  tcp_nodelay        on;
  client_max_body_size 0;
  chunked_transfer_encoding on;
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=nexus:100m inactive=30d  max_size=2g;

  upstream nexus-node {
    server nexus:8081 max_fails=0;
    keepalive 150;
    keepalive_timeout 60s;
    keepalive_time 1h;
    keepalive_requests 1000;
  }

  server {
    listen  80;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    location ~ ^/api/(.*) {
      proxy_pass http://nexus-node/$1$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
      proxy_pass http://nexus-node/repository/library/$1/library/library/$2/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location ~ ^/(v1|v2)/$ {
      proxy_pass http://nexus-node/repository/docker-login/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/(_catalog|search)$ {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$3$is_args$args;
      error_page 400 404 500 = @fallback;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      400 500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback2 {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location / {
      proxy_pass http://nexus-node/;
    }
  }

@yokozu777
Copy link

Hi all.
If I activate cache (including proxy_buffering on; )
docker push stops working
"unauthorized: access to the requested resource is not authorized"

Log with cache off:
171.2.1.4 - admin [11/May/2024:20:19:57 +0100] "POST

Log with cache on:
171.2.1.4 - - [11/May/2024:20:22:16 +0100] "POST

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment