Skip to content

Instantly share code, notes, and snippets.

@abdennour
Last active June 19, 2024 10:38
Show Gist options
  • Save abdennour/74c5de79e57a47f3351217d674238da8 to your computer and use it in GitHub Desktop.
Save abdennour/74c5de79e57a47f3351217d674238da8 to your computer and use it in GitHub Desktop.
Nginx Reverse Proxy for Nexus Docker Registries

Overview

This is a solution of a common problem with Nexus Docker repositories. The administrator has to expose port for "pull", another port for "push", other ports for each hosted repository. This solution is about leveraging Nginx reverse proxy to avoid using these ports.

How it works ?

Given :

  • Nexus hostname is "nexus.example.com"
  • Nexus web port is 8081
  • A hosted repository is named "docker-hosted"
  • A group repository is named "docker-group"
  • Your nginx (with the nginx.conf of this gist) will run for example under cregistry.example.com

The following Nginx configuration file is for a reverse proxy without the need to expose connector ports from nexus :

  • docker pull cregistry.example.com/myimage lets Nginx forward the request to "docker-group"
  • docker push cregistry.example.com/myimage lets Nginx forward the request to "docker-hosted"

Notes

  • If you have more than one hosted repository, create another Nginx reverse proxy for it, then aggregate them using a parent Nginx reverse proxy that forwards the request according to certain criteria (.i.e: Host header).

  • All Nexus repositories must have consistent configuration of authentication: Either all require authentication, or all don't.

  • If TLS is enabled with Nexus, change proxy_set_header X-Forwarded-Proto "http"; by proxy_set_header X-Forwarded-Proto "https";

version: "3"
services:
web:
image: nginx:1.15
hostname: cregistry.example.com
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
ports:
- "80:80"
nexus:
image: sonatype/nexus3
hostname: nexus.example.com
volumes:
- "nexus-data:/nexus-data"
ports:
- "8081:8081"
volumes:
nexus-data: {}
events {
worker_connections 1024;
}
http {
proxy_send_timeout 120;
proxy_read_timeout 300;
proxy_buffering off;
keepalive_timeout 5 5;
tcp_nodelay on;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
server {
listen *:80;
location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD) ) {
rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
}
rewrite ^/(.*)$ /repository/docker-group/$1 last;
}
location ~ ^/(v1|v2)/ {
if ($request_method ~* (POST|PUT|DELETE|PATCH) ) {
rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
}
rewrite ^/(.*)$ /repository/docker-group/$1 last;
}
location / {
proxy_pass http://nexus.example.com:8081/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "http";
}
}
}
@mohammadmet
Copy link

When I attempt to pull my image from Nexus, I currently need to use the following command:
docker pull nexus-repo.de/docker/docker-snapshot-local/final:test

However, I would like to be able to use this command instead:
docker pull nexus-repo.de/docker/final:test

Unfortunately, I am encountering this error when I try:

Unbenannt

PS: Thank you for your quick response.

@AlexGluck
Copy link

@mohammadmet Try docker pull nexus-repo.de/docker-snapshot-local/final:test

@mohammadmet
Copy link

The following Docker pulls are working as expected:

docker pull nexus-repo.de/docker-snapshot-local/final:test ✔️
docker pull nexus-repo.de/docker/nginx:latest ✔️
docker pull nexus-repo.de/docker/docker-snapshot-local/final:test ✔️

However, I'm encountering an issue with:
docker pull nexus-repo.de/docker/final:test ❌

When I push something to the Docker repository hosted in Nexus, I can pull it from the Docker group repository using the full name, like: nexus-repo.de/docker/docker-snapshot-local/final:test.
However, I would like to simplify this naming convention when pulling my images, such as docker pull
nexus-repo.de/docker/final:test.

Do you have any ideas on how I can achieve this?

@AlexGluck
Copy link

You can write me in telegram to discuss possible solutions.

@AlexGluck
Copy link

New interesting solution https://github.com/fraima/nexus-public/blob/release-3.60.0-03/README-PG.md
Nexus OSS with support postgres backend database.
Support for single sign-on (SSO), replacement of hazelcast with redis for distributed (replicated) cache, and full support for high availability are expected.

@tuananh170489
Copy link

@abdennour thanks for your sharing!
I tested it with docker-compose, and it worked perfectly.
Do you have the k8s ingress version of it?

@tuananh170489
Copy link

I figured out the solution, it worked properly with ingress nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nexus-docker
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH|HEAD)) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker-group/$1 last;
      }

      location ~ ^/(v1|v2)/ {
        if ($request_method ~* (POST|PUT|DELETE|PATCH)) {
          rewrite ^/(.*)$ /repository/docker-hosted/$1 last;
        }
        rewrite ^/(.*)$ /repository/docker-group/$1 last;
      }
spec:
  ingressClassName: nginx
  rules:
    - host: nexus.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nexus
                port:
                  number: 8081

@chaitany010
Copy link

chaitany010 commented May 1, 2024

Hello everyone, I have never configured a Nexus before. I received a request to maintain our Docker images in Nexus by setting up a Nexus artifacts server.
My arrangement:
Operating System: Ubuntu 22.04
Nexus: Sonatype RepositoryOSS 3.63.0-01 for Nexus
Nginx in reverse proxy mode.
My Requirement: Docker-proxy is needed to pull the docker images, and Docket-host repository is needed to push the images.
I have set up Nexus and Nginx, and by using the nginx configuration below—which is available in the Sonatype documentation at https://help.sonatype.com/en/docker-repository-reverse-proxy-strategies.html—I can access both the Nexus and the URL. However, the Docker login does not function. Could you please help me obtain the correct nginx configuration file? I sincerely appreciate anyone's assistance with this. Thank you in advance.

server {
listen 80;
server_name nexus.example.com;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl;
server_name nexus.example.com;

ssl on;

ssl_certificate /etc/ssl/certs/appcert/example.com.crt;
ssl_certificate_key /etc/ssl/certs/appcert/example.com.key;

# Docker hosted repository /v2 requests
location /repository/nexus-docker-host/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/repository/nexus-docker-host/;
}

# Docker proxy repository /v2 requests
location /repository/nexus-docker-proxy/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/repository/nexus-docker-proxy/;
}

# Regular Nexus requests
location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto "http";
    proxy_pass http://localhost:8081/;
}

}

@chaitany010
Copy link

Hi @AlexGluck thank you for your quick response. I am trying the config which you shared, getting the below error while testing config by running sudo nginx -t command. am i missing some thing? thank you!
nexus@nexus-test:/etc/nginx/sites-enabled$ sudo nginx -t
nginx: [emerg] "user" directive is not allowed here in /etc/nginx/sites-enabled/nexus.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

@AlexGluck
Copy link

@chaitany010 required code for you (i hope):

  proxy_send_timeout 120;
  proxy_read_timeout 300;
  proxy_buffering    off;
  keepalive_timeout  5 5;
  tcp_nodelay        on;
  client_max_body_size 0;
  chunked_transfer_encoding on;
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=nexus:100m inactive=30d  max_size=2g;

  upstream nexus-node {
    server nexus:8081 max_fails=0;
    keepalive 150;
    keepalive_timeout 60s;
    keepalive_time 1h;
    keepalive_requests 1000;
  }

  server {
    listen  80;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    location ~ ^/api/(.*) {
      proxy_pass http://nexus-node/$1$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)/blobs/uploads/$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$2/$3/blobs/uploads/$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(blobs/sha256.*|manifests/.*)$ {
      proxy_pass http://nexus-node/repository/library/$1/library/library/$2/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location ~ ^/(v1|v2)/$ {
      proxy_pass http://nexus-node/repository/docker-login/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/(_catalog|search)$ {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location ~ ^/(v1|v2)/([-_0-9a-z\.]+)/(.*)$ {
      proxy_pass http://nexus-node/repository/$2/$1/$2/$3$is_args$args;
      error_page 400 404 500 = @fallback;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      400 500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback {
      proxy_pass http://nexus-node/repository/$2/$1/$3$is_args$args;
      error_page 404 500 = @fallback2;
      proxy_intercept_errors on;
      recursive_error_pages on;
      proxy_cache            nexus;
      proxy_cache_valid      500 10d;
      proxy_cache_min_uses   3;
      proxy_cache_valid      404 15m;
      proxy_cache_use_stale  http_500;
    }

    location @fallback2 {
      proxy_pass http://nexus-node/repository/docker-group/$1/$2$is_args$args;
    }

    location / {
      proxy_pass http://nexus-node/;
    }
  }

@yokozu777
Copy link

Hi all.
If I activate cache (including proxy_buffering on; )
docker push stops working
"unauthorized: access to the requested resource is not authorized"

Log with cache off:
171.2.1.4 - admin [11/May/2024:20:19:57 +0100] "POST

Log with cache on:
171.2.1.4 - - [11/May/2024:20:22:16 +0100] "POST

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment