Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Compiling NGINX module as dynamic module for use in docker

Compiling NGINX module as dynamic module for use in docker

NGINX offers some very convenient Docker containers, but it is not documented how to extend the functionality of those containers without re-compiling NGINX yourself and continuously adapting your Dockerfile to match the ones in https://github.com/nginxinc/docker-nginx/.

This gist offers a way to build a NGINX ‘dynamic module’ against the widely used “nginx:alpine” docker image. Simply copy this Dockerfile & replace the arguments of the “wget” & “tar” & “configure” commands to include the module you want to build, then adjust the COPY command copying the module (.so file). In this example the NCHAN module is used as an example.

If you found this helpful, please leave a comment / reaction here! :love:

FROM nginx:alpine AS builder
# nginx:alpine contains NGINX_VERSION environment variable, like so:
# ENV NGINX_VERSION 1.15.0
# Our NCHAN version
ENV NCHAN_VERSION 1.1.15
# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
wget "https://github.com/slact/nchan/archive/v${NCHAN_VERSION}.tar.gz" -O nchan.tar.gz
# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre-dev \
zlib-dev \
linux-headers \
curl \
gnupg \
libxslt-dev \
gd-dev \
geoip-dev
# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
tar -zxC /usr/src -f nginx.tar.gz && \
tar -xzvf "nchan.tar.gz" && \
NCHANDIR="$(pwd)/nchan-${NCHAN_VERSION}" && \
cd /usr/src/nginx-$NGINX_VERSION && \
./configure --with-compat $CONFARGS --add-dynamic-module=$NCHANDIR && \
make && make install
FROM nginx:alpine
# Extract the dynamic module NCHAN from the builder image
COPY --from=builder /usr/local/nginx/modules/ngx_nchan_module.so /usr/local/nginx/modules/ngx_nchan_module.so
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/nginx.conf
COPY default.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]
# Addition: load NCHAN
load_module /usr/local/nginx/modules/ngx_nchan_module.so;
user nginx;
worker_processes 5; # Addition: ultiple workers for NCHAN
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
@jaipresto

This comment has been minimized.

Copy link

@jaipresto jaipresto commented Mar 14, 2019

Very, very helpful Herman. It's unbelievable how shallow the 'official' documentation is on this.

Thankyou!

@grEvenX

This comment has been minimized.

Copy link

@grEvenX grEvenX commented May 26, 2019

Thanks Herman! Given nginx's huge popularity and the rapid usage of docker in production environments, I find it odd that there's no good official "support" in the means of documentation etc. to provide this. Great work 👍 , I will try to see if we can enable some modules we have been looking at in the past with this approach.

@petski

This comment has been minimized.

Copy link

@petski petski commented Jun 20, 2019

Thanks Herman! Some feedback:

  • I'd propose to use https to download the nginx sources; instead of using http
  • CONFARGS isn't properly given to the ./configure command at this moment; an && is missing on line 29
    • To avoid quoting hell, I've used sh -c "./configure ..."
@stengerh

This comment has been minimized.

Copy link

@stengerh stengerh commented Jun 27, 2019

Thanks Herman! Thanks petski, too! When I tried this approach with the current nginx 1.17, the --with-cc-args='...' parameter included spaces.

Note that you can speed up the docker image build by compiling only the dynamic module. I have replaced make && make install with make modules. You can then copy the module from the directory /usr/src/nginx-$NGINX_VERSION/objs/ in the builder image. On my dated laptop this decreases the image build time to about 5%.

@rwrz

This comment has been minimized.

Copy link

@rwrz rwrz commented Jul 18, 2019

One example using the http_echo_module.
I have made some changes:

  • updated to latest dev dependencies
  • changed to use curl (because I was having issues with wget and alpine)
  • removed the -fomit-frame-pointer from CONFARGS, since it was giving me an error to compile
  • all .so will be available on the root ( / ), you can get it with
COPY --from=builder /ngx_http_echo_module.so /usr/local/nginx/modules/ngx_http_echo_module.so

Here is the code

# Download sources
RUN curl "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -o nginx.tar.gz && \
  curl -L "https://github.com/openresty/echo-nginx-module/archive/v${HTTPECHO_VERSION}.tar.gz" -o httpecho.tar.gz

# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer/-Os} && \
    mkdir /usr/src && \
	tar -zxC /usr/src -f nginx.tar.gz && \
  tar -xzvf "httpecho.tar.gz" && \
  HTTPECHODIR="$(pwd)/echo-nginx-module-${HTTPECHO_VERSION}" && \
  cd /usr/src/nginx-$NGINX_VERSION && \
  ./configure --with-compat $CONFARGS --add-dynamic-module=$HTTPECHODIR && \
  make modules && \
  mv ./objs/*.so /
@wtorsi

This comment has been minimized.

Copy link

@wtorsi wtorsi commented Jul 27, 2019

What's the reason to use CONFARGS for --with-compat option?

@hermanbanken

This comment has been minimized.

Copy link
Owner Author

@hermanbanken hermanbanken commented Jul 30, 2019

Thanks everyone for the feedback! I love it that by simply putting this out there I might have helped you guys 😄!

Thanks especially @stengerh for the performance boost by only building modules, @petski for the https tip and CONFARGS changes and @rwrz for the various improvements. I'll try to update (& test) the gist soon.

@dhohengassner

This comment has been minimized.

Copy link

@dhohengassner dhohengassner commented Aug 16, 2019

Thanks Herman!
Very helpful!!!

@jjrdev

This comment has been minimized.

Copy link

@jjrdev jjrdev commented Sep 4, 2019

great! thanks

@mehyedes

This comment has been minimized.

Copy link

@mehyedes mehyedes commented Sep 26, 2019

Thanks a lot!
This was very helpful for me to be able to compile and load a module in the debian based nginx docker image.
Maybe my gist would be helpful for someone out there
https://gist.github.com/mehyedes/bf5e3870907ddb9c9d3c3fe45d68c43d

@buendias-dev

This comment has been minimized.

Copy link

@buendias-dev buendias-dev commented Nov 25, 2019

Thank you

@sidparasnis

This comment has been minimized.

Copy link

@sidparasnis sidparasnis commented Dec 3, 2019

Sweet. I'm going to try this out with ngx_stream_core_module.

@bakkerpeter

This comment has been minimized.

Copy link

@bakkerpeter bakkerpeter commented Dec 11, 2019

👍

@8i5dev

This comment has been minimized.

Copy link

@8i5dev 8i5dev commented Jan 8, 2020

Thank you

@dimaqq

This comment has been minimized.

Copy link

@dimaqq dimaqq commented Mar 16, 2020

I had to add mkdir -p /usr/src :) somehow modern builder base doesn't have that directory.

I'd also recommend wget -q to suppress unnecessary output.

@marty30

This comment has been minimized.

Copy link

@marty30 marty30 commented Mar 20, 2020

Thank you so much. It took me 2 hours to find this approach after thinking of recreating the complete image from scratch... Many problematic solutions were attempted... But this went very easily.

I've updated the dockerfile a bit by making it easier to switch which module should be added and updated the dev dependencies. My version is customized to use the more headers module (nginx-mod-http-headers-more in the alpine package manager).

My version:

ARG VERSION=alpine
FROM nginx:${VERSION} as builder

ENV MORE_HEADERS_VERSION=0.33
ENV MORE_HEADERS_GITREPO=openresty/headers-more-nginx-module

# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
    wget "https://github.com/${MORE_HEADERS_GITREPO}/archive/v${MORE_HEADERS_VERSION}.tar.gz" -O extra_module.tar.gz

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN  apk add --no-cache --virtual .build-deps \
    gcc \
    libc-dev \
    make \
    openssl-dev \
    pcre-dev \
    zlib-dev \
    linux-headers \
    libxslt-dev \
    gd-dev \
    geoip-dev \
    perl-dev \
    libedit-dev \
    mercurial \
    bash \
    alpine-sdk \
    findutils

SHELL ["/bin/ash", "-eo", "pipefail", "-c"]

RUN rm -rf /usr/src/nginx /usr/src/extra_module && mkdir -p /usr/src/nginx /usr/src/extra_module && \
    tar -zxC /usr/src/nginx -f nginx.tar.gz && \
    tar -xzC /usr/src/extra_module -f extra_module.tar.gz

WORKDIR /usr/src/nginx/nginx-${NGINX_VERSION}

# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') && \
    sh -c "./configure --with-compat $CONFARGS --add-dynamic-module=/usr/src/extra_module/*" && make modules


# Production container starts here
FROM nginx:${VERSION}

COPY --from=builder /usr/src/nginx/nginx-${NGINX_VERSION}/objs/*_module.so /etc/nginx/modules/

.... skipped inserting config files and stuff ...

# Validate the config
RUN nginx -t
@sjpatton

This comment has been minimized.

Copy link

@sjpatton sjpatton commented Apr 9, 2020

Just what I needed. Saves me having to start at square one. Thanks.

@abdennour

This comment has been minimized.

Copy link

@abdennour abdennour commented Aug 30, 2020

Any idea how to compile this https://github.com/abdennour/nginx-http-radius-module in dockerfile ?
It seems that it does not follow the same standard

@mschilde

This comment has been minimized.

Copy link

@mschilde mschilde commented Nov 10, 2020

Many many thanks, this gist helped me compile https://github.com/durkie/ngx_http_mbtiles_module

@pbkh-bunthai

This comment has been minimized.

Copy link

@pbkh-bunthai pbkh-bunthai commented Dec 9, 2020

This is a full complete working script.
Thank me later 😄
Also, thank to @rwrz

FROM nginx:alpine AS builder

# nginx:alpine contains NGINX_VERSION environment variable, like so:
# ENV NGINX_VERSION 1.15.0

# Our NCHAN version
# https://nchan.io/
ENV NCHAN_VERSION 1.1.5


# https://github.com/openresty/echo-nginx-module#version
ENV HTTPECHO_VERSION 0.62 

# Download sources
RUN curl "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -o nginx.tar.gz && \
  curl -L "https://github.com/openresty/echo-nginx-module/archive/v${HTTPECHO_VERSION}.tar.gz" -o httpecho.tar.gz

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
  gcc \
  libc-dev \
  make \
  openssl-dev \
  pcre-dev \
  zlib-dev \
  linux-headers \
  curl \
  gnupg \
  libxslt-dev \
  gd-dev \
  geoip-dev

# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer/-Os} && \
    mkdir /usr/src && \
	tar -zxC /usr/src -f nginx.tar.gz && \
  tar -xzvf "httpecho.tar.gz" && \
  HTTPECHODIR="$(pwd)/echo-nginx-module-${HTTPECHO_VERSION}" && \
  cd /usr/src/nginx-$NGINX_VERSION && \
  ./configure --with-compat $CONFARGS --add-dynamic-module=$HTTPECHODIR && \
  make modules && \
  mv ./objs/*.so /

FROM nginx:alpine
# Extract the dynamic module NCHAN from the builder image
# COPY --from=builder /usr/local/nginx/modules/ngx_nchan_module.so /usr/local/nginx/modules/ngx_nchan_module.so
COPY --from=builder /ngx_http_echo_module.so /usr/local/nginx/modules/ngx_http_echo_module.so
RUN rm /etc/nginx/conf.d/default.conf

# COPY nginx.conf /etc/nginx/nginx.conf
# COPY default.conf /etc/nginx/conf.d/default.conf

COPY /html/index.html /usr/share/nginx/html
COPY /config/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]
@jeffmachado

This comment has been minimized.

Copy link

@jeffmachado jeffmachado commented Dec 28, 2020

Helps here too. Thankss guys! 👊

@jaclu010

This comment has been minimized.

Copy link

@jaclu010 jaclu010 commented Jan 21, 2021

This is great, thanks!

@fuhuo

This comment has been minimized.

Copy link

@fuhuo fuhuo commented Feb 3, 2021

wow,Very helpful!Thanks!

Thank you so much. It took me 2 hours to find this approach after thinking of recreating the complete image from scratch... Many problematic solutions were attempted... But this went very easily.

I've updated the dockerfile a bit by making it easier to switch which module should be added and updated the dev dependencies. My version is customized to use the more headers module (nginx-mod-http-headers-more in the alpine package manager).

My version:

ARG VERSION=alpine
FROM nginx:${VERSION} as builder

ENV MORE_HEADERS_VERSION=0.33
ENV MORE_HEADERS_GITREPO=openresty/headers-more-nginx-module

# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
    wget "https://github.com/${MORE_HEADERS_GITREPO}/archive/v${MORE_HEADERS_VERSION}.tar.gz" -O extra_module.tar.gz

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN  apk add --no-cache --virtual .build-deps \
    gcc \
    libc-dev \
    make \
    openssl-dev \
    pcre-dev \
    zlib-dev \
    linux-headers \
    libxslt-dev \
    gd-dev \
    geoip-dev \
    perl-dev \
    libedit-dev \
    mercurial \
    bash \
    alpine-sdk \
    findutils

SHELL ["/bin/ash", "-eo", "pipefail", "-c"]

RUN rm -rf /usr/src/nginx /usr/src/extra_module && mkdir -p /usr/src/nginx /usr/src/extra_module && \
    tar -zxC /usr/src/nginx -f nginx.tar.gz && \
    tar -xzC /usr/src/extra_module -f extra_module.tar.gz

WORKDIR /usr/src/nginx/nginx-${NGINX_VERSION}

# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') && \
    sh -c "./configure --with-compat $CONFARGS --add-dynamic-module=/usr/src/extra_module/*" && make modules


# Production container starts here
FROM nginx:${VERSION}

COPY --from=builder /usr/src/nginx/nginx-${NGINX_VERSION}/objs/*_module.so /etc/nginx/modules/

.... skipped inserting config files and stuff ...

# Validate the config
RUN nginx -t
@swainsubrat

This comment has been minimized.

Copy link

@swainsubrat swainsubrat commented Jun 25, 2021

@marty30
Is load_module available in the free version of nginx, because it's showing that load_module directive is not allowed here in /etc/nginx/conf.d/nginx.conf

@mtgq

This comment has been minimized.

Copy link

@mtgq mtgq commented Jun 26, 2021

image
error!!

@hermanbanken

This comment has been minimized.

Copy link
Owner Author

@hermanbanken hermanbanken commented Jun 26, 2021

@mtgq this gist provides a way to build modules externally, but I’m not maintaining the gist carefully. Previously this worked, maybe a recent update broke it: you can retry with an older base image.

If you find a fix please let us know in this thread and will add the fix.

Note that I’m not affiliated with NGINX in any way.

@mtgq

This comment has been minimized.

Copy link

@mtgq mtgq commented Jun 26, 2021

my docker version is 20.10.7, build f0df350 , I found another solution https://github.com/nginxinc/docker-nginx/blob/master/modules/README.md, but I don't like this method, I try to solve the error

@brockoffdev

This comment has been minimized.

Copy link

@brockoffdev brockoffdev commented Jul 1, 2021

image
error!!

yeah same here

@aloncarme

This comment has been minimized.

Copy link

@aloncarme aloncarme commented Jul 17, 2021

Thanks @hermanbanken it is very helpful .
I used it to build spnego dynamic module.

@mtgq @brockoffdev
I fixed the error by removing the -fomit-frame-pointer and -g from CONFARGS

RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer -g/-Os} \
@mtgq

This comment has been minimized.

Copy link

@mtgq mtgq commented Jul 17, 2021

Thanks @hermanbanken it is very helpful .
I used it to build spnego dynamic module.

@mtgq @brockoffdev
I fixed the error by removing the -fomit-frame-pointer and -g from CONFARGS

RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer -g/-Os} \

Thank you very much. I will try again

@xgp

This comment has been minimized.

Copy link

@xgp xgp commented Aug 12, 2021

This appears to have some linking problems (perhaps because of alpine's use of musl?):

/ # nginx
2021/08/12 18:54:08 [emerg] 7#7: dlopen() "/etc/nginx/modules/ngx_http_remote_passwd.so" failed (Error relocating /etc/nginx/modules/ngx_http_remote_passwd.so: ngx_http_remote_passwd: symbol not found) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/etc/nginx/modules/ngx_http_remote_passwd.so" failed (Error relocating /etc/nginx/modules/ngx_http_remote_passwd.so: ngx_http_remote_passwd: symbol not found) in /etc/nginx/nginx.conf:1
/ # musl-ldd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment