Skip to content

Instantly share code, notes, and snippets.

@morhekil
Created August 14, 2014 12:18
Show Gist options
  • Save morhekil/1ff0e902ed4de2adcb7a to your computer and use it in GitHub Desktop.
Save morhekil/1ff0e902ed4de2adcb7a to your computer and use it in GitHub Desktop.
Full request/response body logging in nginx
http {
log_format bodylog '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" $request_time '
'<"$request_body" >"$resp_body"';
lua_need_request_body on;
set $resp_body "";
body_filter_by_lua '
local resp_body = string.sub(ngx.arg[1], 1, 1000)
ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
if ngx.arg[2] then
ngx.var.resp_body = ngx.ctx.buffered
end
';
}
server {
listen 1.2.3.4;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://127.0.0.1:3000;
access_log /var/log/nginx/server.log bodylog;
}
}
@aehernandez
Copy link

I'm getting a bunch of escaped characters in using this, especially since my payloads tend to be json objects. For example " gets escaped to \x22. Any solution for that?

@shoeb751
Copy link

@aehernandez If you are using nginx > 1.11.8, you can set escape=json in the log_format directive. http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

@morhekil I understand. Thanks

@mallikharjunrao
Copy link

@morhekil
getting the exception: "set" directive is not allowed here in ~ /nginx/conf/nginx.conf
any solution?

@vmanyushin
Copy link

@mallikharjunrao just move it on server { ... } block

@followdarko
Copy link

followdarko commented Oct 6, 2017

I'm using nginx < 1.11.8 and escape=json is disable. Is there any other solution to decode json? Maybe something with lua

@milgner
Copy link

milgner commented Jan 29, 2018

Looks like this isn't working anymore? Just gave it a try with a current distribution of OpenResty (it already has the lua module baked in) but even moving the set directive into the server block will just have the parser complain [emerg] unknown "resp_body" variable.

Update: I managed to get it to work after all. The problem seemed to originate from the missing nesting.
Once I moved the server block into the http block, it started to work.

@jimmyss04
Copy link

jimmyss04 commented Jun 19, 2018

I'm trying to get the response body but it always returns empty, I have tried both ways below:

  set $resp_body "";
#    body_filter_by_lua '
#                       local resp_body = string.sub(ngx.arg[1], 1, 1000)
#                       ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
#                       if ngx.arg[2] then
#                               ngx.var.resp_body = ngx.ctx.buffered
#                       end
#                       ';
   body_filter_by_lua 'ngx.log(ngx.ERR, ngx.var.resp_body, ngx.var.request_body )';

Any ideas how to capture the response body?

@eugenekolo
Copy link

eugenekolo commented Jul 25, 2018

@jimmyss04

A complete example and works as of 7/25/2018 on Ubuntu 16.04. Some APIs have changed, and the concept of contexts and phases is confusing working with nginx/openresty.

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 768;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    gzip on;
    gzip_disable "msie6";

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" $request_time '
    '\n\n"$req_headers" \n"$req_body" \n>"$resp_body"';


    server {
        listen 80 default_server;
        access_log  /var/log/nginx/server.log bodylog;
     
        location / {
        }

        lua_need_request_body on;

        set $resp_body "";
        set $req_body "";
        set $req_headers "";

        client_body_buffer_size 16k;
        client_max_body_size 16k;

        rewrite_by_lua_block {
            local req_headers = "Headers: ";
            ngx.var.req_body = ngx.req.get_body_data();
            local h, err = ngx.req.get_headers()
            for k, v in pairs(h) do
                req_headers = req_headers .. k .. ": " .. v .. "\n";
            end

            ngx.var.req_headers = req_headers;
        }

        body_filter_by_lua '
        local resp_body = string.sub(ngx.arg[1], 1, 1000)
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
          ngx.var.resp_body = ngx.ctx.buffered
        end
        ';
     }
}

@simonecogno
Copy link

Does this require lua-nginx-module?

@csfreebird
Copy link

@simonecogno yes, it requires.

@csfreebird
Copy link

$req_headers

nginx: [emerg] unknown "req_headers" variable
the version of my Nginx on Ubuntu 16.04 is 1.10.3, installed with 'apt-get install nginx-extras'

@nmishr
Copy link

nmishr commented Mar 8, 2019

Thanks Eugene, your working example helped me figure out what my issue was. Basically, I had a leftover content_by_lua_block { …} which was overwriting the response from the remote server

@kaushalshriyan
Copy link

kaushalshriyan commented Dec 18, 2023

@morhekil I am running Red Hat Enterprise Linux release 8.8 (Ootpa)

`# nginx -t
nginx: [emerg] unknown directive "log_by_lua_block" in /etc/nginx/conf.d/microservice.conf:8
nginx: configuration file /etc/nginx/nginx.conf test failed

nginx -v

nginx version: nginx/1.24.0

nginx -V

nginx version: nginx/1.24.0
built by gcc 8.5.0 20210514 (Red Hat 8.5.0-4) (GCC)
built with OpenSSL 1.1.1k FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
#`

@dragosiscru
Copy link

Hi,
First of all thanks for the code, it helped me a lot. Second, how would we tackle logging gzipped response bodies? I tried lua-zlib, but for some reason I can't make it work. Any suggestions? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment