Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Full request/response body logging in nginx
http {
log_format bodylog '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" $request_time '
'<"$request_body" >"$resp_body"';
lua_need_request_body on;
set $resp_body "";
body_filter_by_lua '
local resp_body = string.sub(ngx.arg[1], 1, 1000)
ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
if ngx.arg[2] then
ngx.var.resp_body = ngx.ctx.buffered
end
';
}
server {
listen 1.2.3.4;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://127.0.0.1:3000;
access_log /var/log/nginx/server.log bodylog;
}
}
@morhekil

This comment has been minimized.

Copy link
Owner Author

@morhekil morhekil commented Aug 14, 2014

This will log full body of all requests and responses into nginx log. Depends on lua, so nginx needs to be built with lua module as well.

@xiaochongchong86

This comment has been minimized.

Copy link

@xiaochongchong86 xiaochongchong86 commented Sep 15, 2014

the request body is ok, but the reponse body is null ,

@xiaochongchong86

This comment has been minimized.

Copy link

@xiaochongchong86 xiaochongchong86 commented Sep 15, 2014

I can't get the response body in the access log but , can get it by
ngx.log(ngx.ERR, ngx.var.resp_body, ngx.var.request_body )

@shoeb751

This comment has been minimized.

Copy link

@shoeb751 shoeb751 commented Oct 13, 2016

just wondering why you did local resp_body = string.sub(ngx.arg[1], 1, 1000) instead of local resp_body = string.sub(ngx.arg[1], 1, -1) or perhaps local resp_body = ngx.arg[1] for that matter?

The last two approaches makes sure that you do not lose the response body. Which happens in your case where it gets truncated.

@morhekil

This comment has been minimized.

Copy link
Owner Author

@morhekil morhekil commented Jan 30, 2017

@shoeb751 this is to avoid dumbing big binary files into the logs. I mostly use it to monitor API requests/responses, and 1KB covers all my need - it would make sense for you adjust it per your own requirements, of course.

@aehernandez

This comment has been minimized.

Copy link

@aehernandez aehernandez commented Apr 28, 2017

I'm getting a bunch of escaped characters in using this, especially since my payloads tend to be json objects. For example " gets escaped to \x22. Any solution for that?

@shoeb751

This comment has been minimized.

Copy link

@shoeb751 shoeb751 commented May 25, 2017

@aehernandez If you are using nginx > 1.11.8, you can set escape=json in the log_format directive. http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

@morhekil I understand. Thanks

@mallikharjunrao

This comment has been minimized.

Copy link

@mallikharjunrao mallikharjunrao commented Aug 22, 2017

@morhekil
getting the exception: "set" directive is not allowed here in ~ /nginx/conf/nginx.conf
any solution?

@vmanyushin

This comment has been minimized.

Copy link

@vmanyushin vmanyushin commented Aug 29, 2017

@mallikharjunrao just move it on server { ... } block

@followdarko

This comment has been minimized.

Copy link

@followdarko followdarko commented Oct 6, 2017

I'm using nginx < 1.11.8 and escape=json is disable. Is there any other solution to decode json? Maybe something with lua

@milgner

This comment has been minimized.

Copy link

@milgner milgner commented Jan 29, 2018

Looks like this isn't working anymore? Just gave it a try with a current distribution of OpenResty (it already has the lua module baked in) but even moving the set directive into the server block will just have the parser complain [emerg] unknown "resp_body" variable.

Update: I managed to get it to work after all. The problem seemed to originate from the missing nesting.
Once I moved the server block into the http block, it started to work.

@jimmyss04

This comment has been minimized.

Copy link

@jimmyss04 jimmyss04 commented Jun 19, 2018

I'm trying to get the response body but it always returns empty, I have tried both ways below:

  set $resp_body "";
#    body_filter_by_lua '
#                       local resp_body = string.sub(ngx.arg[1], 1, 1000)
#                       ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
#                       if ngx.arg[2] then
#                               ngx.var.resp_body = ngx.ctx.buffered
#                       end
#                       ';
   body_filter_by_lua 'ngx.log(ngx.ERR, ngx.var.resp_body, ngx.var.request_body )';

Any ideas how to capture the response body?

@eugenekolo

This comment has been minimized.

Copy link

@eugenekolo eugenekolo commented Jul 25, 2018

@jimmyss04

A complete example and works as of 7/25/2018 on Ubuntu 16.04. Some APIs have changed, and the concept of contexts and phases is confusing working with nginx/openresty.

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 768;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    gzip on;
    gzip_disable "msie6";

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" $request_time '
    '\n\n"$req_headers" \n"$req_body" \n>"$resp_body"';


    server {
        listen 80 default_server;
        access_log  /var/log/nginx/server.log bodylog;
     
        location / {
        }

        lua_need_request_body on;

        set $resp_body "";
        set $req_body "";
        set $req_headers "";

        client_body_buffer_size 16k;
        client_max_body_size 16k;

        rewrite_by_lua_block {
            local req_headers = "Headers: ";
            ngx.var.req_body = ngx.req.get_body_data();
            local h, err = ngx.req.get_headers()
            for k, v in pairs(h) do
                req_headers = req_headers .. k .. ": " .. v .. "\n";
            end

            ngx.var.req_headers = req_headers;
        }

        body_filter_by_lua '
        local resp_body = string.sub(ngx.arg[1], 1, 1000)
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
          ngx.var.resp_body = ngx.ctx.buffered
        end
        ';
     }
}
@simonecogno

This comment has been minimized.

Copy link

@simonecogno simonecogno commented Oct 9, 2018

Does this require lua-nginx-module?

@csfreebird

This comment has been minimized.

Copy link

@csfreebird csfreebird commented Feb 14, 2019

@simonecogno yes, it requires.

@csfreebird

This comment has been minimized.

Copy link

@csfreebird csfreebird commented Feb 14, 2019

$req_headers

nginx: [emerg] unknown "req_headers" variable
the version of my Nginx on Ubuntu 16.04 is 1.10.3, installed with 'apt-get install nginx-extras'

@nmishr

This comment has been minimized.

Copy link

@nmishr nmishr commented Mar 8, 2019

Thanks Eugene, your working example helped me figure out what my issue was. Basically, I had a leftover content_by_lua_block { …} which was overwriting the response from the remote server

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.