Deploying NGINX Plus as an API Gateway, Part 3: Publishing gRPC Services

errors.grpc_conf

# Standard HTTP-to-gRPC status code mappings
# Ref: https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md
#
error_page 400 = @grpc_internal;
error_page 401 = @grpc_unauthenticated;
error_page 403 = @grpc_permission_denied;
error_page 404 = @grpc_unimplemented;
error_page 429 = @grpc_unavailable;
error_page 502 = @grpc_unavailable;
error_page 503 = @grpc_unavailable;
error_page 504 = @grpc_unavailable;

# NGINX-to-gRPC status code mappings
# Ref: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
#
error_page 405 = @grpc_internal; # Method not allowed
error_page 408 = @grpc_deadline_exceeded; # Request timeout
error_page 413 = @grpc_resource_exhausted; # Payload too large
error_page 414 = @grpc_resource_exhausted; # Request URI too large
error_page 415 = @grpc_internal; # Unsupported media type;
error_page 426 = @grpc_internal; # HTTP request was sent to HTTPS port
error_page 495 = @grpc_unauthenticated; # Client certificate authentication error
error_page 496 = @grpc_unauthenticated; # Client certificate not presented
error_page 497 = @grpc_internal; # HTTP request was sent to mutual TLS port
error_page 500 = @grpc_internal; # Server error
error_page 501 = @grpc_internal; # Not implemented

# gRPC error responses
# Ref: https://github.com/grpc/grpc-go/blob/master/codes/codes.go
#
location @grpc_deadline_exceeded {
    add_header grpc-status 4;
    add_header grpc-message 'deadline exceeded';
    return 204;
}

location @grpc_permission_denied {
    add_header grpc-status 7;
    add_header grpc-message 'permission denied';
    return 204;
}

location @grpc_resource_exhausted {
    add_header grpc-status 8;
    add_header grpc-message 'resource exhausted';
    return 204;
}

location @grpc_unimplemented {
    add_header grpc-status 12;
    add_header grpc-message unimplemented;
    return 204;
}

location @grpc_internal {
    add_header grpc-status 13;
    add_header grpc-message 'internal error';
    return 204;
}

location @grpc_unavailable {
    add_header grpc-status 14;
    add_header grpc-message unavailable;
    return 204;
}

location @grpc_unauthenticated {
    add_header grpc-status 16;
    add_header grpc-message unauthenticated;
    return 204;
}

# vim: syntax=nginx

grpc_auth_jwt.conf

location /routeguide. {
    auth_jwt realm=routeguide token=$http_auth_token;
    auth_jwt_key_file my_idp.jwk;
    grpc_pass grpc://routeguide_service;
}

# vim: syntax=nginx

grpc_gateway.conf

log_format grpc_json escape=json '{"timestamp":"$time_iso8601",'
           '"client":"$remote_addr","uri":"$uri","http-status":$status,'
           '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
           '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # grpc-status is usually a trailer
    ''      $sent_http_grpc_status; # Else use the header, whatever its source
}

server {
    listen 50051 http2; # In production, comment out to disable plaintext port
    listen 443   http2 ssl;
    server_name  grpc.example.com;
    access_log   /var/log/nginx/grpc_log.json grpc_json;

    # TLS config
    ssl_certificate      /etc/ssl/certs/grpc.example.com.crt;
    ssl_certificate_key  /etc/ssl/private/grpc.example.com.key;
    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout  5m;
    ssl_ciphers          HIGH:!aNULL:!MD5;
    ssl_protocols        TLSv1.2 TLSv1.3;

    # Routing
    location /routeguide. {
        grpc_pass grpc://routeguide_service;
    }
    location /helloworld. {
        grpc_pass grpc://helloworld_service;
    }

    # Error responses
    include conf.d/errors.grpc_conf; # gRPC-compliant error responses
    default_type application/grpc;   # Ensure gRPC for all error responses
}

# Backend gRPC servers
#
upstream routeguide_service {
    zone routeguide_service 64k;
    server 127.0.0.1:10001;
    server 127.0.0.1:10002;
    server 127.0.0.1:10003;
}

upstream helloworld_service {
    zone helloworld_service 64k;
    server 127.0.0.1:20001;
    server 127.0.0.1:20002;
}

# vim: syntax=nginx

grpc_health_check.conf

log_format grpc_json escape=json '{"timestamp":"$time_iso8601",'
           '"client":"$remote_addr","uri":"$uri","http-status":$status,'
           '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
           '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # grpc-status is usually a trailer
    ''      $sent_http_grpc_status; # Else use the header, whatever its source
}

server {
    listen 50051 http2; # Plaintext

    # Routing
    location /routeguide. {
        grpc_pass grpc://routeguide_service;
        health_check type=grpc grpc_status=12; # 12=unimplemented
    }
    location /helloworld. {
        grpc_pass grpc://helloworld_service;
        health_check type=grpc grpc_status=12; # 12=unimplemented
    }
}

# Backend gRPC servers
#
upstream routeguide_service {
    zone routeguide_service 64k;
    server 127.0.0.1:10001;
    server 127.0.0.1:10002;
    server 127.0.0.1:10003;
}

upstream helloworld_service {
    zone helloworld_service 64k;
    server 127.0.0.1:20001;
    server 127.0.0.1:20002;
}

# vim: syntax=nginx

grpc_precise_routing.conf

# Service-level routing
location /routeguide.RouteGuide/ {
    grpc_pass grpc://routeguide_service_default;
}

# Method-level routing
location = /routeguide.RouteGuide/RouteChat {
    grpc_pass grpc://routeguide_service_streaming;
}

# vim: syntax=nginx

helloworld.Dockerfile

# This Dockerfile runs the helloworld server from
# https://grpc.io/docs/quickstart/go.html

FROM golang
RUN go get -u google.golang.org/grpc
WORKDIR $GOPATH/src/google.golang.org/grpc/examples/helloworld

EXPOSE 50051
CMD ["go", "run", "greeter_server/main.go"]

routeguide.Dockerfile

# This Dockerfile runs the RouteGuide server from
# https://grpc.io/docs/tutorials/basic/python.html

FROM python
RUN pip install grpcio-tools 
RUN git clone -b v1.14.x https://github.com/grpc/grpc
WORKDIR grpc/examples/python/route_guide

EXPOSE 50051
CMD ["python", "route_guide_server.py"]

I recently tried out this config with a request streaming gRPC service and found that the client was erroring out consistently after a certain number, (~400 - 1000) of requests were streamed to the server. The error was RESOURCE_EXHAUSTED but the gRPC server showed no such error.

After much hair pulling, I remembered that was was proxying everything through NGINX using the above config. Once I directly connected to the gRPC server, bypassing NGINX, the error went away.

There appears to be at least 1 other person who ran into this, but I haven't been able to find mention of this behavior anywhere else. So I thought I'd comment here to provide another search engine hit for the RESOURCE_EXHAUSTED search term.

@coryvirok did you ever find a solution to the exhausted error?

Unfortunately no. I didn't revisit using NGINX after switching to directly connecting to the gRPC server.

Hey guys, I solved it finally by just increasing the request body size but not sure how efficient that solution is...

@nitheshkumar2000, I ran into the same situation. Can you please share the what parameter value you changed in order to fix this? Thanks

@nitheshkumar2000, I ran into the same situation. Can you please share the what parameter value you changed in order to fix this? Thanks

https://stackoverflow.com/questions/28476643/default-nginx-client-max-body-size
See if this can help!

I still wouldn't suggest if we're using for grpc streaming because it acts as a middleman so the streaming capability of the grpc is affected, there's a lot of latency with this approach!

For a discussion of these files, see Deploying NGINX Plus as an API Gateway, Part 3: Publishing gRPC Services