Buildkit TCP service behind Envoy Proxy

00_README.md

Architecture

Envoy filters out Prune/DiskUsage method. see envoy-config.yaml below.

              +-------+     +-----------+
buidctl <---->+ Envoy +<--->+ Buildkitd +
              +-------+     +-----------+

How to Try

$ git clone https://gist.github.com/ea9582347e2c9d0bb519cdae1b438ead.git envoy-builkitd
$ cd envoy-builkitd

$ docker-compose up
...you see logs...

• envoy listens: localhost:1234
• envoy forwards requests to buildkitd
• buildkitd also listens localhost:11234 for debug

Issue

builctl with direct access to buildkit works.

$ buildctl --debug --addr tcp://127.0.0.1:11234 debug workers
ID                              PLATFORMS
xflodox4is8qyxb9ewq9bl1l3       linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6

But buildctl can't connect to buildkit thgourh envoy.

$ buildctl --debug --addr tcp://127.0.0.1:1234 debug workers
error: rpc error: code = Unavailable desc = transport is closing
failed to list workers
github.com/moby/buildkit/client.(*Client).ListWorkers
        /src/client/workers.go:32
github.com/moby/buildkit/cmd/buildctl/debug.listWorkers
        /src/cmd/buildctl/debug/workers.go:41
github.com/urfave/cli.HandleAction
        /src/vendor/github.com/urfave/cli/app.go:523
github.com/urfave/cli.Command.Run
        /src/vendor/github.com/urfave/cli/command.go:174
github.com/urfave/cli.(*App).RunAsSubcommand
        /src/vendor/github.com/urfave/cli/app.go:404
github.com/urfave/cli.Command.startApp
        /src/vendor/github.com/urfave/cli/command.go:373
github.com/urfave/cli.Command.Run
        /src/vendor/github.com/urfave/cli/command.go:102
github.com/urfave/cli.(*App).Run
        /src/vendor/github.com/urfave/cli/app.go:276
main.main
        /src/cmd/buildctl/main.go:100
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

Envoy's debug lots are like this. invalid frame: Invalid HTTP header field was received on stream 1 is the direct cause.

envoy_1      | [2020-07-14 13:51:17.913][14][debug][conn_handler] [source/server/connection_handler_impl.cc:372] [C8] new connection
envoy_1      | [2020-07-14 13:51:17.914][14][debug][http2] [source/common/http/http2/codec_impl.cc:970] [C8] updating connection-level initial window size to 268435456
envoy_1      | [2020-07-14 13:51:17.916][14][debug][http] [source/common/http/conn_manager_impl.cc:268] [C8] new stream
envoy_1      | [2020-07-14 13:51:17.916][14][debug][http2] [source/common/http/http2/codec_impl.cc:670] [C8] invalid frame: Invalid HTTP header field was received on stream 1
envoy_1      | [2020-07-14 13:51:17.917][14][debug][http] [source/common/http/conn_manager_impl.cc:283] [C8] dispatch error: The user callback function failed
envoy_1      | [2020-07-14 13:51:17.917][14][debug][http] [source/common/http/conn_manager_impl.cc:1936] [C8][S4570179258856159313] stream reset
envoy_1      | [2020-07-14 13:51:17.917][14][debug][connection] [source/common/network/connection_impl.cc:109] [C8] closing data_to_write=0 type=2
envoy_1      | [2020-07-14 13:51:17.917][14][debug][connection] [source/common/network/connection_impl_base.cc:30] [C8] setting delayed close timer with timeout 1000 ms
envoy_1      | [2020-07-14T13:51:17.916Z] "- - HTTP/2" 0 DPE 0 0 1 - "-" "-" "-" "-" "-"
envoy_1      | [2020-07-14 13:51:18.919][14][debug][connection] [source/common/network/connection_impl_base.cc:44] [C8] triggered delayed close
envoy_1      | [2020-07-14 13:51:18.920][14][debug][connection] [source/common/network/connection_impl.cc:200] [C8] closing socket: 1
envoy_1      | [2020-07-14 13:51:18.920][14][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C8] adding to cleanup list
envoy_1      | [2020-07-14 13:51:18.924][12][debug][conn_handler] [source/server/connection_handler_impl.cc:372] [C9] new connection
envoy_1      | [2020-07-14 13:51:18.926][12][debug][connection] [source/common/network/connection_impl.cc:558] [C9] remote close
envoy_1      | [2020-07-14 13:51:18.926][12][debug][connection] [source/common/network/connection_impl.cc:200] [C9] closing socket: 0
envoy_1      | [2020-07-14 13:51:18.927][12][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C9] adding to cleanup list
envoy_1      | [2020-07-14 13:51:19.212][1][debug][main] [source/server/server.cc:177] flushing stats

docker-compose.yml

version: '3'
services:
  envoy:
    image: envoyproxy/envoy:v1.14.2
    command: "--config-path /etc/envoy/envoy-config.yaml --log-level debug"
    volumes:
     - .:/etc/envoy
    ports:
     - "1234:1234" # buidkitd
     - "8001:8001" # envoy admin
    links:
    - buildkitd
  buildkitd:
    image: moby/buildkit:rootless
    # listens on 0.0.0.0 for debug
    command: "--debug --addr=tcp://0.0.0.0:11234 --oci-worker-no-process-sandbox"
    ports:
    - "11234:11234" # direct access (for clarifying buildkitd works)
    security_opt: 
    - seccomp:unconfined
    - apparmor:unconfined

envoy-config.yaml

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 1234
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          http2_protocol_options: {}
          route_config:
            name: local_route
            virtual_hosts:
            - name: service
              domains:
              - "*"
              routes:
              - match:
                  safe_regex:
                    google_re2: {}
                    regex: "\/moby.buildkit.v1.Control\/(Status|Solve|Session|ListWorkers).*"
                route:
                  cluster: local_service
                  timeout: "0s"
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          http_filters:
          - name: envoy.router
            config: {}
  clusters:
  - name: local_service
    http2_protocol_options: {}
    connect_timeout: 0.250s
    type: strict_dns
    lb_policy: round_robin
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 2
      healthy_threshold: 2
      tcp_health_check: {}
    hosts:
    - socket_address:
        address: buildkitd
        port_value: 11234
admin:
  access_log_path: /dev/stdout
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Try this as your envoy-config.yaml

static_resources:
  listeners:
    - name: listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 1234
      filter_chains:
        - filters:
            - name: envoy.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
                stat_prefix: tcp_local_service
                cluster: local_service
  clusters:
    - name: local_service
      connect_timeout: 0.250s
      type: strict_dns
      lb_policy: round_robin
      hosts:
        - socket_address:
            address: buildkitd
            port_value: 11234
admin:
  access_log_path: /dev/stdout
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Thanks. yeah, tcp_proxy works as a charm!! However, I wanted to filter-out several grpc methods in envoy. So I think I need to realize http connection manager. note: I updated my envoy-config.yaml.