Skip to content

Instantly share code, notes, and snippets.

@3Hren

3Hren/IPVS.md

Last active July 19, 2022 09:52
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save 3Hren/0ad6321693f1114dc3af5f13481b8db3 to your computer and use it in GitHub Desktop.
Save 3Hren/0ad6321693f1114dc3af5f13481b8db3 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
IPVS.md

Hub as a Gateway

Miners are required to have a public IP address and a properly configured firewall to make all things work properly. But sometimes it is not possible due to either security, privacy or provider limitations. This also includes miners behind the NAT.

To be able to work with these miners a Hub can be configured as a gateway.

Note: we assume that it is possible to build a private network between Hub and miners.

NAT

During start up miners can automatically determine their firewall settings. This can be, for example, restricted NAT, port restricted NAT (PNAT), symmetric UDP firewall and much more. Also it is possible that there is no restrictions for miner's network.

This is done by requesting a specific server using STUN protocol.

If enabled the miner tries to discover its own public IP address and the firewall configuration. If disabled it is treated as having public IP address that is determined automatically and it's the system administrator's responsibility to provide correct access to the miner from the Internet.

These settings are transmitted to a Hub, which (if properly configured) marks the miner as a private, allowing to route requests through itself, i.e. be a gateway.

For every new instance for a specific container we register a virtual service on a Hub using IPVS.

What Is IPVS?

IPVS (IP Virtual Server) or LVS (Linux Virtual Server) is a Linux kernel module, that is 16 years old, based on netfilter. It supports TCP, SCTP, and UDP and can achieve incredibly fast speeds. Other features include NAT, tunneling, and direct routing.

It works by rewriting L2, L3 or L4 headers to be able to forward traffic. IPVS forwards traffic from clients to back-ends, meaning you can load balance anything.

It can work using the following modes:

  • DR - rewrites DST MAC to forward at L2.
  • DNAT - rewrites DST IP, uses the same L4 and behaves like a NAT, rewriting the ip packets and forwarding request and response traffic.
  • IPIP - encapsulates IP and is routable anywhere.

If nothing else configured the response skips the load balancer, so only the request goes through IPVS.

How do we use it?

A Hub configured as a gateway attaches to a public network interface to publish virtual services.

In our configuration we use IPVS in DNAT mode, because both DR and IPIP modes require to have the same virtual service and back-ends ports.

Every container, that requires network, spawned by a hidden miner, exports one or more ports to be able to accept requests. In a Hub we register a new virtual service on a public interface with DNAT mode that forwards all requests to a miner. As clients can be behind the firewall too we also forward all responses back through a Hub.

Note: all forwarding is done inside the kernel, making it blazingly fast.

After containers start successfully, we provide you a virtual service address (usually Hub public IP) and a virtual service port through which requests will be performed.

That's how it works in two words.

Note: that the routing mechanism consumes your Hub's traffic, which is a limited resource. We provide a precise accounting through our Hub API, however it is also possible to see it directly using ipvsadm command.

Configuration

Miners behind the firewall are required to explicitly mark it by configuring a firewall discovery settings:

firewall:
   server: "stun.ekiga.net:3478"

Note, that in the case a miner has public IP address it's always insecure to open all ports for the Internet access. It's a 99.9% probability that auto-discovering detects your router's or OS's firewall, so make sure you'd properly configured your network.

And that's all for miners.

Hubs has more restrictions:

  • It should have a public IP address (better more than one).
  • It has to be run on a Linux machine with IPVS module configured.
  • Administrator privileges are required to run a Hub.
  • Initial kernel tweaking is also required.

The configuration steps (under the root):

Note: do not trust us. Read the manual about each command and sysctl to make a clear understanding what's going on.

  1. Install IPVS module and ipvsadm package for deeper inspection.
apt-get install ipvsadm
  1. Run ipvsadm once to preconfigure the module. Ensure that IPVS table is empty.
esafronov@h:~$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  1. Tells the kernel that you want to allow IP forwarding.
echo 1 > /proc/sys/net/ipv4/ip_forward
  1. Enable conntrack.
echo 1 > /proc/sys/net/ipv4/vs/conntrack
  1. At last, enable SNAT using iptables.
iptables -t nat -A POSTROUTING -j MASQUERADE

If your network settings aren't change frequently it is suggested to configure SNAT directly specifying private IP address, this gives you a more performant solution.

  1. Add the following lines in your Hub's config.
# Hub as a gateway settings. Can be omitted indicating that the Hub should not be a gateway.
gateway:
  # Port range allocated for virtual services if any.
  ports: [32768, 33768]

Example

On a Hub machine start a Hub:

esafronov@h:~/go/src/github.com/sonm-io/core$ sudo ./sonmhub
2017-08-28T19:23:42.114+0300	INFO	gateway/gateway_linux.go:47	initializing IPVS context
INFO [08-28|19:23:42] Starting P2P networking
...
2017-08-28T19:23:44.674+0300	INFO	hub/server.go:361	listening for connections from Miners	{"address": "[::]:10002"}
2017-08-28T19:23:44.675+0300	INFO	hub/server.go:370	listening for gRPC API connections	{"address": "[::]:10001"}

On a Miner machine check that Docker is enabled.

root@m:/home/esafronov# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Start the miner:

esafronov@m:~/go/src/github.com/sonm-io/core$ sudo ./sonmminer
2017-08-28T19:27:08.433+0300	DEBUG	miner/builder.go:75	building a miner	{"config": {"HubConfig":{"Endpoint":"m:10002","Resources":null},"FirewallConfig":{"Server":"stun.ekiga.net:3478"},"GPUConfig":null,"SSHConfig":null,"LoggingConfig":{"Level":-1}}}
2017-08-28T19:27:08.433+0300	DEBUG	miner/builder.go:92	discovering public IP address with NAT type, this may take a long ...
2017-08-28T19:27:18.771+0300	INFO	miner/builder.go:109	discovered public IP address	{"addr": "93.255.215.130", "nat": "Symetric UDP firewall"}
...

On the hub machine ensure that IPVS table is still empty:

esafronov@h:~$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

On the client machine start some task:

⇒  ./sonmcli miner list --addr="m:10001"
Miner: [4c01:9b1:0:1f12::d1:30]:33716		Idle

⇒  ./sonmcli task start "[4c01:9b1:0:1f12::d1:30]:33716" task.yaml --addr="m:10001"
Starting "cocaine/cocaine-core:latest" on miner [4c01:9b1:0:1f12::d1:30]:33716...
ID 755fe2a8-479a-43fb-a842-f4ef7e94cdd0
Endpoint [10053/tcp->93.255.215.128:32898]

Note that a task exports a single TCP port 10053 and we've created a virtual service on 93.255.215.128:32898 endpoint.

Ensure that there is actually IPVS table record on a Hub machine:

root@h:/home/esafronov# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  93.255.215.128:32898 wrr
  -> 93.255.215.130:32801          Masq    100    0          0

Perform some request on a client machine using provided virtual service endpoint:

⇒  cocaine-tool locate -n locator --host=93.255.215.128 --port=32898
{
    "api": {
        "0": "resolve",
        "1": "connect",
        "2": "refresh",
        "3": "cluster",
        "4": "publish",
        "5": "routing"
    },
    "endpoints": [
        "172.17.0.2:10053"
    ],
    "version": 3
}

At last let's see some gateway network statistics (on a Hub machine):

root@h:/home/esafronov# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  93.255.215.128:32898                 1        6        4      337      486
  -> 93.255.215.130:32801                 1        6        4      337      486

Congratulations! You've just configured and tested a Gateway Hub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment