Edit this page

Hub as a Gateway

SONM Worker node are required to have a public IP address and a properly configured firewall to make all things work properly. But sometimes it is not possible due to either security, privacy or provider limitations. This also includes workers behind the NAT.

To be able to work with these workers a Hub can be configured as a gateway.

Note: we assume that it is possible to build a private network between Hub and workers.

NAT

During start up workers can automatically determine their firewall settings. This can be, for example, restricted NAT, port restricted NAT (PNAT), symmetric UDP firewall and much more. Also it is possible that there is no restrictions for worker's network.

This is done by requesting a specific server using STUN protocol.

If enabled the worker tries to discover its own public IP address and the firewall configuration. If disabled it is treated as having public IP address that is determined automatically and it's the system administrator's responsibility to provide correct access to the worker from the Internet.

These settings are transmitted to a Hub, which (if properly configured) marks the worker as a private, allowing to route requests through itself, i.e. be a gateway.

For every new instance for a specific container we register a virtual service on a Hub using IPVS.

What Is IPVS?

IPVS (IP Virtual Server) or LVS (Linux Virtual Server) is a Linux kernel module, that is 16 years old, based on netfilter. It supports TCP, SCTP, and UDP and can achieve incredibly fast speeds. Other features include NAT, tunneling, and direct routing.

It works by rewriting L2, L3 or L4 headers to be able to forward traffic. IPVS forwards traffic from clients to back-ends, meaning you can load balance anything.

It can work using the following modes:

  • DR - rewrites DST MAC to forward at L2.
  • DNAT - rewrites DST IP, uses the same L4 and behaves like a NAT, rewriting the ip packets and forwarding request and response traffic.
  • IPIP - encapsulates IP and is routable anywhere.

If nothing else configured the response skips the load balancer, so only the request goes through IPVS.

How do we use it?

A Hub configured as a gateway attaches to a public network interface to publish virtual services.

In our configuration we use IPVS in DNAT mode, because both DR and IPIP modes require to have the same virtual service and back-ends ports.

Every container, that requires network, spawned by a hidden worker, exports one or more ports to be able to accept requests. In a Hub we register a new virtual service on a public interface with DNAT mode that forwards all requests to a worker. As clients can be behind the firewall too we also forward all responses back through a Hub.

Note: all forwarding is done inside the kernel, making it blazingly fast.

After containers start successfully, we provide you a virtual service address (usually Hub public IP) and a virtual service port through which requests will be performed.

That's how it works in two words.

Note: that the routing mechanism consumes your Hub's traffic, which is a limited resource. We provide a precise accounting through our Hub API, however it is also possible to see it directly using ipvsadm command.

Configuration

Workers behind the firewall are required to explicitly mark it by configuring a firewall discovery settings:

firewall:
   server: "stun.ekiga.net:3478"

Note, that in the case a worker has public IP address it's always insecure to open all ports for the Internet access. It's a 99.9% probability that auto-discovering detects your router's or OS's firewall, so make sure you'd properly configured your network.

And that's all for workers.

Hubs has more restrictions:

  • It should have a public IP address (better more than one).
  • It has to be run on a Linux machine with IPVS module configured.
  • Administrator privileges are required to run a Hub.
  • Initial kernel tweaking is also required.

The configuration steps (under the root):

Note: do not trust us. Read the manual about each command and sysctl to make a clear understanding what's going on.

  1. Install IPVS module and ipvsadm package for deeper inspection.

    apt-get install ipvsadm
  2. Run ipvsadm once to preconfigure the module. Ensure that IPVS table is empty.

    [email protected]:~$ ipvsadm -Ln
    IP Virtual Server version 1.2.1 (size=65536)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  3. Tells the kernel that you want to allow IP forwarding.

    echo 1 > /proc/sys/net/ipv4/ip_forward
  4. Enable conntrack.

    echo 1 > /proc/sys/net/ipv4/vs/conntrack
  5. At last, enable SNAT using iptables.

    iptables -t nat -A POSTROUTING -j MASQUERADE

    If your network settings aren't change frequently it is suggested to configure SNAT directly specifying private IP address, this gives you a more performant solution.

  6. Add the following lines in your Hub's config.

    # Hub as a gateway settings. Can be omitted indicating that the Hub should not be a gateway.
    gateway:
    # Port range allocated for virtual services if any.
    ports: [32768, 33768]

Example

On a Hub machine start a Hub:

[email protected]:~/go/src/github.com/sonm-io/core$ sudo ./sonmhub
2017-08-28T19:23:42.114+0300    INFO    gateway/gateway_linux.go:47 initializing IPVS context
INFO [08-28|19:23:42] Starting P2P networking
...
2017-08-28T19:23:44.674+0300    INFO    hub/server.go:361   listening for connections from Miners   {"address": "[::]:10002"}
2017-08-28T19:23:44.675+0300    INFO    hub/server.go:370   listening for gRPC API connections  {"address": "[::]:10001"}

On a Worker node check that Docker is enabled.

[email protected]:/home/esafronov# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Start the worker:

[email protected]:~/go/src/github.com/sonm-io/core$ sudo ./sonmworker
2017-08-28T19:27:08.433+0300    DEBUG   miner/builder.go:75 building a miner    {"config": {"HubConfig":{"Endpoint":"m:10002","Resources":null},"FirewallConfig":{"Server":"stun.ekiga.net:3478"},"GPUConfig":null,"SSHConfig":null,"LoggingConfig":{"Level":-1}}}
2017-08-28T19:27:08.433+0300    DEBUG   miner/builder.go:92 discovering public IP address with NAT type, this may take a long ...
2017-08-28T19:27:18.771+0300    INFO    miner/builder.go:109    discovered public IP address    {"addr": "93.255.215.130", "nat": "Symetric UDP firewall"}
...

On the hub machine ensure that IPVS table is still empty:

[email protected]:~$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

On the client machine start some task:

⇒  ./sonmcli hub worker list --addr="m:10001"
Worker: [4c01:9b1:0:1f12::d1:30]:33716      Idle

⇒  ./sonmcli task start "[4c01:9b1:0:1f12::d1:30]:33716" task.yaml --addr="m:10001"
Starting "cocaine/cocaine-core:latest" on miner [4c01:9b1:0:1f12::d1:30]:33716...
ID 755fe2a8-479a-43fb-a842-f4ef7e94cdd0
Endpoint [10053/tcp->93.255.215.128:32898]

Note that a task exports a single TCP port 10053 and we've created a virtual service on 93.255.215.128:32898 endpoint.

Ensure that there is actually IPVS table record on a Hub machine:

[email protected]:/home/esafronov# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  93.255.215.128:32898 wrr
  -> 93.255.215.130:32801          Masq    100    0          0

Perform some request on a client machine using provided virtual service endpoint:

⇒  cocaine-tool locate -n locator --host=93.255.215.128 --port=32898
{
    "api": {
        "0": "resolve",
        "1": "connect",
        "2": "refresh",
        "3": "cluster",
        "4": "publish",
        "5": "routing"
    },
    "endpoints": [
        "172.17.0.2:10053"
    ],
    "version": 3
}

At last let's see some gateway network statistics (on a Hub machine):

[email protected]:/home/esafronov# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=65536)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  93.255.215.128:32898                 1        6        4      337      486
  -> 93.255.215.130:32801                 1        6        4      337      486

Congratulations! You've just configured and tested a Gateway Hub.