Passing TCP socket descriptors around

README.md

Passing TCP socket descriptors around

In linux, normally, it is impossible to "bind()" to the same TCP port twice. If you try to bind to the same port from second proces unix processes you'll see:

socket.error: [Errno 98] Address already in use

It is possible on UDP, using SO_REUSEPORT option, but that's separate story.

There are two possible ways to pass a socket to a different process without interruption.

If you can sustain interruption - just "close" the socket in one process and "bind" in another one.

This is what haproxy appears to be doing:

http://comments.gmane.org/gmane.comp.web.haproxy/7815

During the process switch, nobody is listening anymore, for a very
short time: the old process must release the listening port to
let the new one bind a new socket on it.

fork

After a parent does fork(), in linux, both parent and child may do "accept" and receive a connection. When doing blocking "accept" syscall, the incoming connections will be spread in round-robin fashion, for example (see fork_server.py):

connection got in parent from ('127.0.0.1', 48306)
connection got in child from ('127.0.0.1', 48308)
connection got in parent from ('127.0.0.1', 48310)
connection got in child from ('127.0.0.1', 48312)
connection got in parent from ('127.0.0.1', 48314)

sendmsg/recvmsg API

Sendmsg and Recvmsg syscalls can be used to pass file descriptor over a unix socket. See "man cmsg" and "man 7 unix" under "Ancillary Messages" sections.

http://linux.die.net/man/3/cmsg

http://linux.die.net/man/7/unix

The API is not well documented, but that's for example what node.js (actually libuv) uses:

https://github.com/joyent/node/blob/d2dd9d108d6bd61ac49522450b98ad57eac5be45/deps/uv/src/unix/stream.c#L409-412

This must be passed over unix socket. I'm not really sure how that works internally in the kernel.

basic_server.py

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# REUSEPORT: 15
#s.setsockopt(socket.SOL_SOCKET, 15, 1)


s.bind(('0.0.0.0', 1234))
s.listen(10)

print " [*] listening on :1234"
while True:
    (c, addr) = s.accept()
    print 'got ', addr
    c.send('hello world\n')
    c.close()

fork_server.py

import os
import socket
import time

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

s.bind(('0.0.0.0', 1234))
s.listen(10)


def accept(descr):
    (c, addr) = s.accept()
    print 'connection got in', descr, 'from', addr
    c.send('hello world\n')
    c.close()


print " [*] listening on :1234, next three connections will be handled by parent"

for i in range(3):
    accept('parent')

print " [*] good. forking now"
child_pid = os.fork()
if child_pid == 0:
    # child
    print " [*] child: okay, accepting now!"
    while True:
        accept('child')

else:
    # parent
    print " [*] parent: okay, accepting next three connections"
    for i in range(3):
        accept('parent')
    print " [*] parent: I'm done, waiting few sec"
    time.sleep(5)
    print " [*] parent: closing bind fd"
    s.close()
    print " [*] parent: waiting again"
    time.sleep(5)
    print " [*] parent: quitting, remember to kill the child! run:"
    print "kill ", child_pid

btw haproxy long read on the topic — https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/