sp-serial-mapping-00.txt

TTY/Serial Scalability Protocol Transport
==========================================

This document describes a new transport for the Scalability Protocols (SP). The
Scalability Protocols (as implemented in the nanomsg library) specify
lightweight protocols for common message-passing patterns in distributed
systems. They are specifically desgined to accomodate both new message types
and new message transports (as well as being completely agnostic to the content
or encoding of all messages).

We follow a layered protocol approach. The physical layer is assumed to be a
simple duplex (bidirectional) byte stream with relatively low throughput (tens
or hundreds of kilobaud, eg 115200 baud, or 14.4 kilobytes per second). No
assumptions are made about control flow or out of band signalling: the serial
stream may or may not be RS-232 compliant and may or may not indicate link-up
or channel resets.  Ordered transmission is assumed, but data reliability is
not assumed.

This transport is originally intended for use over multi-meter
low-conductor-count "UART" links between embedded microcontroller systems and a
host computer, with just two conductors used for RX/TX serial transmission. The
transport is probably flexible enough to be used in other simple physical layer
links like plastic optical cabling, free-space optical (infrared LED), simple
low-baud radio links, etc.

The closest existing SP transport is probably UDP, and UDP was used as a
guideline for this protocol. Note that as of Summer 2015 the nanomsg library
does not actually implement the UDP transport.

To achieve robust message framing, SLIP encoding is used. The motivation is for
end nodes to synchronize to the start and end of message streams if they reset
or connect in the middle of transmissions, or to recover from dropped bytes.

SLIP is defined in RFC 1055. In summary, SLIP means that binary or ASCII data
is sent raw over the wire in blocks seperated by SLIP_END bytes. When the
SLIP_END byte occurs in the raw message, it is escaped using SLIP_ESC and
SLIP_ESC_END; likewise for SLIP_ESC and SLIP_ESC_ESC. The process is reversed
at the receiving end.

The SLIP ASCII codes (in hex) are:

    =============   =========
    SLIP_END        0xC0
    SLIP_ESC        0xDB
    SLIP_ESC_END    0xDC
    SLIP_ESC_ESC    0xDD
    =============   =========

Preceeding an END byte just before every message is optional; zero-length
messages should be ignored.

To achieve at least partial data reliability in the face of corruption,
checksums are included in the protocol header. In this version of the transport
protocol, the checksum is simply used to detect corruption, not correct for it.
Similar to the SP UDP transport, there is no mechanism for requesting
re-transmission of dropped or corrupted data. Corruption or loss can be
corrected in the case of the REQ/REP protocol by resending the request, but
there is no work around for the PUB/SUB or other protocols.

If TCP-style guarantees on message transmission are required, it is noted that 
TCP/IP tunneling is possible with either the SLIP or PPP and a full TCP stack
running on the microcontroller.

Header Definition
---------------------

The SP-specific serial transport header should be at the start of every nanomsg
SLIP message. It is two 32-bit words long::

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   Version     |     Subport   |         nanomsg Type          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |            Checksum           |         Payload Length        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

*Version* is nanomsg version (currently using 98 (0x62) for experimental use
with this serial transport).

*Subport* is a UDP-like port. As an optional/recommended convention, when
mirroring or mapping subports to "real" 16-bit UDP or TCP ports, add 0x0800
(4096). Eg, a proxy program could connect serial:// subport 5  to tcp:// port
4101.

*nanomsg Type* selects the message payload type/role (eg, PUB, SUB, REQ, REP).
See below.

*Checksum* is calculated UDP style. It is calculated over the entire message
contents (before SLIP encoding), with the checksum field set to 0x0000. If the
total message is not 16-bit aligned, it should be extended with a single NULL
(0x00) byte.

*Payload Length* is the length of the message payload, not including the
header.


The SP types are documented in per-protocol RFC documents. Here is a partial
summary:

    =====   ==========  ==================
    Hex     Pattern     Role    
    =====   ==========  ==================
    0x10    PAIR        (any)
    0x20    PUB/SUB     PUB
    0x21    PUB/SUB     SUB
    0x30    REQ/REP     REQ
    0x31    REQ/REP     REP
    =====   ==========  ==================


General Comments
--------------------

For microcontroller applications which already have an RTOS with a network
stack, using an IP-over-Serial protocol like PPP or SLIP might be more
appropriate than this encoding.

It could potentially make microcontroller implementation much faster to move
the checksum to the end of the SLIP message, so that is can be calculated or
verified in a streaming fashion.

For this serial transport, both memory (RAM) and transport bandwidth are scarce
resources. Message content should be kept 32-bit word aligned so that binary
data words can be copied or accessed faster (eg, arrays of 32 bit floating
point values, or arrays of 16 bit unsigned integer values).

As much fixed-size processing as possible is very helpful for portable
microcontroller code with no (or minimal) heap usage. This rules out variable
size fields. The MCU could limit all messages to a fixed length (eg 512 bytes).

When using SLIP encoding and framing, try to ensure that the bytes 0xC0 and
0xDB don't end up in the header. This guides subport selection.

Note that SLIP has a worst-case encoding overhead of 100% (a raw message
entirely made of SLIP_END or SLIP_ESC packets would be twice as long on the
wire). Bandwidth safety margins should be planned for accordingly to meet the
required reliability of data transport without delays or buffer overflows.

Potential Changes
--------------------

- move checksum to end of message payload for streaming calculation
- switch to a shorter or longer or stronger checksum
- could use a 16-bit subport for compatibility with network ports
- could add a single "magic" byte (or multiple) at the begining of header
  (nanomsg does this for, eg, TCP)
- could compress the 'type' field down to 8 bits (nanomsg uses full 16 bit
  fields)

Could add a mechanism for "connecting" and "disconnecting" to subports?
Probably not necessary, current behavior is UDP-like. Should follow UDP
transport guidelines from nanomsg UDP RFC.