Skip to content

Instantly share code, notes, and snippets.

@kofemann

kofemann/dcap-client-server.md

Created July 1, 2019 12:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kofemann/56339b92553bf5d2ea05e27559c21f6e to your computer and use it in GitHub Desktop.
Save kofemann/56339b92553bf5d2ea05e27559c21f6e to your computer and use it in GitHub Desktop.
Download ZIP
DCAP data channel wire protocol
Raw
dcap-client-server.md

Overview

  • The Mover-Client connection is a binary connection which, TCP wise, may be initiated by either parties.
  • The mover starts the communication by sending the Mover HELLO BLOCK (see below).
  • All subsequent commands are initiated by the client.
  • The protocol is limited to synchronous request-reply communication. Therefore a command can only be issued after the previous command was completed.
  • The connection can either be in Command Mode, which is the initial mode, or in Data Transfer Mode.
  • Each command has to be acknowledged immediately by an REQUEST ACK. The REQUEST ACK reports the request state and in some cases additional, request dependent arguments . (e.g. LOCATE)
  • If the REQUEST ACK indicates success, the connection switches to Data Transfer Mode for READ and WRITE requests.
  • In Data Transfer Mode the data is send in DATA CHAINS ( see below).
  • The Data Chain (READ and WRITE) is followed by a REQUEST FIN, sent from the mover to the client, which may contain error conditions observed by the server (mover).
  • While the connection is in Data Transfer Mode, the 'non transferring' party may send an INTERRUPT REQUEST BLOCK. The 'interrupt receiving' party should end the datasending as soon as possible by finishing the current block in the sequence and then sending the final '-1' data block chain delimiter.
  • Command Data is sent in Network Byte Order

Command - Data Flow

Write Request Sequence

Client-Server Request / Data
--> WRITE REQUEST
<-- WRITE REQUEST ACK
--> DATA CHAIN
<-- WRITE REQUEST FIN

Read Request Sequence

Client-Server Request / Data
--> READ REQUEST
<-- READ REQUEST ACK
<-- DATA CHAIN
<-- READ REQUEST FIN

Seek/Locate/Status Request Sequence

Client-Server Request / Data
--> SEEK/LOCATE/STATUS REQUEST
<-- SEEK REQUEST ACK

Close Request Sequence

Client-Server Request / Data
--> CLOSE REQUEST
<-- CLOSE REQUEST ACK

Interrupt Request Sequence

Data Chain sender - Data Chain receiver Request / Data
<-- LOCATE REQUEST

Overview

Command REQUEST_ACK DATA CHAIN REQUEST_FIN
READ (AND SEEK) X X X
WRITE (AND SEEK) X X X
SEEK X - -
LOCATE X - -
STATUS X - -
CLOSE X - -

Generic Building Blocks

Mover HELLO Block

Value Length (Bytes)
Session ID 4
Number of bytes following for the challenge 4
Challenge Command dependent

Generic REQUEST Block

Value Length (Bytes)
Number of Bytes following 4
Command code 4
Arguments Command dependent

Generic ACK/FIN Block

Success

Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK=6 / REQUEST_FIN=7 4
Corresponding IOCMD code 4
Return code (success==0) 4
[arg1] length(arg1)
[...] length(...)
Return arguments Command dependent

Failure

Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK=6 / REQUEST_FIN=7 4
Corresponding IOCMD code 4
Return code (failure!=0) 4
UTF-8 UNICODE string length is defined by UTF-8

Generic DATA Chain

Requests in Detail

The Write (SeekAndWrite) Request

The client initiates the transaction by sending the Write Request Block, WRB. The WRB doesn't contain any information about the number of bytes, the client intends to write, in order to allow streaming from an unknown data source. In the case of SeekAndWrite the Request block additionally contains the seek offset and the seek 'whence' code. ****

Write Seek And Write

|

Value Length (Bytes)
4 4
WRITE=1 4
Value Length (Bytes)
16 4
SEEK_AND_WRITE=12 4
offset 8
whence 4
Whence Meaning Code
SEEK_SET From Beginning of File 0
SEEK_CURRENT From Current File Position 1
SEEK_END From End Of File 2

The Server returns the Write Request Ack, WRA indicating whether or not the Write Request is granted.

The server agrees to take any amount of data coming from the client.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK = 6 4
IOCMD code WRITE (1), SEEK_AND_WRITE (12) 4
success= 0 4

The server limits the maximum number of bytes sent by the client.

Value Length (Bytes)
Number of Bytes following = 24 4
REQUEST_ACK (6) 4
IOCMD code WRITE (1), SEEK_AND_WRITE (12) 4
success= 0 4
DONT_SEND_MORE 4
8

The server asks the client to expect a new data connection from a different server. The client has to issue this command again after the new server has been successfully connected. The client must not send a DCAP CLOSE command on the current connection but should assume that the server is going to close this connection rather soon.

Value Length (Bytes)
Number of Bytes following = 16 4
REQUEST_ACK = 6 4
IOCMD code WRITE (1), SEEK_AND_WRITE (12) 4
success= 0 4
EXPECT_NEW_CONNECTION 4
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (WRITE=1,SEEK_AND_WRITE=12) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

In case, the request is denied, the Write Request Sequence has ended. If the server replies with success, the client is requested to send the Data Chain.

Value Length (Bytes)
Number of Bytes following = 4 4
DATA = 8 4
Number of Bytes following = 4
Number of Bytes following = 4
... ...
Number of Bytes following = -1 4

In case, the server was able to store all the data sent by the client it (the server) sends the Write Request Fin packet.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_FIN = 7 4
IOCMD code (WRITE = 1) 4
success= 0 4

otherwise the server reports on an error.

Value Length (Bytes)
Number of Bytes following 4
REQUEST_FIN = 7 4
IOCMD code (WRITE = 1) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Read (SeekAndRead) Request

The client initiales the transaction by sending the Read Request Block, RRB. The RRB contains the number of bytes the client want to read from the connected dataset. In the case of SeekAndRead the Request block additionally contains the seek offset and the seek 'whence' code. ****

|

Value Length (Bytes)
12 4
READ=2 4
readBlockLength 8
Value Length (Bytes)
24 4
SEEK_AND_READ=11 4
offset 8
whence 4
readBlockLength 8
Whence Meaning Code
SEEK_SET From Beginning of File 0
SEEK_CURRENT From Current File Position 1
SEEK_END From End Of File 2

The server returns the Read Request Ack, indicating whether or not the Read Request is granted.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK (6) 4
IOCMD code READ (2), SEEK_AND_READ (11) 4
success= 0 4

The server asks the client to expect a new data connection from a different server. The client has to issue this command again after the new server has been successfully connected. The client must not send a DCAP CLOSE command on the current connection but should assume that the server is going to close this connection rather soon.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK (6) 4
IOCMD code READ (2), SEEK_AND_READ (11) 4
success= 0 4
EXPECT_NEW_CONNECTION 4
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (READ=2,SEEK_AND_READ=11) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

In case, the request is denied, the Read Request Sequence has ended. If the server replies with success, it will proceed by sending the Data Chain. The server will send exactly the number of bytes requested except :

  • EOF is encountered.
  • An interrupt was sent asynchronously.
  • Any kind of server error was detected.
Value Length (Bytes)
Number of Bytes following = 4 4
DATA = 8 4
Number of Bytes following = 4
Number of Bytes following = 4
... ...
Number of Bytes following = -1 4

If no error was detected while sending the data to the client, the server sends the Read Request Fin Ok.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_FIN = 7 4
IOCMD code (READ = 2) 4
success= 0 4

otherwise the server reports an error.

Value Length (Bytes)
Number of Bytes following 4
REQUEST_FIN = 7 4
IOCMD code (READ = 2) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Vector Read Request

The client initiales the transaction by sending the ReadV Request Block, RVRB. The RVRB contains a sequence of number of bytes the client want to read from the connected dataset and file position.

Value Length (Bytes)
8+nBlocks*12 4
READV=13 4
nBlocks 4
Block1 Offset 8
Block1 length 4
..... .....
BlockN Offset 8
BlockN length 4

The server returns the Read Request Ack, indicating whether or not the Read Request is granted. The date returned as a continues sequence of bytes.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK (6) 4
IOCMD code READV (13) 4
success= 0 4

The server asks the client to expect a new data connection from a different server. The client has to issue this command again after the new server has been successfully connected. The client must not send a DCAP CLOSE command on the current connection but should assume that the server is going to close this connection rather soon.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK (6) 4
IOCMD code READV (13) 4
success= 0 4
EXPECT_NEW_CONNECTION 4
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (READ=2,SEEK_AND_READ=11) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

In case, the request is denied, the Read Request Sequence has ended. If the server replies with success, it will proceed by sending the Data Chain. The server will send exactly the number of bytes requested or error condition is triggered:

  • EOF was reached
  • An interrupt was sent asynchronously.
  • Any kind of server error was detected.
Value Length (Bytes)
Number of Bytes following = 4 4
DATA = 8 4
Number of Bytes following = 4
Number of Bytes following = 4
... ...
Number of Bytes following = -1 4

If no error was detected while sending the data to the client, the server sends the Read Request Fin Ok.

Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_FIN = 7 4
IOCMD code (READ = 2) 4
success= 0 4

otherwise the server reports an error.

Value Length (Bytes)
Number of Bytes following 4
REQUEST_FIN = 7 4
IOCMD code (READ = 2) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Seek Request

Value Length (Bytes)
16 4
SEEK=3 4
(offset) 8
(whence) 4
Whence Meaning Code
SEEK_SET From Beginning of File 0
SEEK_CURRENT From Current File Position 1
SEEK_END From End Of File 2
Value Length (Bytes)
Number of Bytes following = 20 4
REQUEST_ACK = 6 4
IOCMD code (SEEK = 3) 4
success= 0 4
8
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (SEEK = 3) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Locate Request

Value Length (Bytes)
4 4
LOCATE=9 4
Value Length (Bytes)
Number of Bytes following = 28 4
REQUEST_ACK = 6 4
IOCMD code (LOCATE = 9) 4
success= 0 4
8
8
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (LOCATE = 9) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Status Request

Value Length (Bytes)
Number of Bytes following = 4 4
STATUS = 10 4
Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK = 6 4
IOCMD code (STATUS = 10) 4
success= 0 4
protection (mode bits) 4
number of links 4
user id of owner 4
group id of owner 4
file size 8
time of last access 8
time of last modification 8
time of creation 8
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (STATUS = 10) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Close Request

Value Length (Bytes)
4 4
CLOSE = 4 4
Value Length (Bytes)
data following = 20 4
CLOSE = 4 4
data following = 12 4
DCAP_DATA_SUM = 1 4
checksum type = DCAP_DEFAULT_SUM = ADLER32 = 1 4
checksum 4
Value Length (Bytes)
Number of Bytes following = 12 4
REQUEST_ACK = 6 4
IOCMD code (CLOSE = 4) 4
success= 0 4
Value Length (Bytes)
Number of Bytes following 4
REQUEST_ACK = 6 4
IOCMD code (CLOSE = 4) 4
failure=<ErrorNumber!=0> 4
Detail : UTF-8 UNICODE length is defined by UTF-8

The Interrupt Request

The Interrupt Request is sent by the data receiving party. If the Data Chain is still in process, the sending party tries to stop the sending by transfering the current subblock and then ending the transfer by sending the '-1' data chain end block. In all other situations the Interrupt is simply ignored. ****

Value Length (Bytes)
8 4
INTERRUPT=5 4
Interrupt Reason code 4

Neither a Request Ack nor a Request Fin is send in response to an Interrupt Request.

Constants

Keyword Value
WRITE 1
READ 2
SEEK 3
CLOSE 4
INTERRUPT 5
REQUEST_ACK 6
REQUEST_FIN 7
DATA 8
LOCATE 9
STATUS 10
SEEK_AND_READ 11
SEEK_AND_WRITE 12
SEEK_SET 0
SEEK_CURRENT 1
SEEK_END 2
DONT_SEND_MORE 20
EXPECT_NEW_CONNECTION 21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment