Serialization.md

IDL:

• Creates an Avro-like Schema
• Desn't require labeling field location as in Thrift / Protocol Buffers
• Compiler to generate output boilerplate code in multiple languages
• Runtime library for supported languages which boilerplate code will use (targeting C++, Java, Python, JavaScript, TypeScript, C#, Kotlin, Scala)
• Supports storing data in Array of Struct (Row) or Struct of Array format (Column) per record type
• Allows Specifying RPC calls
• Annotations for optional implementation details
• Inheritance

Runtime:

• Efficiently Allocate lightweight objects to access underlying data
• Ideally immutable only interface with efficient updates
• Connecting to an RPC takes: IP/Port, RPC Schema. Calls over the wire contain interface, method, and data. Disconnected clients immediately throw, reconnection should happen via callbacks. Only the minimum necessary amount of Schema information is exchanged. The connection handshake first sends the name and GUID/Hash of the schema, and only sends the schema itself if the server has a different one.
• Pipelining of RPC calls (data or exceptions propagate)
• Underlying data changes, such as when occur from schema mismatches, only occur at write time. The same data objects can exist with many difference schemas. This helps in the common case where data does not need to be translated, as well as the case of creating zerocopy proxy rpcs, which can simply forward calls in a very lightweight way.
• Serialization over HTTP should also be supported

Generate a JSON Schema output from the IDL:

• Namespaces are linked as SHA1 hashes
• Query the service to see if the hashes are present for those namespaces
• For any hashes not present, send the schema. The server saves some number of schemas

Would ideally look something like this:

@name(".cs", "Person")
@arrayFormat(column)
record user {
    @name(".cs", "FirstName")
    first_name: string
    @name(".cs", "LastName")
    last_name: string
    @name(".cs", "DateOfBirth")
    date_of_birth: int64
}

enum gender {
  MALE
  FEMALE
  OTHER
}

record extended_user extends user {
    gender: gender
}

type union_type = void | gender | extended_user;

// All functions return futures/promises/tasks. Omitting the type means the function returns nothing and acts as a fire and forget
// Endpoints with multiple arguments implicitely create records
server user_store {
  get(name: string) => extended_user
  // Implicitely creates record user_store.set { name: string, user: extended_user }
  set(name: string, user: extended_user) => bool
  delete(name: string)
}

server processor {
  special(arg: union_type) => union_type
  another(arg: void | string) => string
}

import { user, extended_user, gender } from './datatypes'
let alice = await userStore.get("Alice").send();
let setBob = await userStore.set("Bob", alice).send();
let charlie = extended_user.from({
  first_name: 'Charlie',
  last_name: 'Bucket',
  date_of_birth: Date.now(),
  gender: gender.MALE,
});

charlie.first_name = 'Charles'; // calls setter function to access underlying data
let last_name = charlie.last_name; // calls getter function to access underlying data

Requests can be pipelined, resulting in a single round trip

let setBob = await userStore
  .get("Alice") // user_store::PromiseBuilder<extended_user>
  .set("Bob")   // user_store::PromiseBuilder<extended_user>.set(string) => user_store::PromiseBuilder<boolean>
                // user_store::PromiseBuilder<string>.set(extended_user) => user_store::PromiseBuilder<boolean>
  .send(); // PromiseBuilder<T>.send() => Promise<T>

Pipelining is only available on a particular RPC server connection, it can't span multiple connections

Arrays

Some languages without operator overloading on array access require using functions instead:

import { List } from 'serializer';
import { user } from './datatypes';

let arr = List.of(user.from({
  first_name: 'Alice',
  last_name: 'Rabbit',
  date_of_birth: Date.now(),
}, {
  first_name: 'Bob',
  last_name: 'Barker',
  date_of_birth: Date.now(),
}, {
  first_name: 'Charlie',
  last_name: 'Bucket',
  date_of_birth: Date.now(),
});

let el3 = arr.get(3);

arr.set(3, user.from({
  first_name: 'Daryl',
  last_name: 'Hannah',
  date_of_birth: Date.now(),
});

// Alternatively
el3.first_name = 'Daryl'
el3.last_name = 'Hannah'
el3.date_of_birth = Date.now()

Related but a bit different:

A request should generally have the following:

userid, sessionid, requestid, messageid. This allows logging and tracing that ties together information for a particular user, a particular session, a particular request, and a particular message (internal API call to another service). This allows for the easy tracking of messages across services for a particular user.

We can add in "time travel" with future arbitrary data arguments:

Regular JS, 3 Round Trips:

api.getUser(userid).then(function(user: User) {
    return api.getUser(user.friends[2]);
}).then(function(user: User) {
    return api.getQuestion(user.questionids[0]);
}).then(function(question: Question) {
    console.log(question);
});

Pipelined Call Style A, 1 Round Trip:

api.pipeline().getUser(userid).getUser(function(user: FutureUser) {
    return user.friends[2]; // This is the argument to the getUser function
}).getQuestion(function(user: FutureUser) => FutureQuestionId {
    return user.questionids[0]; // This is the argument to the getQuestion function
}).send().then(function(question: Question) {
    console.log(question);
});

Pipelined Call Style B, 1 round trip:

api.pipeline(function(api) {
    let user: FutureUser = api.getUser(userid);
    let friends: FutureList<FutureUser> = user.friends.map(api.getUser);
    let goodFriends: FutureList<FutureUser> = friends.filter(api.isGoodFriend(user));
    let badFriend: FutureList<FutureUser> = friends.filter(function(friend: FutureUser) {
        return api.util.not(api.isGoodFriend(user, friend));
    });

    // Just the parts you want to return back to the client
    return api.multiResponse(user, goodFriends, badFriends);
}).then(function(user: User, goodFriends: List<User>, badFriends: List<User>) {
    console.log(user, goodFriends, badFriends);
}, function(userError: UserError, goodFriendsError: ListError<User>, goodFriendsError: ListError<User>) {
    /* Handle Errors */
});

FutureList.map(transform: (FutureItem) => FutureSomething);
FutureList.filter(predicate: (FutureItem) => FutureBool);

Pipelined Call Style C, 1 Round Trip:

api.pipeline().getUser(userid).getUser(function(user: FutureUser) {
    return user.friends[2]; // This is the argument to the getUser function
}).getQuestion(function(user: FutureUser) {
    return user.questionids[0]; // This is the argument to the getQuestion function
}).send().then(function(question: Question) {
    console.log(question);
}, function(error: CombinedError<GetUserError | GetUserError | GetQuestionError>) {
    /* handle the errors */
});

Pipelined Call Style D

api.pipeline().getUser(userid).onError(function(err: GetUserError) {
    /* Handle error on first call */
}).getUser(function(user: FutureUser) => FutureUserId {
    return user.friends[2]; // This is the argument to the getUser function
}).onError(function(err: GetUserError) {
    /* Handle error on second call */
}).getQuestion(function(user: FutureUser) => FutureQuestionId {
    return user.questionids[0]; // This is the argument to the getQuestion function
}).onError(function(err: GetQuestionError) {
    /* Handle error on third call */
}).send().then(function(question: Question) {
    console.log(question);
});

function(userid) {
    graphql`
    fragment on User(id: ${userid} {
        name
        age
        ${store.getFragment('friend')}
    }
`;
}

let getUser = api.buildSchema(
    api.getUser,
    (user) => return { name: user.name });
function(userid) {
    let user = getUser(userid);
    return [user, getFriends(user)];
}