Skip to content

Instantly share code, notes, and snippets.

@vtrehan
Last active November 4, 2015 22:45
Show Gist options
  • Save vtrehan/b543b6639ca0fbf8f7fc to your computer and use it in GitHub Desktop.
Save vtrehan/b543b6639ca0fbf8f7fc to your computer and use it in GitHub Desktop.

Layer Data Export

Our users are exposed to our conversation and message data model, we need to enable them to be able to look at their messaging data and perform their own analysis on it to improve their product.

As part of the data export service, we will write certain events to a file, compress and encrypt it. These files will be uploaded to Google Cloud Services bucket and a link will be sent to the customers using a webhook notification at the end of the day. The payloads will be encrypted using RSA 2048 asymmetric cryptography. The clients will have access to the public key (via the developer portal) to decrypt the payloads after receiving the link. The files will be compressed using tar gzip.

##Events

We can collect daily dumps of Events from Layer that can be stored and sliced/diced based on your needs.

Each event type corresponds to a specific action that can occur within your Layer application. Customers will have to option to pick the events they are interested in via the developer portal. The current set of available event types are:

Event Description
message.sent When a Message is sent.
message.delivered When a client acknowledges delivery of a Message.
message.read When a client marks a Message as read.
message.deleted When a client deletes a Message (Global deletion mode only).
conversation.created When a new Conversation is created.
conversation.updated.participants When a Conversation is updated participant changes.
conversation.updated.metadata When a Conversation is updated for metadata changes.
conversation.deleted When a Conversation is deleted (Global deletion mode only).

message.sent

Written when a new Message is sent.

{
	"event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"message.delivered",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"1234"
		}
	},
	"message": {
	    "id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67",
	    "conversation": {
	    	"id": "layer:///conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
	    	"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f"
	    },
	    "parts": [
	        {
	        	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/0",
	            "mime_type": "text/plain",
	            "body": "This is the message.",
	            "size": 20
	        },
	        {
	            "mime_type": "image/png",
	            "content": {
	           	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/1",
	                "download_url": "http://google-testbucket.storage.googleapis.com/some/download/path",
	                "expiration": "2014-09-09T04:44:47+00:00",
	                "refresh_url": "https://api.layer.com/content/7a0aefb8-3c97-11e4-baad-164230d1df67",
	                "size": 172114124
	            },
	            "size": 172114124
	        }
	    ],
	    "sent_at": "2014-09-09T04:44:47+00:00",
	    "sender": {
			"id": "12345"
	    },
	    "recipient_status": {
	        "12345": "read",
	        "999": "sent",
	        "111": "sent"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

message.delivered

Written when a Message recipient acknowledges delivery of a Message.

{	
	"event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"message.delivered",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"111"
		}
	},
	"message": {
	    "id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67",
	    "conversation": {
	    	"id": "layer:///conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
			"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f"
	    },
	    "parts": [
	        {
	        	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/0",
	            "mime_type": "text/plain",
	            "body": "This is the message.",
	            "size": 20
	        },
	        {
	            "mime_type": "image/png",
	            "content": {
	            	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/1",
	                "download_url": "http://google-testbucket.storage.googleapis.com/some/download/path",
	                "expiration": "2014-09-09T04:44:47+00:00",
	                "refresh_url": "https://api.layer.com/content/7a0aefb8-3c97-11e4-baad-164230d1df67",
	                "size": 172114124
	            },
	            "size": 172114124
	        }
	    ],
	    "sent_at": "2014-09-09T04:44:47+00:00",
	    "sender": {
			"name": "t-bone"
	    },
	    "recipient_status": {
	        "777": "sent",
	        "12345": "read",
	        "111": "delivered"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

message.read

Written when a Message recipient marks a Message as read.

{
   "event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"message.read",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"999"
		}
	},
	"message": {
	    "id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67",
	    "conversation": {
	    	"id": "layer:///conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
			"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f"
	    },
	    "parts": [
	        {
	        	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/0",
	            "mime_type": "text/plain",
	            "body": "This is the message.",
	            "size": 20
	        },
	        {
            	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/1",
	            "mime_type": "image/png",
	            "content": {
	            	"id": "layer:///content/3d0736d9-1a50-4e9a-a9b3-2400caa9e161",
	                "download_url": "http://google-testbucket.storage.googleapis.com/some/download/path",
	                "expiration": "2014-09-09T04:44:47+00:00",
	                "size": 172114124
	            },
	        }
	    ],
	    "sent_at": "2014-09-09T04:44:47+00:00",
	    "sender": {
			"id": "12345"
	    },
	    "recipient_status": {
	        "12345": "read",
	        "999": "read",
	        "111": "delivered"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

message.deleted

Written when a Message is globally deleted.

{
   "event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"message.deleted",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"999"
		}
	},
	"message": {
	    "id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67",
	    "conversation": {
	    	"id": "layer:///conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
			"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f"
	    },
	    "parts": [
	        {
	        	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/0",
	            "mime_type": "text/plain",
	            "body": "This is the message."
	        },
	        {
            	"id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/1",
	            "mime_type": "image/png",
	            "content": {
	            	"id": "layer:///content/3d0736d9-1a50-4e9a-a9b3-2400caa9e161",
	                "download_url": "http://google-testbucket.storage.googleapis.com/some/download/path",
	                "expiration": "2014-09-09T04:44:47+00:00",
	                "size": 172114124
	            }
	        },
			{
		    	"mime_type": "image/jpeg",
			    "body": "iVBORw0KGgoAAAANSUhEUgAAACA=",
			    "encoding": "base64",
			    "id": "layer:///messages/940de862-3c96-11e4-baad-164230d1df67/parts/2",
			}
	    ],
	    "sent_at": "2014-09-09T04:44:47+00:00",
	    "sender": {
			"id": "12345"
	    },
	    "recipient_status": {
	        "12345": "read",
	        "999": "read",
	        "111": "delivered"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

conversation.created

Written when a new Conversation is created.

{   
	"event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"message.deleted",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"1234"
		}
	},
    "conversation": {
		"id": "layer:///conversations/f3cc7b32-3c92-11e4-baad-164230d1df67",
		"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
	    "created_at": "2014-09-15T04:44:47+00:00",    
		"messages_url": "https://api.layer.com/conversations/c12fd916-1390-464b-850f-1380a051f7c8/messages",
	    "distinct": false,
	    "participants": [
	        "1234",
	        "5678"
	    ],
	    "metadata": {
	    	"favorite": "true",
	    	"background_color": "#3c3c3c"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

conversation.updated.participants

Written when a Conversation is updated through mutation of the participants list.

{
	"event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"conversation.updated.participants",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"1234"
		}
	},
	"conversation": {
	    "id": "layer:///conversations/f3cc7b32-3c92-11e4-baad-164230d1df67",
		"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
	    "created_at": "2014-09-15T04:44:47+00:00",    
		"messages_url": "https://api.layer.com/conversations/c12fd916-1390-464b-850f-1380a051f7c8/messages",
	    "distinct": false,
	    "participants": [
	        "1234",
	        "5678"
	    ],
	    "metadata": {
	    	"favorite": "true",
	    	"background_color": "#3c3c3c"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

conversation.updated.metadata

Written when a Conversation is updated through mutation of metadata.

{
	"event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"conversation.updated.metadata",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"1234"
		}
	},
	"conversation": {
	    "id": "layer:///conversations/f3cc7b32-3c92-11e4-baad-164230d1df67",
		 "url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
	    "created_at": "2014-09-15T04:44:47+00:00",    
		 "messages_url": "https://api.layer.com/conversations/c12fd916-1390-464b-850f-1380a051f7c8/messages",
	    "distinct": false,
	    "participants": [
	        "1234",
	        "5678"
	    ],
	    "metadata": {
	    	"favorite": "true",
	    	"background_color": "#3c3c3c"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}
}

conversation.deleted

Written when a Conversation is globally deleted.

{	
	"event" : {
		"timestamp":"2015-09-17T20:46:47.561Z",
		"type":"conversation.deleted",
		"id":"c12f340d-3b62-4cf1-9b93-ef4d754cfe69",
		"actor": {
			"id":"1234"
		}
	},
	"conversation": {
	    "id": "layer:///conversations/f3cc7b32-3c92-11e4-baad-164230d1df67",
		"url": "https://api.layer.com/apps/082d4684-0992-11e5-a6c0-1697f925ec7b/conversations/e67b5da2-95ca-40c4-bfc5-a2a8baaeb50f",
	    "created_at": "2014-09-15T04:44:47+00:00",    
		"messages_url": "https://api.layer.com/conversations/c12fd916-1390-464b-850f-1380a051f7c8/messages",
	    "distinct": false,
	    "participants": [
	        "1234",
	        "5678"
	    ],
	    "metadata": {
	    	"favorite": "true",
	    	"background_color": "#3c3c3c"
	    }
	},
	"target_config" : {
		"key1" : "value1",
		"key2" : "value2"
	}	
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment