Disclaimer: This document includes Layer features that are in progress or are planned in the future. This should be not interpreted as a commitment that these features will be built, or will be built as described. The purpose of this document is to share our plans with select customers to aid them in planning, and for us to receive their feedback.
Layer takes the position that our customers' data belongs to them, and that they have full access to it. However, we do not just give our customers access to their raw data, but also provide a number of data services that process and organize the data on our customers' behalf so as to minimize or eliminate the work they have to do before they receive value from the data. To the extent that these services require additional design, development, processing and storage on our part, some of them may be offered as paid services or as part of higher tiers of service. This document describes data services that are currently in development or are being planned.
In general, our customers require access to their data for operational or analytical reasons. Operational requirements include debugging or ensuring adequate performance, and may require data like logs or operational metrics. Analytical requirements are much broader, and require data that gives our customers better insights into how their users are using their application. This may help them improve or add features to their application, target ads, or otherwise improve the adoption, engagement and monetization of their users. Also, analytical data from Layer will often be combined with data from other sources (e.g. from their own app or from third party databases) before analysis, so it is helpful if we deliver the data in a form that be be easily combined.
There is of course overlap between these two requirements, and some data (e.g. the number of messages sent in the last hour) may be useful for both operational and analytical purposes.
Another distinction in the type of data is whether it is about events that occur in the system, or about the state of the system. Events are discrete and occur at a point in time, whereas the state of the system is a result of all the past events. Event data is generally obtained from an event stream, while state data is genarally obtained by querying aggregated state. Both types or data are useful for operational and analytical purposes; they just need different infrastructure.
Layer provides several services that can be used to obtain operational or analytical event or state data, as described below.
The Layer developer dashboard includes a section for operational logs that list messaging and other events as they flow through the system. These are primarily intended for debugging and observing performance.
More information can be found here: Developer Dashboard
Although the Layer Client API is primarily intended to create Web-based clients, it can also be used to query for state information from the perspective of a specific user. For example, one can use the Client API to obtain a list of conversations for a specific user, and for the messages in those conversations.
More information can be found here: Client API Specification
While the Client API presents a view from the perspective of a specific user, the Layer Platform API takes the perspective of the application as a whole. While the scope of this API is currently limited to creating conversations, and sending messages or announcements into Layer, it will be extended over time to support querying the state from the perspective of an application. Thus it will be possible, for example, to query for a list of conversations in an application and for the messages in those conversations.
More information can be found here: Platform API Docmentation
While the Client and Platform APIs allow the state to be queried, Insights focuses on capturing events, computing metrics based on them, and pre-aggregating these metrics along a specific set of dimensions. It then provides an API for time-based queries on these metrics. This API will also be used to create a set of pre-designed dashboards that will become part of the Layer portal. The Insights API can be used to query, for example, for the number of message read events over the last several hours, grouped by hour and by user.
More information can be found here: Insights API Specification
Webhooks allow our customers to receive data about events as they occur by having Layer invoke an endpoint on their servers when an event occurs. The Webhooks API allows a customer to specify the type of events they are interested in and to provide a URL to be invoked when events occur. This is most useful for situations where an application backend wants to respond to events as they occur in Layer.
More information can be found here: Webhooks API Specification
Data Export is similar to Webhooks in that it gives our customers data about events that occur in Layer, but instead of requiring them to set up a server to receive event data, it delivers the data periodically in batches. This is particularly useful if the data is intended for use in analytics, since it can be imported into the analytical database where the analysis is done, often in combination with other data.
The default export format is a JSON file containing information for all events in a given period (e.g. a day). We are also investigating a tabular format (e.g. a SQLite database or CSV files), which may be simpler to import into external databases like Amazon Redshift or Google BigQuery. Since Google BigQuery supports data being streamed into it, we are also exploring the possibility of streaming event data directly into a set of Google BigQuery tables as the events occur.
More information can be found here: Data Export Format