In mobile, sync typically refers to the ability to use an application while out of network, e.g. "offline". In order to achieve this, a portion of the back end data is stored locally on the device in non volatile memory, e.g. SQL Lite, which can also, optionally, be encrypted for additional security. The application will read from and write to the local database and this database will be kept "in sync" with the back end data. This means that when the device is in network, the following sync operations can occur:
- Retreive new records from the back end
- Retrieve modified records from the back end
- Send locally modified records to the back end
- Send new locally create records to the back end
- Manage deleted records (locally and from the back end)
- Deal with conflicts (e.g. locally modified record that was also modified by another user)
- Create a strategy for Sync (e.g. every 15 minutes, once a day, etc...)
Note that not all of the above operations are required for all applications. For example, a read-only application may only need to deal with items 1 and 2 above.
The sync operations described above have implications on the back end data source and the API's it exposes. For example, does the back end data source (and it's APIs) provide a means of retreiving newly created or recently modified records, that is, records that were created and/or modified after a certain timestamp.
A general rule of thumb for syncing is that the back end data source should have 'add', 'mod' and 'delete' dates for each record as well as the means to query the data based on these dates. If using a middle tier server, such as Arrow, the connector (or connection to the back end data source) must support the queries required to implement sync.
Also, what is typically desired is that the mobile app developer can be shielded from the nuances of managing the sync operations and have all of this just 'work'.
This example will demonstrate some sync fundamentals and explore ways to move some of the 'sync' work from the client to the server, reducing some of the burden from the mobile app developer.
We will use a data source created in Arrow using the ArrowDB connector. The model, MyAccount, will also contain a create date and a modification data. Then in order to sync the data, that is, retreive new and modified records, the client app will make a GET request with a 'where' query.
The MyAccount model is shown below:
var Arrow = require("arrow");
var Model = Arrow.createModel("MyAccount",{
"fields": {
"name": {
"type": "String",
"required": true
},
"type": {
"type": "String"
},
"createDate": {
"type": "Date",
},
"modDate": {
"type": "Date",
}
},
"connector": "appc.arrowdb",
"before":"setDates",
"actions": [
"create",
"read",
"update",
"delete",
"deleteAll"
],
"singular": "MyAccount",
"plural": "MyAccounts"
});
module.exports = Model;In order to automatically set the create and modified dates, I use a pre-block on the MyAccount model. On a POST (create), I set the createDate and on a PUT (modify), I set the modDate as follows:
var Arrow = require('arrow');
var PreBlock = Arrow.Block.extend({
name: 'setDates',
description: 'set createDate and modDate to support sync operations',
action: function (req, resp, next) {
if(req.method==="POST") {
var now = new Date();
req.params.createDate = now;
} else if(req.method==="PUT") {
var now = new Date();
req.params.modDate = now;
}
next();
}
});
module.exports = PreBlock;Consider the following data set:
{
"success": true,
"request-id": "2a543d38-a782-4798-93e6-4a345595d312",
"key": "myaccounts",
"myaccounts": [
{
"id": "55eb3375647b3f0bc80aa413",
"name": "Account 04",
"type": "Propsect",
"createDate": "2015-09-05T18:24:54.736Z"
},
{
"id": "55eb3362421c440bc30b1330",
"name": "Account 03",
"type": "Propsect",
"createDate": "2015-09-05T18:24:36.336Z"
},
{
"id": "55eb334f647b3f0bc00a841f",
"name": "Account 02",
"type": "Propsect",
"createDate": "2015-09-05T18:24:17.389Z"
},
{
"id": "55eb333a421c440bc30b12c1",
"name": "Account 01",
"type": "Client",
"createDate": "2015-09-05T18:23:57.003Z",
"modDate": "2015-09-05T18:25:20.233Z"
}
]
}The accounts above were created in the following order: Account 01, Account 02, Account 03, and then Account 04. Then Account 01 was modified.
Assuming that the client application already synced Account 01 - Account 03, then the following query would get all accounts created since the last sync:
where={"createDate":{"$gt":"2015-09-05T18:24:36.336Z"}}or the equivalent curl command:
curl "http://127.0.0.1:8080/api/myaccount/query?where=%7B%22createDate%22%3A%7B%22%24gt%22%3A%222015-09-05T17%3A16%3A37.223Z%22%7D%7D"which returns the following:
{
"success": true,
"request-id": "3d07b951-00bc-4d81-b2ad-86a647a58de8",
"key": "myaccounts",
"myaccounts": [
{
"id": "55eb3375647b3f0bc80aa413",
"name": "Account 04",
"type": "Propsect",
"createDate": "2015-09-05T18:24:54.736Z"
}
]
}The following query would get all accounts modified since the last sync:
where={"modDate":{"$gt":"2015-09-05T18:24:36.336Z"}}or the equivalent curl command:
curl "http://127.0.0.1:8080/api/myaccount/query?where=%7B%22modDate%22%3A%7B%22%24gt%22%3A%222015-09-05T18%3A24%3A36.336Z%22%7D%7D"which returns the following:
{
"success": true,
"request-id": "1bf9cb09-0aad-4dd2-8bd3-a38472744cb5",
"key": "myaccounts",
"myaccounts": [
{
"id": "55eb333a421c440bc30b12c1",
"name": "Account 01",
"type": "Client",
"createDate": "2015-09-05T18:23:57.003Z",
"modDate": "2015-09-05T18:25:20.233Z"
}
]
}This enables the following basic sync scheme on the client:
-
When app starts for first time, query the data with a low createDate query (e.g. {"createDate":{"$gte":"0"}}) to get all the records
-
Save the time stamp for when the last query above was made, this is the lastSyncDate
-
Whenever you want to get the "deltas" (i.e. the new records that were created since the last query) then make a query with the last time stamp ({"createDate":{"$gt":lastSyncDate}})
-
Perform similar query with the modDate to get the records that have been modififed
-
On success, save the lastSyncDate
Note that the client app is responsible for storing and updating this data in SQL Lite so that the data is available offline.
Using Titanium and performing client side lastSyncDate management, one could manage the lastSyncDate and perform queries as follows:
url = baseURL+"/api/myaccount/query?where="+encodeURIComponent('{"createDate":{"$gt":"'+lastSyncDate+'"}}');
url = baseURL+"/api/myaccount/query?where="+encodeURIComponent('{"modDate":{"$gt":"'+lastSyncDate+'"}}');To ease client side development, it would be nice to remove the lastSyncDate management and let Arrow manage that. One way to do that is the have the client pass a username (or other unique identifier) with the API request. Then Arrow can check the username against a sync user database and store and manage the lastSyncDate for each user.
Arrow can then create the necessary queries based on the lastSyncDate and then update the lastSyncDate upon successful transfer of data.
The following custom Arrow API, myaccountsync, is an example of how to perform these operations against the MyAccount model:
var Arrow = require('arrow');
var myaccountsync = Arrow.API.extend({
group: 'myaccountsync',
path: '/api/myaccountsync',
method: 'GET',
description: 'get new and modified myaccount records',
parameters: {
getAll: {
description:'optional flag to retrieve all data',
optional: true
}
},
action: function (req, resp, next) {
if(!req.headers['user-name']){ //make sure user-name header is not null
resp.response.status(500);
resp.send({"error": "pass username in the header, user-name"});
next(false);
} else {
var sysusermodel = Arrow.getModel("sysuser");
sysusermodel.query({username: req.headers['user-name']}, function(err, data){
if(err) {
resp.response.status(500);
resp.send({"error": "cannot access user sync database"});
next(false);
} else {
if(data.length <= 0) {
resp.response.status(500);
resp.send({"error": "user not found"});
next(false);
} else { //retreived user sync record
var replyObject = {
lsd: "",
data: {
new: [],
mod: []
}
};
if(data[0].lastSyncDate && !req.params.getAll){
var myaccountmodel = Arrow.getModel("MyAccount");
myaccountmodel.query({"createDate": {"$gt":data[0].lastSyncDate}}, function(er, dat){ // get new records
if(er){
console.log('error getting MyAccount database, er = '+er);
resp.response.status(500);
resp.send({"error": "cannot access MyAccount database"});
next(false);
} else {
replyObject.data.new = dat;
myaccountmodel.query({"modDate": {"$gt":data[0].lastSyncDate}}, function(e, d){ // get modified records
if(e){
console.log('error getting MyAccount database, e = '+e);
resp.response.status(500);
resp.send({"error": "cannot access MyAccount database"});
next(false);
} else {
data[0].lastSyncDate = new Date();//set sync date
data[0].update();
replyObject.data.mod = d;
replyObject.lsd = data[0].lastSyncDate;
resp.send(replyObject);
next();
}
});
}
});
} else { //lastSync undefined
var myaccountmodel = Arrow.getModel("MyAccount");
myaccountmodel.findAll(function(er, dat){ // get all records and place in "new" object array
if(er){
console.log('error getting MyAccount database, er = '+er);
resp.response.status(500);
resp.send({"error": "cannot access MyAccount database"});
next(false);
} else {
data[0].lastSyncDate = new Date();//set sync date
data[0].update();
replyObject.lsd = data[0].lastSyncDate;
replyObject.data.new = dat;
resp.send(replyObject);
next();
}
});
}
}
}
});
}
}
});
module.exports = myaccountsync;If you want to allow the user to modify records on the device, while offline, and sync them to the back end, then you may want to add a field to the SQL Lite database, e.g. isModified, that indicates that a record has been modified. During the sync you can then determine what local records were modifed and then perform a PUT to the back end to update the record. On each successful reply, you can update the field to indicate that the record is not modified (e.g. the modifications have been synced).
This is depicted
If you want to allow the user to create new records in the app, while offline, and sync them to the back end, then one needs to consider how the primary key (or id) is managed. For example, when the new record is created in the local SQL Lite database, you will not have a back end primary key (id) for it and also may want to add a field to the SQL Lite database, e.g. isNew, to indicate that it is a locally created record. Also a temporary unique Primary Key should be stored in the primary key field until the back end primary key is retrieved.
This is depicted
Then when you perform the sync you can search for newly created local records and POST each one to the server. The server will respond to the POST with a reply that contains the id of the record.
In ArrowDB a POST will respond with a header that contains the full path, in the form:
Location: /api/myaccount/55f08b1d595cd60bcd1b08cdYour client app can parse this reply header to get the id and store this in the local record to update the primary key.
In Titanium, the code may look like this:
.
.
.
var xhr = Ti.Network.createHTTPClient({
onload: function(e) {
var elements = xhr.getResponseHeader('Location').split('/');
var id = elements[elements.length-1];
...
},
onerror: function(e) {
Ti.API.info(e.error);
...
},
timeout:5000
});
.
.
.