sms message status.md

Intro

This document expands on some of the technical details of SMSSync Issue 11 and proposes a solution for communicating results of outgoing SMS messages processsed by SMSSync.

SMSSync has a basic HTTP client in MainHttpClient.java it should suffice for the purposes of this feature.

Prerequisites

Issue 120 should be completed first, see https://gist.github.com/mandric/11287748.

Settings

A new setting should be added, a boolean for now like "Enable Message Results API" unless someone can think of better name. This allows support for the HTTP API to be optional, otherwise the sms send logic and server requirments remain backwards compatible.

HTTP API

GET ?task=send

To get a list of messages to send SMSSync will make a request using GET and passing the parameter task=send to the Sync URL. The server should respond with the following changes to the current payload JSON structure:

• message_uuid a UUID generated by the server for outgoing messages, additional property in the messages objects.

If this property is not present in a message then we can default to current behavior; results are not communicated to the server. A UUID is required for us to sync message result data.

Response Handling

200

When boolean setting is true, parse response and loop through the list of messages making a POST queued_messages request and then saving and sending the message once the server acknwledges it.

Not 200

SMSSync would ignore the response and let the task checking continue on next interval.

Request Examples

GET /api/smssync?task=send HTTP/1.1
Host: testserver.local

HTTP/1.1 200 OK
Server: nginx/1.5.2
Content-Type: application/json; charset=utf-8

{
    "payload": {
        "task": "send",
        "secret": "",
        "messages": [
            {
                "to": "+254700709142",
                "message": "first message goes here",
                "message_uuid": "aada21b0-0615-4957-bcb3-71fb766aaa9a"
            },
            {
                "to": "+254700709142",
                "message": "second message goes here",
                "message_uuid": "1ba368bd-c467-4374-bf28-2bcf648cfa6d"
            },
            {
                "to": "+254700709142",
                "message": "third message goes here",
                "message_uuid": "95df126b-ee80-4175-a6fb-3d481ef3ef49"
            }
        ]
    }
}

POST queued_messages

For each message or group of messages in the outgoing task list, SMSSync will send an HTTP request to acknowledge messages are being processed, by using the following JSON structure:

• queued_messages array of UUIDs of the messages being sent.

Note: In practice only one item will be in the list becasue we will be calling this once per message while looping over the GET ?task=send response body.

When the the server receives the list of message IDs it would modify some state information in the database so the messages that are being processed by SMSSync do not appear in the outgoing task list anymore. The server might have a process that checks periodically to reprocess messages that have gone stale and never received an update from SMSSync.

This helps to avoid a race condition when two or more SMSSyncs are processing messages on the same server. If another SMSSync has already started processing a specfic ID then it would not be included in the response body.

Response Handling

200

The server response should be 200 on success and include the list of message IDs in the response as a confirmation those messages should be processed.

• message_uuids - array of message UUIDs that are ready to be sent.

The server has acknowledges so now we can call ProcessSms.sendSMS for each message in the list, and save message data including result properites and broadcasting the sent and deliver intents.

Not 200

If 200 status code is not received then SMSSync should stop processing the list of messages and let the task polling continue on the next interval.

Request Examples

Server confirms processing of message queue

POST /api/smssync?task=sent HTTP/1.1
Host: testserver.local
Content-Type: application/json; charset=utf-8

{ 
    "queued_messages": [
        "aada21b0-0615-4957-bcb3-71fb766aaa9a", 
        "1ba368bd-c467-4374-bf28-2bcf648cfa6d",
        "95df126b-ee80-4175-a6fb-3d481ef3ef49"
    ]
}

HTTP/1.1 200 OK
Server: nginx/1.5.2
Content-Type: application/json; charset=utf-8

{
    "message_uuids": [
        "aada21b0-0615-4957-bcb3-71fb766aaa9a", 
        "1ba368bd-c467-4374-bf28-2bcf648cfa6d",
        "95df126b-ee80-4175-a6fb-3d481ef3ef49"
    ]
}

Server denies processing of message queue

POST /api/smssync?task=sent HTTP/1.1
Host: testserver.local
Content-Type: application/json; charset=utf-8

{ 
    "queued_messages": [
        "aada21b0-0615-4957-bcb3-71fb766aaa9a", 
        "1ba368bd-c467-4374-bf28-2bcf648cfa6d",
        "95df126b-ee80-4175-a6fb-3d481ef3ef49"
    ]
}

HTTP/1.1 200 OK
Server: nginx/1.5.2
Content-Type: application/json; charset=utf-8

{
    "message_uuids": []
}

GET ?task=results

When the boolean setting is true, SMSSync should use the task checking frequency to poll this URL, the same way as ?task=send, but with ?task=results parameters the server should return a list of message UUIDs that are waiting to receive result data. Similarly as ?task=send the server will manage the load it passes to SMSSync.

• message_uuids - array of message UUIDs that are ready to be sent.

Response Handling

200

SMSSync should parse response and loop through the list of messages ids and collect result data for all the messages that has been saved locally and construct a request for POST ?task=results message_results API call.

Not 200

SMSSync would ignore the response and let the task checking continue on next interval.

Request Examples

GET /api/smssync?task=results HTTP/1.1
Host: testserver.local

HTTP/1.1 200 OK
Server: nginx/1.5.2
Content-Type: application/json; charset=utf-8

{
    "message_uuids": [
        "aada21b0-0615-4957-bcb3-71fb766aaa9a", 
        "1ba368bd-c467-4374-bf28-2bcf648cfa6d",
        "95df126b-ee80-4175-a6fb-3d481ef3ef49"
    ]
}

POST ?task=results message_results

To send result data to the server, create objects out of the message property names, and append them to an array. For example:

• message_results - the name of the array contains the result objects.
• uuid - the UUID of the message the results correspond to.
• sent_result_code - property value on the message.
• sent_result_message - property value on the message.
• delivery_result_code - property value on the message.
• delivery_result_message - property value on the message.

Response Handling

200

SMSSync is done communicating result data back to server.

Not 200

Then skip and let polling continue.

Request Examples

POST /api/smssync?task=results HTTP/1.1
Host: testserver.local
Content-Type: application/json; charset=utf-8

{ 
    "message_results": [
    	{
        	"uuid": "052bf515-ef6b-f424-c4ee-da7ee61ce81a", 
        	"sent_result_code": 0,
        	"sent_result_message": "SMSSync Message Sent"
        	"delivered_result_code": -1,
        	"delivered_result_message": ""
    	}, 
    	{
        	"uuid": "aada21b0-0615-4957-bcb3-71fb766aaa9a", 
        	"sent_result_code": 0,
        	"sent_result_message": "SMSSync Message Sent",
        	"delivered_result_code": 0,
        	"delivered_result_message": "SMS Delivered"
    	},
    	{
        	"uuid": "1ba368bd-c467-4374-bf28-2bcf648cfa6d",
        	"sent_result_code": 1,
        	"sent_result_message": "Failed to send SMS - Maybe insufficient air time on the phone.",
        	"delivered_result_code": -1,
        	"delivered_result_message": ""
        },
    	{
        	"uuid": "95df126b-ee80-4175-a6fb-3d481ef3ef49",
        	"sent_result_code": 4,
        	"sent_result_message": "No service",
        	"delivered_result_code": -1,
        	"delivered_result_message": ""
    	},
    	...
    ]
}

Concerns

I wonder what considerations we should make for Android 4.4 (API level 19) which has a more sophisticated Telephony API.

To avoid the race condition when two SMSSyncs are connected, we could return an error code on the second POST which would dequeue the message on the SMSSync side. If you did this you would need to fire one POST per message, rather than send a list, to make it clear which one failed.

A message could get lost if the SMSSync consumes it, then goes down permanently.

We could ping SMSSync if we haven't received a result POST after some time, and dequeue the message.

@mandric This is looking good. Personally I would prefer sending a JSON data to the server as opposed to POST params. It's way easier to read the JSON response (especially by humans ) and much easier to deserialize JSON data these days. Afterall if you use SMSSync, you cannot avoid JSON. Either approach is fine by me. If we're going to go with POST params then the ids should be sent as array.

php queued_messages=1&id[]=aada21b0-0615-4957-bcb3-71fb766aaa9a&id[]=1ba368bd-c467-4374-bf28-2bcf648cfa6d&id[]=95df126b-ee80-4175-a6fb-3d481ef3ef49

@garethbowen Or another way to fix the race condition issue is to use SMSSync's unique ID so the server decides which SMSSync install processes which Task. This means, SMSSync has to send it's unique ID when making a request to the server.

We need to figure out a way the server can talk to SMSSync. At the moment, SMSSync does all the talking to the server. The server literally waits for SMSSync to talk to it. The only simpler solution I see is a human manually triggering SMSSync to make a request to the server via SMS

Ok @eyedol, I'm going to change back to all JSON then. Also I think we came up with a solution to avoid the race condition. In the POST queued_messages request the server can return a confirmation in the response using an array of IDs, and an empty array means ignore the queued messages. I'll update the doc so it's more clear.

@eyedol I like the idea of passing in the SMSSync ID too, that sounds like it would be useful but also optional for the server to care about it. This would also solve a problem of when having multiple SMSSyncs connected that the server could choose which gateway to use so conversations could stay on the same phone number, so it's a better user experience when the phone number does not change mid-conversation.

@eyedol another question I have is about settings. I'm thinking that we won't even need extra settings for this. 1) SMSSync can look for message_id values in the GET response, if the server adds them then it's an implicit affirmation to do message result processing. 2) If the server doesn't respond with 200 and a message_ids list from a POST queued_messages request then the result processing is stopped. But it would cause extra requests coming from SMSSync (maybe some unwanted load in some cases), in which case it might warrant a setting to "Enable Message Results Processing" or something.

What extra settings? the Unique id? If we need to add a new Settings, let go for it.

@eyedol the benefit of having a setting is SMSSync will generate less HTTP requests and the server will have less load. But thinking it's minor load that is generated.

I'm also thinking the delivery and sent values should get saved to the messages object as well, and can be displayed in SMSSync somewhere? No sense in throwing that information away if the server doesn't respond?

We should probably move this discussion to ushahidi/SMSSync#11 since we don't get notification.

Late entry, but I just started using the SMSSync library. The lack of status delivery on the backend is a deal breaker which is how I came across this thread. I would like to propose the following:

• SMSSync App sends a GET request to the server same as before.

GET /api/smssync?task=send

• Server responds with list of tasks same as before. If it wants SMS status delivery, it should send a special flag as shown below. Also, note the additional id field that needs to be sent for all messages.

{
  "payload": {
    "task": "send",
    ...
    "status_report": true
    ...
    "messages": [
      {
        "id": "...",
        "to": "",
        "message": ""
      }
    ]
  }
}

• Once all the sms have been sent (or failed), the SMSSync App should send the result and fetch the next set of tasks

POST /api/smssync?task=send

{
  "payload": {
    "failed": [
      {
        "id": "...",
        "error": "...",
        "timestamp": "..."
      }
    ],
    "done": [
      {
        "id": "...",
        "timestamp": "..."
      }
    ]
  }
}

• If the server doesn't respond with any more tasks, the app should revert back to its set polling interval. Else, it should go back to Step 3.

Backwards compatibility:
Since this introduces a breaking change by switching GET to POST, Step 1 above can be changed to send a GET request. If the result_delivery boolean is not found, the app shouldn't switch to a POST request.

If an app is out of date, the server can detect this using the User Agent and not send any messages if report delivery is critical.