Example Web REST API with loop through record sets

README.md

Quite often, web REST API's return JSON data but are limited to the number of records they return.

Good API's also return the total number of records that could be returned along with the current offset (from start), the record limit and perhaps even a URL that lets you make the next call in the series.

Mailroute is a cloud service that pre-checks email for you, filtering out spam and viruses and letting you white-/black-list sending domains and email addresses.

It has a pretty good API that returns 20 records at a time. So, if you want to return a large number (all) of the white-/black-list entries, you need a flow that will loop through each recordset.

The attached flow does just that. It also makes use of links to keep the looping flow neat. Also uses JSONata in Change nodes instead of function nodes so reducing the level of JavaScript coding knowledge required.

Embedded in the flow is a comment node with instructions on how to set some required in-memory variables that I've used so that I don't need to share my secrets with you.

flow.json

[
    {
        "id": "2f95e587.302f6a",
        "type": "change",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "rules": [
            {
                "t": "delete",
                "p": "headers",
                "pt": "msg"
            },
            {
                "t": "set",
                "p": "headers",
                "pt": "msg",
                "to": "{\t    \"Accept\": \"application/json\",\t    \"Content-Type\": \"application/json\",\t    \"Authorization\": $flowContext('mailroute-api-key')\t}",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 420,
        "y": 4200,
        "wires": [
            [
                "e403fab4.7cd128"
            ]
        ]
    },
    {
        "id": "3ddbbae4.830b56",
        "type": "inject",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "repeat": "",
        "crontab": "",
        "once": false,
        "x": 140,
        "y": 4160,
        "wires": [
            [
                "e7385cba.f3325"
            ]
        ]
    },
    {
        "id": "e403fab4.7cd128",
        "type": "http request",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "method": "GET",
        "ret": "txt",
        "url": "",
        "tls": "",
        "x": 570,
        "y": 4180,
        "wires": [
            [
                "7ec73a54.23a274"
            ]
        ]
    },
    {
        "id": "7ec73a54.23a274",
        "type": "json",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "pretty": false,
        "x": 649.0000343322754,
        "y": 4155.000413894653,
        "wires": [
            [
                "db8b73b7.5f5f4"
            ]
        ]
    },
    {
        "id": "788ab243.258bbc",
        "type": "switch",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "property": "payload.meta.next",
        "propertyType": "msg",
        "rules": [
            {
                "t": "null"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "false",
        "outputs": 2,
        "x": 910,
        "y": 4180,
        "wires": [
            [
                "9d358f5c.6e362"
            ],
            [
                "dbc80fbf.2d03b"
            ]
        ]
    },
    {
        "id": "dccc8e77.6f654",
        "type": "debug",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "active": true,
        "console": "false",
        "complete": "false",
        "x": 1250,
        "y": 4100,
        "wires": []
    },
    {
        "id": "dbc80fbf.2d03b",
        "type": "change",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "url",
                "pt": "msg",
                "to": "$replace(url, /(limit=)(.+)(&offset=)(.+)/, \"$1\" & $string(payload.meta.limit) & \"$3\" & $string(payload.meta.offset + payload.meta.limit))",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 1050,
        "y": 4200,
        "wires": [
            [
                "6faae389.342f5c"
            ]
        ]
    },
    {
        "id": "e7385cba.f3325",
        "type": "change",
        "z": "6a32c4d1.8c37e4",
        "name": "Set url, reset wblist",
        "rules": [
            {
                "t": "set",
                "p": "url",
                "pt": "msg",
                "to": "\"https://admin.mailroute.net/api/v1/\" & $flowContext('mailroute-api-path') & \"?limit=20&offset=0\"",
                "tot": "jsonata"
            },
            {
                "t": "delete",
                "p": "wbrules",
                "pt": "flow"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 250,
        "y": 4180,
        "wires": [
            [
                "2f95e587.302f6a"
            ]
        ]
    },
    {
        "id": "6faae389.342f5c",
        "type": "switch",
        "z": "6a32c4d1.8c37e4",
        "name": "Offset > total?",
        "property": "payload.meta.offset",
        "propertyType": "msg",
        "rules": [
            {
                "t": "lte",
                "v": "payload.meta.total_count",
                "vt": "msg"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "false",
        "outputs": 2,
        "x": 1280,
        "y": 4240,
        "wires": [
            [
                "436c23a4.e12ffc"
            ],
            []
        ]
    },
    {
        "id": "db8b73b7.5f5f4",
        "type": "function",
        "z": "6a32c4d1.8c37e4",
        "name": "Merge",
        "func": "var wbrules = flow.get('wbrules') || []\n\nwbrules = wbrules.concat(msg.payload.objects)\n//Array.prototype.push.apply(wbrules, msg.payload.objects)\n\nflow.set('wbrules', wbrules)\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 770,
        "y": 4180,
        "wires": [
            [
                "788ab243.258bbc"
            ]
        ]
    },
    {
        "id": "9d358f5c.6e362",
        "type": "change",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "wbrules",
                "tot": "flow"
            },
            {
                "t": "set",
                "p": "filename",
                "pt": "msg",
                "to": "$globalContext('user-home-folder') & \"/Downloads/wblist.csv\"",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 1060,
        "y": 4160,
        "wires": [
            [
                "dccc8e77.6f654",
                "90630247.fbb5a"
            ]
        ]
    },
    {
        "id": "b4d88961.3abf88",
        "type": "comment",
        "z": "6a32c4d1.8c37e4",
        "name": "Mailroute Query API - Loop through all whitelist entries for an account",
        "info": "Mailroute is a cloud service that pre-filters\nemail traffic. It has a number of settings including\na whitelist that allows you to force allow or \nblock for email from a specific domain or email\naddress.\n\nThis flow is an example of the API and an example\nof looping through an API that returns a limited\nset of records.\n\nThe flow returns an array of objects detailing\nall of the entries in the white-/black-list.\n\nIt also writes to a CSV file.\n\n## Flow Inputs\n\nNOTE that a separate flow is used to set three flow \nvariables containing information that you would not\nwant public. You will need to set these yourself:\n\n- (flow) mailroute-api-key - \n  The API key required for authorisation.\n  Obtain from the Mailroute Settings\n\n  Format is:\n  `ApiKey <apiUser>:<apiKey>`\n  \n- (flow) mailroute-api-path - \n  The sub-path that you want to use within\n  the API end-points.\n  See the [Mailroute API Documentation](https://support.mailroute.net/hc/en-us/sections/205312008-API)\n  for details.\n\n  To get the global whitelist entries:\n  `wblist/`\n  \n  To get the whitelist entries for a specific mail user:\n  `email_account/<userEmailAddress>/wblist/`\n\n- (global) user-home-folder - \n  Set to either the HOME or HOMEPATH environment\n  variable (dependent on command shell running\n  Node-RED)\n\n## Output Data Format\n\nOutput data from the API is JSON. But it is limited\nto 20 records at a time. This flow loops through\nto get all records and returns an array.\n\n- *email*: Domain or email address to block/allow.\n- *wb*: B = Block, A = Allow",
        "x": 310,
        "y": 4120,
        "wires": []
    },
    {
        "id": "d1b158de.0f8958",
        "type": "file",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "filename": "",
        "appendNewline": true,
        "createDir": false,
        "overwriteFile": "true",
        "x": 1310,
        "y": 4140,
        "wires": []
    },
    {
        "id": "90630247.fbb5a",
        "type": "csv",
        "z": "6a32c4d1.8c37e4",
        "name": "",
        "sep": ",",
        "hdrin": "",
        "hdrout": true,
        "multi": "one",
        "ret": "\\n",
        "temp": "domain,email,email_account,id,mail_addr,resource_url,wb",
        "x": 1250,
        "y": 4180,
        "wires": [
            [
                "d1b158de.0f8958"
            ]
        ]
    },
    {
        "id": "436c23a4.e12ffc",
        "type": "link out",
        "z": "6a32c4d1.8c37e4",
        "name": "Loop-Out",
        "links": [
            "8fd170c.c8a909"
        ],
        "x": 1433.079969406128,
        "y": 4278.531560897827,
        "wires": []
    },
    {
        "id": "8fd170c.c8a909",
        "type": "link in",
        "z": "6a32c4d1.8c37e4",
        "name": "Loop-In",
        "links": [
            "436c23a4.e12ffc"
        ],
        "x": 296.076397895813,
        "y": 4228.302665710449,
        "wires": [
            [
                "2f95e587.302f6a"
            ]
        ]
    }
]