Skip to content

Instantly share code, notes, and snippets.

@coderofsalvation
Last active August 29, 2015 14:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save coderofsalvation/fef94d99990952693d68 to your computer and use it in GitHub Desktop.
Save coderofsalvation/fef94d99990952693d68 to your computer and use it in GitHub Desktop.
filter which drops 'seenbefore' items

You can (re)use this filter (the 'filter new items'-machine) to prevent duplicate dataflows. It installs a (userlimited) cache which is active between requests.

So let say we have this payload:

msg.payload = ["foo","bar"]

and we sent it twice to a 'debug'-machine using two button-presses on a 'inject'-machine

the output of the first mousepress will be:

["foo","bar"]

and the second mousepress:

[]

(because the message was seen before and stored in the cache)

[{"id":"5795e277.a86a1c","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":116,"y":640,"z":"886b17b1.7794e8","wires":[["85b26b47.7a4d98"]]},{"id":"d0d04ab0.2f2fb8","type":"debug","name":"","active":true,"console":"true","complete":"false","x":680,"y":639,"z":"886b17b1.7794e8","wires":[]},{"id":"b0d6e4a1.4f2918","type":"function","name":"filter new items","func":"// this machine will delete items which have been seen already\n// please the following id unique\nvar cacheid = \"foo-bar-cache\";\n\n// prepare cache\nvar enabled = true;\nvar now = new Date().getTime();\nvar old = now - ( (60 * 60) * 24 ); // now-1 day\nif( context.global.cache == undefined ) context.global.cache = {};\nvar cache = context.global.cache;\nif( cache[cacheid] == undefined ) cache[cacheid] = {};\n\n// loop thru items and delete if seen before\nvar items = msg.payload;\nfor( i in items ){\n var jsonstr = JSON.stringify( msg.payload[i] );\n if( cache[cacheid][jsonstr] != undefined && enabled) delete msg.payload[i];\n else cache[cacheid][jsonstr] = now;\n}\n\nfor( i in cache[cacheid] ) // cleanup cache\n if( cache[cacheid][i] < old ) delete cache[cacheid][i];\n\ncontext.global.cache = cache;\n\nreturn msg;","outputs":1,"x":489,"y":639,"z":"886b17b1.7794e8","wires":[["d0d04ab0.2f2fb8"]]},{"id":"85b26b47.7a4d98","type":"function","name":"dummy data","func":"\nmsg.payload = [\"foo\",\"bar\"];\nreturn msg;","outputs":1,"x":293,"y":639,"z":"886b17b1.7794e8","wires":[["b0d6e4a1.4f2918"]]}]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment