Skip to content

Instantly share code, notes, and snippets.

@gkousiouris

gkousiouris/README.md

Last active April 25, 2023 17:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gkousiouris/f91edbe34663d0e4074d5ab37b65a3e0 to your computer and use it in GitHub Desktop.
Save gkousiouris/f91edbe34663d0e4074d5ab37b65a3e0 to your computer and use it in GitHub Desktop.
Download ZIP
Minio Webhook Automated Annotator
Raw
README.md

The specific subflow can be used as a webhook endpoint (/events) for getting notifications from a Minio installation and automating the enrichment of a newly inserted object with metadata.

It filters notification messages based on new objects put but also if they already have the specific metadatum needed. This is because the flow puts the object again with the new metadata, so it creates a new notification and thus an infinite loop if this comparison is not made.

The flow requires the node-red collection @reggae_ulli/node-red-contrib-minio-all-fix for the interface with the Minio server.

Pattern Images-MINIO WEBHOOK ANNOTATOR drawio

The user can configure which metadata field this skip comparison can be performed in the relevant switch node 'if already has metadata skip'.

The user also needs to configure the details of the minio instance in the minio nodes.

The flow shows an example of getting a csv file with columns id and value and extracting the min and max of the column name set in the flow UI (default: value). The user can change this in the function node 'calc and add metadata', where the relevant logic can be adapted.

This is not the most performing way to do this automation, since it doubles the transfers to Minio (needs to get the object, calculate the values and then put it back). In the future the flow will be updated to use the copyObject method of minio that can change only the metadata of the file without putting it back.

Raw
flow.json
[
{
"id": "c172de7194fdd7be",
"type": "subflow",
"name": "Minio WebHook Annotator",
"info": "The specific subflow can be used as a webhook endpoint (/events) for getting notifications from a Minio installation and automating the enrichment of a newly inserted object with metadata.\n\nIt filters notification messages based on new objects put but also if they already have the specific metadatum needed. This is because the flow puts the object again with the new metadata, so it creates a new notification and thus an infinite loop if this comparison is not made.\n\nThe user can configure which metadata field this skip comparison can be performed in the relevant switch node 'if already has metadata skip'.\n\nThe user also needs to configure the details of the minio instance in the minio nodes.\n\nThe flow shows an example of getting a csv file with columns id and value and extracting the min and max of the column name set in the flow UI (default: value). The user can change this in the function node 'calc and add metadata', where the relevant logic can be adapted.\n\nThis is not the most performing way to do this automation, since it doubles the transfers to Minio (needs to get the object, calculate the values and then put it back). In the future the flow will be updated to use the copyObject method of minio that can change only the metadata of the file without putting it back. ",
"category": "PHYSICS Helpers",
"in": [],
"out": [
{
"x": 530,
"y": 400,
"wires": [
{
"id": "3e8a76f94c81e9b2",
"port": 0
}
]
},
{
"x": 530,
"y": 480,
"wires": [
{
"id": "3e8a76f94c81e9b2",
"port": 1
}
]
}
],
"env": [
{
"name": "field",
"type": "str",
"value": "value"
}
],
"meta": {},
"color": "#cc0000",
"outputLabels": [
"Output",
"Error"
],
"icon": "@reggae_ulli/node-red-contrib-minio-all-fix/minio.png"
},
{
"id": "e3ba90d1.97c9c",
"type": "http in",
"z": "c172de7194fdd7be",
"name": "",
"url": "/events",
"method": "post",
"upload": false,
"swaggerDoc": "",
"x": 210,
"y": 80,
"wires": [
[
"83e3a574.a8c25",
"bbdc41e8.bab26"
]
]
},
{
"id": "83e3a574.a8c25",
"type": "http response",
"z": "c172de7194fdd7be",
"name": "",
"statusCode": "",
"headers": {},
"x": 530,
"y": 80,
"wires": []
},
{
"id": "98d97a02.8a3c3",
"type": "debug",
"z": "c172de7194fdd7be",
"name": "",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 510,
"y": 200,
"wires": []
},
{
"id": "b7731bac.3fe7b8",
"type": "file in",
"z": "c172de7194fdd7be",
"name": "",
"filename": "",
"format": "utf8",
"chunk": false,
"sendError": false,
"encoding": "none",
"x": 200,
"y": 280,
"wires": [
[
"b047299b.3e3b6"
]
]
},
{
"id": "b047299b.3e3b6",
"type": "csv",
"z": "c172de7194fdd7be",
"name": "",
"sep": ",",
"hdrin": true,
"hdrout": "none",
"multi": "mult",
"ret": "\\n",
"temp": "id,value",
"skip": "0",
"strings": true,
"include_empty_strings": "",
"include_null_values": "",
"x": 200,
"y": 320,
"wires": [
[
"9f310277.ca9b8"
]
]
},
{
"id": "9f310277.ca9b8",
"type": "function",
"z": "c172de7194fdd7be",
"name": "calc and add metadata",
"func": "msg.min = Math.min(...msg.payload.map(item => item[env.get('field')]));\nmsg.max = Math.max(...msg.payload.map(item => item[env.get('field')]));\nmsg.sourceObject=msg.bucketName+'/'+msg.objectName;\nmsg.metaData={'x-amz-meta-min':msg.min,'x-amz-meta-max':msg.max};//, 'x-amz-meta-directive': 'REPLACE'};\n//msg.setReplaceMetadataDirective=true;\nreturn msg;",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 250,
"y": 360,
"wires": [
[
"15a793c27e79c091",
"3e8a76f94c81e9b2"
]
]
},
{
"id": "9044990.d7819e8",
"type": "function",
"z": "c172de7194fdd7be",
"name": "add file details",
"func": "\nmsg.bucketName=msg.payload.Records[0].s3.bucket.name;\nmsg.objectName=msg.payload.Records[0].s3.object.key;\nmsg.filePath='./'+msg.objectName;\nmsg.filename='./'+msg.objectName;//needed by file node\n\nreturn msg;",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 220,
"y": 200,
"wires": [
[
"4e60ca7fa61688b8"
]
]
},
{
"id": "bbdc41e8.bab26",
"type": "switch",
"z": "c172de7194fdd7be",
"name": "if new object",
"property": "payload.EventName",
"propertyType": "msg",
"rules": [
{
"t": "eq",
"v": "s3:ObjectCreated:Put",
"vt": "str"
},
{
"t": "else"
}
],
"checkall": "true",
"repair": false,
"outputs": 2,
"x": 210,
"y": 120,
"wires": [
[
"dc23d348b93cbe01"
],
[]
]
},
{
"id": "4e60ca7fa61688b8",
"type": "files",
"z": "c172de7194fdd7be",
"files_name": "File Object Operations-GET OBJECT",
"host": "71d633cd4c03ddc4",
"files_operation": "fGetObject",
"files_bucket": "",
"files_object": "",
"files_filepath": "",
"files_metadata": "",
"x": 290,
"y": 240,
"wires": [
[
"b7731bac.3fe7b8",
"98d97a02.8a3c3"
],
[]
]
},
{
"id": "3e8a76f94c81e9b2",
"type": "files",
"z": "c172de7194fdd7be",
"files_name": "",
"host": "71d633cd4c03ddc4",
"files_operation": "fPutObject",
"files_bucket": "",
"files_object": "",
"files_filepath": "",
"files_metadata": "",
"x": 240,
"y": 420,
"wires": [
[
"fd5479a62ce20534"
],
[]
]
},
{
"id": "15a793c27e79c091",
"type": "debug",
"z": "c172de7194fdd7be",
"name": "",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 450,
"y": 320,
"wires": []
},
{
"id": "dc23d348b93cbe01",
"type": "switch",
"z": "c172de7194fdd7be",
"name": "if already has metadata skip",
"property": "payload.Records[0].s3.object.userMetadata",
"propertyType": "msg",
"rules": [
{
"t": "hask",
"v": "X-Amz-Meta-Max",
"vt": "str"
},
{
"t": "else"
}
],
"checkall": "true",
"repair": false,
"outputs": 2,
"x": 260,
"y": 160,
"wires": [
[],
[
"9044990.d7819e8"
]
]
},
{
"id": "fd5479a62ce20534",
"type": "file",
"z": "c172de7194fdd7be",
"name": "",
"filename": "local",
"appendNewline": true,
"createDir": false,
"overwriteFile": "delete",
"encoding": "none",
"x": 550,
"y": 440,
"wires": [
[]
]
},
{
"id": "71d633cd4c03ddc4",
"type": "minio-config",
"name": "MinIO Instance",
"host": "10.100.59.208",
"port": "9000",
"useSsl": false
},
{
"id": "23f3b9ba.688376",
"type": "debug",
"z": "76be78ac197e5776",
"name": "Output",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 510,
"y": 140,
"wires": []
},
{
"id": "59fef36a17ac7eb3",
"type": "subflow:c172de7194fdd7be",
"z": "76be78ac197e5776",
"name": "",
"x": 310,
"y": 160,
"wires": [
[
"23f3b9ba.688376"
],
[
"1af62e78eb50e855"
]
]
},
{
"id": "1af62e78eb50e855",
"type": "debug",
"z": "76be78ac197e5776",
"name": "Error",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 510,
"y": 200,
"wires": []
}
]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment