Skip to content

Instantly share code, notes, and snippets.

@gkousiouris
gkousiouris / README.md
Last active November 16, 2023 02:04
Dynamic OW Orchestrator Action

The goal of this subflow is to enable the use of dynamically located Openwhisk action invocations in hybrid cloud/edge or multi-cloud scenarios, in which the location of the target function is not known at design time.

Screenshot 2023-09-15 193648

Its main operation is rather simple, it exploits incoming information from the message in order to adapt the url to which the function invocation will be performed. The incoming message to the Dynamic OW Action needs to include the action name in the node UI as well as the following fields in the msg.payload.value:

-__<action_name>_HOST

-___NAMESPACE

@gkousiouris
gkousiouris / README.md
Last active June 19, 2023 13:29
High Availability Router for Openwhisk

This is a pattern subflow in order to implement a routing logic. The subflow is used to divert traffic among locations based on a condition set. The default location is the first output whereas the fallback location is the second output.

router

The user needs to select through the subflow UI the following parameters:

  • redirection metric: one of the monitored parameters of the Openwhisk monitoring subflow, i.e. wait time, init time, duration or success percentage
  • redirection value is the target value above or under which the redirection should occur
  • comparison logic through the boolean redirectIfLarger parameter. This is used to define the comparison between the target value and the current value. So if redirectIfLargeris set to true, the redirection will occur if the redirection metric is larger than the redirection value
  • primary ratio may be used to define whether the redi
@gkousiouris
gkousiouris / README.md
Last active May 31, 2023 10:22 — forked from vasKatevas/README.md
K-means clustering flow

Test extension

This is a flow that implements a k-means clustering operation as a service. As such it can be executed inside any Node-RED environment in a service manner. It also implements the Openwhisk API specification so that it can be executed directly as a custom docker action of Openwhisk. The inputs include arrays of objects and their values and the output returns clusters with three centroids for any given input, using the k-means implementation provided by the clusters npm library. clustering_flow

Example Input:

{
   "mode": "multiple",
@gkousiouris
gkousiouris / README.md
Last active April 25, 2023 17:26
Minio Webhook Automated Annotator

The specific subflow can be used as a webhook endpoint (/events) for getting notifications from a Minio installation and automating the enrichment of a newly inserted object with metadata.

It filters notification messages based on new objects put but also if they already have the specific metadatum needed. This is because the flow puts the object again with the new metadata, so it creates a new notification and thus an infinite loop if this comparison is not made.

The flow requires the node-red collection @reggae_ulli/node-red-contrib-minio-all-fix for the interface with the Minio server.

Pattern Images-MINIO WEBHOOK ANNOTATOR drawio

The user can configure which metadata field this skip comparison can be performed in the relevant switch node 'if already has metadata skip'.

@gkousiouris
gkousiouris / README.md
Last active February 4, 2023 13:10
Producer-Consumer Pattern Implementation using the Semaphore Service Subflow

This is an example distributed producer/consumer problem implementation using the Semaphore Service Subflow.

This is based on a typical implementation that includes the use of a mutex, a full counter (initialized at 0) and an empty counter (initialized at the MAX needed slots, in this case 3). Pattern Images-SEMAPHORE SERVICE drawio

The logic of the producer/consumer appears below:

producer() {
	while (TRUE) {
@gkousiouris
gkousiouris / README.md
Last active February 8, 2023 07:30
Semaphore Service Subflow

This is a subflow to create a semaphore service to act as a distributed lock/synchronization mechanism. It uses flow variables in order to store the state of a semaphore. Changes in the semaphore value are uninterruptable at the level of the node.js process. This is due to the fact that the latter executes them in the single threaded eventloop and does not interrupt their execution unless an async method or worker thread is used in the function's implementation.

main

The subflow defines 5 endpoints:

  • POST /semaphore for new semaphore creation. The body should contain name and initialization value of the created semaphore: {"name":"<semname>","value":1}. The value needs to be a positive integer. If not positive, a 400 HTTP error code is returned. A positive float will be converted to integer. If the semaphore already exists, a 303 HTTP error code is returned.
  • DELETE /semaphore/:name nee
@gkousiouris
gkousiouris / README.md
Last active June 16, 2023 08:56
Openwhisk Sliding Window Action Monitor

This is a subflow node in order to monitor the latest performance (sliding window) of Openwhisk actions. The flow pings periodically the target Openwhisk installation in order to retrieve the last executed actions and extract statistics from their execution. The flow has been extended in order to create a third output which includes the detailed logs of failed actions during the observed window.

owmonitor_new

Configuration includes:

  • the polling time of the monitor (default 30 seconds, msg.pollingPeriod)
  • the target Openwhisk endpoint (msg.targetEndpoint)
  • credentials for that endpoint (msg.creds) in the form of user:key
  • the window of time (in minutes) in the past for which you want to retrieve results (msg.window)
  • an optional action name (msg.action), if one wants to filter specific action activations
@gkousiouris
gkousiouris / README.md
Last active February 4, 2023 13:29
OpenWhisk Skeleton Interface for Node-RED flows as functions

This flow implements the necessary interface needed by a Node-RED flow, in order to be executed as a function inside Openwhisk . Openwhisk can execute in a serverless manner any type of Docker container, as long as it implements two methods on port 8080:

owskeletonflow_witherror

-A POST /init , used in the typical OW runtime images (such as node.js etc) to retrieve the function code from the OW function DB. In the Node-RED function runtime this is not needed for functional purposes, since the code is already embedded in the flow, but it is needed in order to abide by the OW interface that expects this method. It needs to be stressed that in a warm execution (reuse of existing container from previous same function execution), the /init method is not re-executed. Hence any needed initialization per invocation should not be performed inside this method.

-A POST /run method, which is called upon inv

@gkousiouris
gkousiouris / README.md
Last active July 9, 2022 16:57
Edge Extract Transform Load (ETL) Data Collection Pattern

This subflow handles the pushing of data to an HTTP out endpoint. It accepts JSON input (object or array) in the msg.payload, which is considered the data to be pushed. The flow is intended to be run as a service at the edge, serving as an intermediate data collector. The main goal is to ensure a robust and reliable process for data collection, that is not affected by network connectivity loss or other disturbing factors at the edge. The main logic of the flow appears in the following figure.

Pattern Images-Edge ETL Service drawio

The Node-RED input layer can be used to adapt to diverse protocols needed at the Edge IoT side. An example of an HTTP In local interface that then uses the Edge ETL subflow appears in the following figure. Of course any other type of Node-RED connector node can be used (AMQP, MQTT etc) for getting the data in the msg.payload.

![usage](https://user-images.githubusercontent.

@gkousiouris
gkousiouris / README.md
Last active February 4, 2023 13:15
Request Aggregator Service Flow

This is a flow to be run as a service, in order to perform request aggregation towards a target endpoint. The purpose is to reduce the stress towards the back-end system, by minimizing the arrival rate of requests and the number of needed serving threads or containers (in the case of FaaS systems). Thus it also reduces costs and improves performance. It is mostly beneficial in cases where the request needs to set up a rather heavyweight environment (thread or container) for performing a relatively small computation (e.g. in model inference, simulation etc.).

initial

The RA gets as input the target endpoint, as well as the method to be invoked and the input data to be the payload of the aggregated call. It holds the requests until a threshold of incoming messages is reached (set in the setBatchSize endpoint), at which point it collates the inputs of the calls into one array request towards the target e