Skip to content

Instantly share code, notes, and snippets.

Last active May 28, 2023 21:45
Show Gist options
  • Save vsoch/756f10b52f7889e1b781ccdc599fa8cc to your computer and use it in GitHub Desktop.
Save vsoch/756f10b52f7889e1b781ccdc599fa8cc to your computer and use it in GitHub Desktop.
Steps to develop a plugin for Flux Framework

Developing a Plugin


Flux plugins live in the Flux source code, typically under "src." First you should try to identify where your plugin belongs.

  • src/shell: built-in plugins for Flux, including task and shell operations. Likely you wouldn't want to add a plugin here unless it was something core to Flux.
  • bindings/python/flux/job: is where the validator and frobnicator plugins live, and they are specific to Python.
  • modules: is where most modules live, including plugins. This includes ones core to Flux (e.g., kvs, job-info, heartbeat) but also can include those that are custom.

Likely you want to put your plugin under "src/modules" and then decide the context for which to load it.

Build Logic

Our module is not going to be considered core to Flux, so likely we won't want to build it most of the time. How do we do that? Let's start with deciding how to decide (meaning a user building the software) when to build our module. As an example, the content-s3 module isn't loaded by default, but rather conditionally. This means in "src/modules/" there are three statements you want to mimic for your module:

SUBDIRS += prometheus

fluxmod_LTLIBRARIES +=

prometheus_la_SOURCES =
prometheus_la_LIBADD = \
	$(builddir)/prometheus/ \
	$(top_builddir)/src/common/ \
	$(top_builddir)/src/common/ \
prometheus_la_LDFLAGS = $(fluxmod_ldflags) -module

For the above, we are adding an extra set of statements to build our module given that ENABLE_PROMETHEUS is defined. But wait, where does that come from? Let's look at the other examples to figure this out. It's actually logic that is in the top level of the repository in If you've use automake before, you'll be familiar with it. Here is similar logic for an existing module, linked to the ENABLE_S3 flag.

AS_IF([test "x$enable_content_s3" = "xyes"], [
    X_AC_CHECK_COND_LIB(s3, S3_initialize)
    AS_IF([test "x$ac_cv_lib_s3_S3_initialize" != "xyes"], [
      AC_MSG_ERROR([configured with --enable-content-s3, but libs3 not found])

        [AC_LANG_PROGRAM([#include <libs3.h>],
                         [S3_create_bucket (0,0,0,0,0,0,0,0,0,0,0);])],
        AC_DEFINE([HAVE_S3_AUTH_REGION], [1], [S3_create_bucket has 11 args]),
            [AC_LANG_PROGRAM([#include <libs3.h>],
                             [S3_create_bucket (0,0,0,0,0,0,0,0,0,0,0,0,0);])],
            AC_DEFINE([HAVE_S3_TIMEOUT_ARG], [1], [S3_create_bucket has 13 args])

AM_CONDITIONAL([ENABLE_CONTENT_S3], [test "x$enable_content_s3" = "xyes"])

So this is where we want to define the flag, and ensure that any additional programs / libraries are present for what we need. For the case of what I want to develop with Prometheus, there is a C++ helper library that I'd likely want to check for here. I started out very simply with this:

# prometheus plugin / module
    AS_HELP_STRING([--enable-prometheus], [Enable Prometheus metric endpoint]))

AS_IF([test "x$enable_prometheus" = "xyes"], [
    X_AC_CHECK_COND_LIB(prometheus-cpp-core, prometheus_initialize)

AM_CONDITIONAL([ENABLE_PROMETHEUS], [test "x$enable_prometheus" = "xyes"])

And likely I'll want to extend this to have more detail for the dependency. For now I'm going to put the library include in the header of my module, and the build will throw up if it doesn't have it.


1. Add Skeleton Files

Given the above, my recommendation for step 1 is to create your new module folder:

$ mkdir -p src/modules/prometheus

Add a largely empty c file there. I added the main module function main (which seemed to be needed) and a bunch of imports I probably need to clean up. This I called src/modules/prometheus/prometheus.c

 * Copyright 2020 Lawrence Livermore National Security, LLC
 * This file is part of the Flux resource manager framework.
 * For details, see
 * SPDX-License-Identifier: LGPL-3.0

#include "config.h"
#include <jansson.h>
#include <flux/core.h>
#include <assert.h>

#include "src/common/libutil/blobref.h"
#include "src/common/libutil/log.h"
#include "src/common/libutil/errprintf.h"

#include "src/common/libcontent/content-util.h"

#include "src/common/libtomlc99/toml.h"
#include "src/common/libutil/tomltk.h"

#include "src/common/libyuarel/yuarel.h"
#include "ccan/str/str.h"
// This is for prometheus - it won't build if I don't have it!
#include <prom.h>

int mod_main (flux_t *h, int argc, char **argv)
    // struct content_s3 *ctx;
    int rc = -1;

    // if (parse_args (h, argc, argv) < 0)
    //    return -1;
    // TODO run metrics here
    if (flux_reactor_run (flux_get_reactor (h), 0) < 0) {
        flux_log_error (h, "flux_reactor_run");
        goto done_unreg;
    rc = 0;
    (void)content_unregister_backing_store (h);
    // Do some destroy here
    return rc;

Add a too. I changed the library name from another simple module, and commented out tests for now (we can update that later):



	-I$(top_srcdir) \
	-I$(top_srcdir)/src/include \
	-I$(top_srcdir)/src/common/libccan \


libprometheus_la_SOURCES = \

test_ldadd = \
	$(builddir)/ \
	$(top_builddir)/src/common/ \
	$(top_builddir)/src/common/ \
	$(top_builddir)/src/common/libtap/ \

test_ldflags = \

test_cppflags = $(AM_CPPFLAGS)

check_PROGRAMS = \
	test_load \

test_load_SOURCES = test/load.c
test_load_CPPFLAGS = $(test_cppflags)
test_load_LDADD = $(test_ldadd)
test_load_LDFLAGS = $(test_ldflags)

test_store_SOURCES = test/store.c
test_store_CPPFLAGS = $(test_cppflags)
test_store_LDADD = $(test_ldadd)
test_store_LDFLAGS = $(test_ldflags)

Since I largely didn't know what I'd be testing or even building, I decided to copy another modules tests (and leave the above matching from the module's to my module, for the time being.

$ cp -R content-s3/test/ prometheus/test

2. Figure out Dependencies

and document!

When it was time to go to the and add the logic for looking for other needed libraries, I realized I needed to both figure out myself how to build the libraries, and how to then check for them in the Since I want others to be able to develop, I decided to add this build logic to documentation for flux core, and (for my purposes, and likely I will comment it out for the actual pull request) I added the logic to the developer environment container. I quickly looked in doc/ and realized that this library is organized by man pages, so my best strategy was to look for where other modules are mentioned (I was using content-s3 as an example) and add my new module there.

Unforunately it wasn't there, so I realized I'd need to make a new page. I created the "doc/plugins" section to talk about my plugin (and hopefully others can be added too). In a new page, "doc/plugins/prometheus.rst" I was able to write the instructions for compiling and installing my dependency and then went back to the

3. Write and test

Super importantly - the will generate the configure file when you run, so if you add your dependnecy logic from above (e.g., a new --enable-something flag with a check for the library) you should delete the previous configure file, run autogen again, and then you'll be able to test if your flag works:

rm configure
./configure --enable-prometheus

If you see something like:

configure: WARNING: unrecognized options: --enable-prometheus

and you definitely added it, you likely ran into this. My first go was very simple - just checking that the library was present. Since I installed to a standard location /usr/local/lib that was already added to the LD_LIBRARY_PATH I didn't need to add it.

# prometheus plugin / module
    AS_HELP_STRING([--enable-prometheus], [Enable Prometheus metric endpoint]))

AS_IF([test "x$enable_prometheus" = "xyes"], [
    X_AC_CHECK_COND_LIB(prometheus-cpp-core, prometheus_initialize)

AM_CONDITIONAL([ENABLE_PROMETHEUS], [test "x$enable_prometheus" = "xyes"])

The other place you'll need to add in the is the epilogue that says "make this Makefile" otherwise you'll get an error about no rule to make it. E.g.,:

# Epilogue
  Makefile \
  src/Makefile \
  src/common/Makefile \
  src/common/libtap/Makefile \
  src/common/liblsd/Makefile \
  src/modules/job-manager/Makefile \
  src/modules/job-list/Makefile \
  src/modules/job-exec/Makefile \
  # I added this here!
  src/modules/prometheus/Makefile \

4. Test compiling

Before you write anything complex, a basic compile should work, indicating that our build setup is good. After configure you should be able to run make, and remember that if you installed your one or more dependencies to a non-standard location, you'll need a strategy for indicating that (e.g., editing LD_LIBRARY_PATH as shown below):

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/my/special/lib
$ make

With the empty skeleton files above, you should be able to build your module!

$ ls prometheus/
Makefile  prometheus.c  prometheus.lo  prometheus.o  test

Mind you, it's useless and does nothing, but at least you are starting from a template that you know will work to build off from.

5. Write your Module

And now, write your module! May the force be with you as you navigate the C/C++ language :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment