Flux plugins live in the Flux source code, typically under "src." First you should try to identify where your plugin belongs.
- src/shell: built-in plugins for Flux, including task and shell operations. Likely you wouldn't want to add a plugin here unless it was something core to Flux.
- bindings/python/flux/job: is where the validator and frobnicator plugins live, and they are specific to Python.
- modules: is where most modules live, including plugins. This includes ones core to Flux (e.g., kvs, job-info, heartbeat) but also can include those that are custom.
Likely you want to put your plugin under "src/modules" and then decide the context for which to load it.
Our module is not going to be considered core to Flux, so likely we won't want to build it most of the time. How do we do that? Let's start with deciding how to decide (meaning a user building the software) when to build our module. As an example, the content-s3 module isn't loaded by default, but rather conditionally. This means in "src/modules/Makefile.am" there are three statements you want to mimic for your module:
if ENABLE_PROMETHEUS
SUBDIRS += prometheus
endif
if ENABLE_PROMETHEUS
fluxmod_LTLIBRARIES += prometheus.la
endif
if ENABLE_PROMETHEUS
prometheus_la_SOURCES =
prometheus_la_LIBADD = \
$(builddir)/prometheus/libprometheus.la \
$(top_builddir)/src/common/libflux-internal.la \
$(top_builddir)/src/common/libflux-core.la \
$(LIBS3)
prometheus_la_LDFLAGS = $(fluxmod_ldflags) -module
endif
For the above, we are adding an extra set of statements to build our module given that ENABLE_PROMETHEUS is defined. But wait,
where does that come from? Let's look at the other examples to figure this out. It's actually logic that is in the top level
of the repository in configure.ac. If you've use automake before, you'll be familiar with it. Here is similar logic for an existing
module, linked to the ENABLE_S3 flag.
AS_IF([test "x$enable_content_s3" = "xyes"], [
X_AC_CHECK_COND_LIB(s3, S3_initialize)
AS_IF([test "x$ac_cv_lib_s3_S3_initialize" != "xyes"], [
AC_MSG_ERROR([configured with --enable-content-s3, but libs3 not found])
])
AC_COMPILE_IFELSE(
[AC_LANG_PROGRAM([#include <libs3.h>],
[S3_create_bucket (0,0,0,0,0,0,0,0,0,0,0);])],
AC_DEFINE([HAVE_S3_AUTH_REGION], [1], [S3_create_bucket has 11 args]),
AC_COMPILE_IFELSE(
[AC_LANG_PROGRAM([#include <libs3.h>],
[S3_create_bucket (0,0,0,0,0,0,0,0,0,0,0,0,0);])],
AC_DEFINE([HAVE_S3_TIMEOUT_ARG], [1], [S3_create_bucket has 13 args])
)
)
])
AM_CONDITIONAL([ENABLE_CONTENT_S3], [test "x$enable_content_s3" = "xyes"])
So this is where we want to define the flag, and ensure that any additional programs / libraries are present for what we need. For the case of what I want to develop with Prometheus, there is a C++ helper library that I'd likely want to check for here. I started out very simply with this:
# prometheus plugin / module
AC_ARG_ENABLE([prometheus],
AS_HELP_STRING([--enable-prometheus], [Enable Prometheus metric endpoint]))
AS_IF([test "x$enable_prometheus" = "xyes"], [
X_AC_CHECK_COND_LIB(prometheus-cpp-core, prometheus_initialize)
])
AM_CONDITIONAL([ENABLE_PROMETHEUS], [test "x$enable_prometheus" = "xyes"])
And likely I'll want to extend this to have more detail for the dependency. For now I'm going to put the library include in the header of my module, and the build will throw up if it doesn't have it.
Given the above, my recommendation for step 1 is to create your new module folder:
$ mkdir -p src/modules/prometheusAdd a largely empty c file there. I added the main module function main (which seemed to be needed)
and a bunch of imports I probably need to clean up. This I called src/modules/prometheus/prometheus.c
/************************************************************\
* Copyright 2020 Lawrence Livermore National Security, LLC
* (c.f. AUTHORS, NOTICE.LLNS, COPYING)
*
* This file is part of the Flux resource manager framework.
* For details, see https://github.com/flux-framework.
*
* SPDX-License-Identifier: LGPL-3.0
\************************************************************/
#if HAVE_CONFIG_H
#include "config.h"
#endif
#include <jansson.h>
#include <flux/core.h>
#include <assert.h>
#include "src/common/libutil/blobref.h"
#include "src/common/libutil/log.h"
#include "src/common/libutil/errprintf.h"
#include "src/common/libcontent/content-util.h"
#include "src/common/libtomlc99/toml.h"
#include "src/common/libutil/tomltk.h"
#include "src/common/libyuarel/yuarel.h"
#include "ccan/str/str.h"
// This is for prometheus - it won't build if I don't have it!
#include <prom.h>
int mod_main (flux_t *h, int argc, char **argv)
{
// struct content_s3 *ctx;
int rc = -1;
// if (parse_args (h, argc, argv) < 0)
// return -1;
// TODO run metrics here
if (flux_reactor_run (flux_get_reactor (h), 0) < 0) {
flux_log_error (h, "flux_reactor_run");
goto done_unreg;
}
rc = 0;
done_unreg:
(void)content_unregister_backing_store (h);
// Do some destroy here
return rc;
}Add a Makefile.am too. I changed the library name from another simple module, and commented out tests for now (we can update that later):
AM_CFLAGS = \
$(WARNING_CFLAGS) \
$(CODE_COVERAGE_CFLAGS)
AM_LDFLAGS = \
$(CODE_COVERAGE_LIBS)
AM_CPPFLAGS = \
-I$(top_srcdir) \
-I$(top_srcdir)/src/include \
-I$(top_srcdir)/src/common/libccan \
-I$(top_builddir)/src/common/libflux
noinst_LTLIBRARIES = libprometheus.la
libprometheus_la_SOURCES = \
prometheus.c
test_ldadd = \
$(builddir)/libprometheus.la \
$(top_builddir)/src/common/libflux-internal.la \
$(top_builddir)/src/common/libflux-core.la \
$(top_builddir)/src/common/libtap/libtap.la \
$(LIBPTHREAD) $(LIBS3)
test_ldflags = \
-no-install
test_cppflags = $(AM_CPPFLAGS)
check_PROGRAMS = \
test_load \
test_store
test_load_SOURCES = test/load.c
test_load_CPPFLAGS = $(test_cppflags)
test_load_LDADD = $(test_ldadd)
test_load_LDFLAGS = $(test_ldflags)
test_store_SOURCES = test/store.c
test_store_CPPFLAGS = $(test_cppflags)
test_store_LDADD = $(test_ldadd)
test_store_LDFLAGS = $(test_ldflags)
Since I largely didn't know what I'd be testing or even building, I decided to copy another modules tests (and leave the above matching from the module's Makefile.am) to my module, for the time being.
$ cp -R content-s3/test/ prometheus/testand document!
When it was time to go to the configure.ac and add the logic for looking for other needed libraries, I realized I needed to both figure out myself how to build the libraries, and how to then check for them in the configure.ac. Since I want others to be able to develop, I decided to add this build logic to documentation for flux core, and (for my purposes, and likely I will comment it out for the actual pull request) I added the logic to the developer environment container. I quickly looked in doc/ and realized that this library is organized by man pages, so my best strategy was to look for where other modules are mentioned (I was using content-s3 as an example) and add my new module there.
Unforunately it wasn't there, so I realized I'd need to make a new page. I created the "doc/plugins" section to talk about my plugin (and hopefully others can be added too). In a new page, "doc/plugins/prometheus.rst" I was able to write the instructions for compiling and installing my dependency and then went back to the configure.ac/
Super importantly - the configure.ac will generate the configure file when you run autogen.sh, so if you add
your dependnecy logic from above (e.g., a new --enable-something flag with a check for the library) you should
delete the previous configure file, run autogen again, and then you'll be able to test if your flag works:
rm configure
autogen.sh
./configure --enable-prometheusIf you see something like:
configure: WARNING: unrecognized options: --enable-prometheusand you definitely added it, you likely ran into this. My first go was very simple - just checking that the library
was present. Since I installed to a standard location /usr/local/lib that was already added to the LD_LIBRARY_PATH
I didn't need to add it.
# prometheus plugin / module
AC_ARG_ENABLE([prometheus],
AS_HELP_STRING([--enable-prometheus], [Enable Prometheus metric endpoint]))
AS_IF([test "x$enable_prometheus" = "xyes"], [
X_AC_CHECK_COND_LIB(prometheus-cpp-core, prometheus_initialize)
])
AM_CONDITIONAL([ENABLE_PROMETHEUS], [test "x$enable_prometheus" = "xyes"])
The other place you'll need to add in the configure.ac is the epilogue that says "make this Makefile"
otherwise you'll get an error about no rule to make it. E.g.,:
##
# Epilogue
##
AC_CONFIG_FILES( \
Makefile \
src/Makefile \
src/common/Makefile \
src/common/libtap/Makefile \
src/common/liblsd/Makefile \
...
src/modules/job-manager/Makefile \
src/modules/job-list/Makefile \
src/modules/job-exec/Makefile \
# I added this here!
src/modules/prometheus/Makefile \
...
Before you write anything complex, a basic compile should work, indicating that our build setup is good.
After configure you should be able to run make, and remember that if you installed your one or more dependencies
to a non-standard location, you'll need a strategy for indicating that (e.g., editing LD_LIBRARY_PATH as shown below):
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/my/special/lib
$ makeWith the empty skeleton files above, you should be able to build your module!
$ ls prometheus/
Makefile Makefile.am Makefile.in libprometheus.la prometheus.c prometheus.lo prometheus.o testMind you, it's useless and does nothing, but at least you are starting from a template that you know will work to build off from.
And now, write your module! May the force be with you as you navigate the C/C++ language :)