flux-sched issue #931 dmesg output

flux-dmesg.out

$ flux dmesg                                                                                           
2022-04-14T13:40:22.028442Z broker.debug[0]: insmod connector-local                                                                                  
2022-04-14T13:40:22.028556Z broker.info[0]: start: none->join 0.0248116s                                                                             
2022-04-14T13:40:22.028602Z broker.info[0]: overlay auth 1 OK
2022-04-14T13:40:22.028867Z broker.info[0]: parent-none: join->init 0.00030235s
2022-04-14T13:40:22.030284Z broker.info[0]: accepting connection from slurm-spack-flux-login0 (rank 1) uuid 03cc7648-59d8-4f1d-82d1-f59e15b2d3ac status partial
2022-04-14T13:40:22.030441Z broker.info[0]: online:  (ranks )
2022-04-14T13:40:22.031045Z broker.info[0]: overlay auth 2 OK
2022-04-14T13:40:22.031485Z broker.info[0]: accepting connection from slurm-spack-flux-login0 (rank 2) uuid 2d4b5c7a-641c-4593-8a73-260e59b94875 status full
2022-04-14T13:40:22.033405Z connector-local.debug[0]: allow-guest-user=false
2022-04-14T13:40:22.033417Z connector-local.debug[0]: allow-root-owner=false
2022-04-14T13:40:22.077864Z broker.debug[0]: insmod barrier
2022-04-14T13:40:22.101204Z broker.debug[0]: insmod content-sqlite
2022-04-14T13:40:22.103154Z broker.debug[0]: content backing store: enabled content-sqlite
2022-04-14T13:40:22.123489Z broker.debug[0]: insmod kvs
2022-04-14T13:40:22.145877Z broker.debug[0]: insmod kvs-watch
2022-04-14T13:40:22.177529Z broker.debug[0]: insmod resource
2022-04-14T13:40:22.188905Z resource.debug[0]: reslog_cb: resource-init event posted
2022-04-14T13:40:22.207084Z broker.debug[0]: insmod cron
2022-04-14T13:40:22.207822Z cron.info[0]: synchronizing cron tasks to event heartbeat.pulse
2022-04-14T13:40:22.230813Z broker.debug[0]: insmod job-manager
2022-04-14T13:40:22.231875Z job-manager.info[0]: restart: 0 jobs
2022-04-14T13:40:22.231891Z job-manager.info[0]: restart: 0 running jobs
2022-04-14T13:40:22.232046Z job-manager.info[0]: restart: no checkpoint object
2022-04-14T13:40:22.232060Z job-manager.debug[0]: restart: max_jobid=0
2022-04-14T13:40:22.253583Z broker.debug[0]: insmod job-info
2022-04-14T13:40:22.278034Z broker.debug[0]: insmod job-list
2022-04-14T13:40:22.278956Z job-list.debug[0]: job_state_init_from_kvs: read 0 jobs
2022-04-14T13:40:22.314229Z broker.debug[0]: insmod job-ingest
2022-04-14T13:40:22.314951Z job-ingest.debug[0]: configuring with plugins=(null), args=(null)
2022-04-14T13:40:22.315160Z job-ingest.debug[0]: fluid ts=1ms
2022-04-14T13:40:22.332740Z broker.debug[0]: insmod job-exec
2022-04-14T13:40:22.333451Z job-exec.debug[0]: using default shell path /sw/spack/opt/spack/linux-centos7-cascadelake/gcc-10.3.0/flux-core-master-jobhovyngogyyh6nkxejqfu32o2nv6io/libexec/flux/flux-shell
2022-04-14T13:40:22.355950Z broker.debug[0]: insmod heartbeat
2022-04-14T13:40:22.358746Z broker.info[0]: rc1.0: running /sw/spack/opt/spack/linux-centos7-cascadelake/gcc-10.3.0/flux-core-master-jobhovyngogyyh6nkxejqfu32o2nv6io/etc/flux/rc1.d/02-cron
2022-04-14T13:40:22.377411Z broker.info[0]: rc1.0: running /sw/spack/opt/spack/linux-centos7-cascadelake/gcc-10.3.0/flux-sched-master-i6pjazqxxez2jut4f5bfiqdv4z7yaygr/etc/flux/rc1.d/01-sched-fluxion-resource-start
2022-04-14T13:40:22.441748Z broker.debug[0]: insmod sched-fluxion-resource
2022-04-14T13:40:22.443322Z sched-fluxion-resource.debug[0]: mod_main: resource module starting
2022-04-14T13:40:22.444960Z broker.info[0]: rc1.0: running /sw/spack/opt/spack/linux-centos7-cascadelake/gcc-10.3.0/flux-sched-master-i6pjazqxxez2jut4f5bfiqdv4z7yaygr/etc/flux/rc1.d/02-sched-fluxion-qmanager-start
2022-04-14T13:40:22.500258Z broker.debug[0]: insmod sched-fluxion-qmanager
2022-04-14T13:40:22.501129Z sched-fluxion-qmanager.debug[0]: service_register
2022-04-14T13:40:22.501224Z sched-fluxion-qmanager.debug[0]: enforced policy (queue=default): fcfs
2022-04-14T13:40:22.501249Z sched-fluxion-qmanager.debug[0]: effective queue params (queue=default): default
2022-04-14T13:40:22.501256Z sched-fluxion-qmanager.debug[0]: effective policy params (queue=default): default
2022-04-14T13:40:22.642002Z broker.info[0]: rc1.0: /sw/spack/opt/spack/linux-centos7-cascadelake/gcc-10.3.0/flux-core-master-jobhovyngogyyh6nkxejqfu32o2nv6io/etc/flux/rc1 Exited (rc=0) 0.6s
2022-04-14T13:40:22.642075Z broker.info[0]: rc1-success: init->quorum 0.613199s
2022-04-14T13:40:22.743037Z broker.debug[0]: groups: broker.online=0
2022-04-14T13:40:22.743171Z broker.info[0]: online: slurm-spack-flux-login0 (ranks 0)
2022-04-14T13:40:22.744011Z resource.debug[0]: reslog_cb: online event posted
2022-04-14T13:40:23.154151Z job-manager.debug[0]: duration-validator: updated expiration to 0.00
2022-04-14T13:40:23.154748Z resource.debug[0]: reslog_cb: resource-define event posted
2022-04-14T13:40:23.155243Z sched-fluxion-resource.warning[0]: create_reader: allowlist unsupported
2022-04-14T13:40:23.157177Z sched-fluxion-resource.debug[0]: resource graph datastore loaded with rv1exec reader
2022-04-14T13:40:23.157223Z sched-fluxion-resource.info[0]: populate_resource_db: loaded resources from core's resource.acquire
2022-04-14T13:40:23.159091Z sched-fluxion-resource.debug[0]: resource status changed (rankset=[all] status=DOWN)
2022-04-14T13:40:23.159131Z sched-fluxion-resource.debug[0]: resource status changed (rankset=[0] status=UP)
2022-04-14T13:40:23.159137Z sched-fluxion-resource.debug[0]: mod_main: resource graph database loaded
2022-04-14T13:40:23.159604Z sched-fluxion-qmanager.debug[0]: handshaking with sched-fluxion-resource completed
2022-04-14T13:40:23.159755Z job-manager.debug[0]: scheduler: hello
2022-04-14T13:40:23.159920Z job-manager.debug[0]: scheduler: ready unlimited
2022-04-14T13:40:23.160075Z sched-fluxion-qmanager.debug[0]: handshaking with job-manager completed
2022-04-14T13:40:23.232134Z broker.debug[0]: groups: broker.online=0-2
2022-04-14T13:40:23.232243Z broker.info[0]: online: slurm-spack-flux-login[0,0,0] (ranks 0-2)
2022-04-14T13:40:23.233420Z resource.debug[0]: reslog_cb: online event posted
2022-04-14T13:40:23.233632Z sched-fluxion-resource.debug[0]: resource status changed (rankset=[1-2] status=UP)
2022-04-14T13:40:23.556398Z broker.debug[0]: groups: broker.online=0-3
2022-04-14T13:40:23.556539Z broker.info[0]: online: slurm-spack-flux-login[0,0,0,0] (ranks 0-3)
2022-04-14T13:40:23.556563Z broker.info[0]: quorum-full: quorum->run 0.914479s
2022-04-14T13:40:23.559912Z resource.debug[0]: reslog_cb: online event posted
2022-04-14T13:40:23.560169Z sched-fluxion-resource.debug[0]: resource status changed (rankset=[3] status=UP)
2022-04-14T13:40:37.244658Z job-ingest.debug[0]: validator[0]: flux-job-validator: CRITICAL: validator plugin 'jobspec' not found
2022-04-14T13:40:37.257333Z job-ingest.err[0]: validator[0]: exited with rc=1
2022-04-14T13:44:07.601812Z job-ingest.debug[0]: validator[0]: flux-job-validator: CRITICAL: validator plugin 'jobspec' not found
2022-04-14T13:44:07.612673Z job-ingest.err[0]: validator[0]: exited with rc=1
2022-04-14T13:44:20.665750Z job-ingest.debug[0]: validator[0]: flux-job-validator: CRITICAL: validator plugin 'jobspec' not found
2022-04-14T13:44:20.676583Z job-ingest.err[0]: validator[0]: exited with rc=1
2022-04-14T13:44:34.064119Z job-ingest.debug[0]: validator[0]: flux-job-validator: CRITICAL: validator plugin 'jobspec' not found
2022-04-14T13:44:34.074957Z job-ingest.err[0]: validator[0]: exited with rc=1