netbeam next steps

next-steps.md

Workflows

description	pipeline	in	out	data	airflow?
enrich	EnrichCounters.scala	PS	PS	iface, sap	n/a
real-time	cloud functions	PS	BT	iface, sap	n/a
streaming metadata	SnmpMetadataBigtalbeIngest.scala	PS	BT	iface, sap
rollups	BigtableSNMPInterfaceRollup.scala	BT	BT	iface	yes
tag totals (intercloud, etc)	AcceptedTraffic.scala	BQ	BT
daily volume	InterfaceDailySums.scala	BQ	BT	iface
monthly volume	InterfaceMonthlySums.scala	BT	BT	iface
future
finalize	SnmpToBigtable.scala	BQ	BT	iface, sap

Proposed changes:

Short term:

• BigtableSNMPInterfaceRollup.scala -> BigtableRollup.scala
• Update BigtableRollup to handle configurable resource/metric -- solves SAP issue
• EnrichCounters.scala -> SNMPEnrichment.scala
• Add support for errors and discards, etc
• AcceptedTraffic.scala -> TagTotals.scala
• InterfaceDailySums.scala -> DailyVolume.scala
• InterfaceMonthlySums.scala -> MonthlyVolume.scala
• Ingest.scala -> SNMPImmutableIngest.scala

Medium term:

• move SNMPEnrichment.scala and SNMPImmutableIngest.scala into SNMP project
• create ImmutableIngest.scala that works in terms of namespace/resource/metrics

Evolving SNMP metadata to plain resource/metrics

• EnrichSNMP has rules to convert raw SNMP into metadata resource/metrics ->
  • -> PubSub that feeds ImmutableIngest and cloud functions in core (transitional)
  • -> gRPC Sink that uses gRPC API to put plain resource/metrics into core
• EnrichSNMP updated to read from plain resource/metrics
  • needs to read metadata resource/metric ranges via core gRPC API
• Move to SNMP project
• TODO: How to handle requirement for simple access to raw data?
  • configure enrich to pass data through
  • configure core to do minimal processing into BT

Rollups

Currently only setup for interfaces. Needs changing to either accept the regex to use or have a flag per data type.

Rename SnmpToBigtable to finalize
look at finalize in more detail
MetadataReconciler -> Metadata finalize.