description | pipeline | in | out | data | airflow? |
---|---|---|---|---|---|
enrich | EnrichCounters.scala | PS | PS | iface, sap | n/a |
real-time | cloud functions | PS | BT | iface, sap | n/a |
streaming metadata | SnmpMetadataBigtalbeIngest.scala | PS | BT | iface, sap | |
rollups | BigtableSNMPInterfaceRollup.scala | BT | BT | iface | yes |
tag totals (intercloud, etc) | AcceptedTraffic.scala | BQ | BT | ||
daily volume | InterfaceDailySums.scala | BQ | BT | iface | |
monthly volume | InterfaceMonthlySums.scala | BT | BT | iface | |
future | |||||
finalize | SnmpToBigtable.scala | BQ | BT | iface, sap |
Proposed changes:
Short term:
- BigtableSNMPInterfaceRollup.scala -> BigtableRollup.scala
- Update BigtableRollup to handle configurable resource/metric -- solves SAP issue
- EnrichCounters.scala -> SNMPEnrichment.scala
- Add support for errors and discards, etc
- AcceptedTraffic.scala -> TagTotals.scala
- InterfaceDailySums.scala -> DailyVolume.scala
- InterfaceMonthlySums.scala -> MonthlyVolume.scala
- Ingest.scala -> SNMPImmutableIngest.scala
Medium term:
- move SNMPEnrichment.scala and SNMPImmutableIngest.scala into SNMP project
- create ImmutableIngest.scala that works in terms of namespace/resource/metrics
Evolving SNMP metadata to plain resource/metrics
- EnrichSNMP has rules to convert raw SNMP into metadata resource/metrics ->
- -> PubSub that feeds ImmutableIngest and cloud functions in core (transitional)
- -> gRPC Sink that uses gRPC API to put plain resource/metrics into core
- EnrichSNMP updated to read from plain resource/metrics
- needs to read metadata resource/metric ranges via core gRPC API
- Move to SNMP project
- TODO: How to handle requirement for simple access to raw data?
- configure enrich to pass data through
- configure core to do minimal processing into BT
Currently only setup for interfaces. Needs changing to either accept the regex to use or have a flag per data type.
Rename SnmpToBigtable to finalize
look at finalize in more detail
MetadataReconciler -> Metadata finalize.