The Spline module collects run-time data lineage information from Spark jobs.
The Spline agent is a Scala library built into the Spark driver that listens to Spark job events and collects logical execution plans. The collected metadata is then passed to the origin manager, from where it can either be sent to the Spline server, for example via a REST API or Kafka, or used in other ways, depending on the type of manager chosen.
- add configs