Open-source, easily-accessible datasets have been shown to spurn innovation in fields like machine learning and data visualization. While the DeWi Metabase platform is suitable for ad hoc analyses and dashboards, queries can be slow and rate-limited. On the other hand, blockchain-etl allows for direct programmatic access to the entire ledger, but extensive hardware requirements and the lengthy initial sync time present a significant barrier to entry.
This grant serves a gap in Helium’s existing data engineering infrastructure by producing periodic static data dumps of key tables over discrete time periods.
The offering will accelerate exploratory analyses related to anti-gaming, token flow, coverage mapping, data usage, and more by enabling data scientists to interact with manageable datasets locally, in the programming language of their choice.
- Specifically, this could be as simple as a cron job that queries the DeWi ETL database directly, extracts important tables (e.g. gateway-inventory, last 50k blocks worth of payment transactions), and uploads them in JSON/CSV format to an S3 bucket. It should be easy for us to adjust the queries and their frequency.
- Add these public links to a static site for download.
Evan Diewald
- evan@dewi.org
- Discord: @ediewald#8607