Skip to content

Instantly share code, notes, and snippets.

@mjpowersjr
mjpowersjr / web3-etl-example.js
Last active July 12, 2022 05:22
This is a basic web3 / ETL demo. Note: 'eth-event-stream' package + refs would need to be replaced with your own abstraction layer.
require("dotenv").config();
const axios = require('axios');
const _ = require('lodash');
const util = require('util');
const sqlite3 = require('sqlite3').verbose();
const Web3 = require('web3');
// REPLACE WITH YOUR OWN web3 polling / ws ABSTRACTION!
const {
@mjpowersjr
mjpowersjr / gist:740a9583e9ec8b49e0a3
Last active May 2, 2024 01:26
Parsing the MySQL slow query log via Logstash (the easy way?)

The MySQL slow query log is a difficult format to extract information from. After looking at various examples with mixed results, I realized that it's much easier to configure MySQL to write the slow query log to a table in CSV format!

From the MySQL documentation:

By default, the log tables use the CSV storage engine that writes data in comma-separated values format. For users who have access to the .CSV files that contain log table data, the files are easy to import into other programs such as spreadsheets that can process CSV input.

my.cnf

Note: don't forget to open up permissions on your slow query log CSV file so logstash can read it!

# enable slow query log