Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

First Peer

Role: Harvesting GDELT and publishing a master data feed over a Textile Thread

Create Master Thread

Use Textile Schema to match GDELT source: https://github.com/pinata-textile-collab-space/ETHDenver2019/blob/master/gdelt-schema.json

Master Thread ID: 12D3KooWEWCJWvByS69g6ihozWxXdc6e5egyAaBCauRttQJ6kkzC

Harvest GDELT

Python script

Each record is added as a new IPFS pin via the Textile Thread

Second Remote Peer

Role: Worker to transform data & publish a new Thread for consumption by other workers

Description: Subscribes to master Thread and performs a data transformation to prepare for writing to Eth. Publishes results to a second Thread

Subscribe to the master Thread

textile invites accept INVITE_ID -k SHARED_KEY

Create second Thread

textile threads add gdelt_eth -k eth-scan-reformat -t public -s invite_only --schema=Desktop/ethschema.json

Subscribe to shared thread | write to new thread

textile sub --type FILES --thread 12D3KooWJo51aEpftjXaNWnbGU8phUrM2888NgeXkmLEGHqLo1Wk |\
jq --unbuffered  -r -c \
'{"DataLink":.block.target, "EventId":".info.date.seconds", "EventType":"GDELTV2", "DataLinkType": "IPFS"}' |\
while read -r LINE;\
  do echo $LINE | textile files add -t 12D3KooWSDwKv71XnqqEcu6g6z1foXT1PexZXm5QQs8mLNoh9asy;  \
done`

Third Remote Peer

Role: Worker to transform data

Description: Subscribe to Second thread with prepared data, write that data to Ethereum

textile sub --type FILES --thread 12D3KooWSDwKv71XnqqEcu6g6z1foXT1PexZXm5QQs8mLNoh9asy |\
jq --unbuffered -r '.info.files[0].file.hash' | \
while read -r LINE;     \
    do  curl -H "Content-Type: application/json" -s -d $(textile cat $LINE)  http://localhost/postNewHashToETH;\
done

Ethereum

Timestamped GDELT data on Ethereum https://kovan.etherscan.io/address/0x074d32dccad3e3dd9c383900dfdfbba366861f6f

IPFS pins organized the Threads

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.