inertia186/.description.regult.txt

## .description.regult.txt
Regult ('regult.rb') for STEEM.  Please have a look at the README.md file.

## .gitignore
last-block-num.txt
proxy-list.txt

## README.md

      
    Raw
  

              README.md
            
          
Title: regult.rb - Proxy Sentinel
Tags: radiator ruby steem steemdev witness-category
Notes:

Overview

Regult (regult.rb) is a proxy sentinel that streams the blockchain, looking for proxy/unproxy operations to update a local text file with a list of accounts that proxy their witness vote to a particular account you're interested in tracking.
In addition to streaming the blockchain, in order to save time and resources, this script will also initialize the list from data on Steem SQL.  Since Steem SQL is paid service, I figure using Steem SQL should be optional.  If you can manually initialize the proxy-list.txt file using some other method, using Steem SQL should not be required.
It's also possible to just let Regult build the proxy-list.txt file from the Genesis block, but that's a bad idea if you're using a public node.  Don't do that.

Install

To use this Radiator script:
Linux

$ sudo apt-get update
$ sudo apt-get install ruby-full git openssl libssl1.0.0 libssl-dev
$ sudo apt-get upgrade
$ gem install bundler
macOS

$ gem install bundler
I've tested it on various versions of ruby.  The oldest one I got it to work was:
ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-darwin17]
First, clone this gist and install the dependencies:
$ git clone https://gist.github.com/cfce5b47fa0f5b1eae0c6dd9c653ca7d.git regult
$ cd regult
$ bundle install
You should edit regult.yml and point it at the correct account (proxy_target_account).
If you intend to use Steem SQL, you can provide the credentials in authorize-steem-sql.sh, then use this terminal command to enable Steem SQL just before running this script:
source authorize-steem-sql.sh
Now you can skip to ruby regult.rb.
Otherwise, if you do not intend to use Steem SQL, in order to avoid long replays, get a list of accounts that currently proxy to proxy_target_account and create a file called proxy-list.txt with the entire list of account names only, one account name per line (you can typically get this list from steemdb.com, for example: https://steemdb.com/@jesta/proxied is a list of accounts that proxy to jesta).
Then create a file called last-block-num.txt and add a single line with the latest block number, or whatever number you'd like to begin streaming from.
Once those two files exist, you can run this script and the proxy-list.txt file will be kept current whenever a relevant proxy/unproxy operation is seen.
Then run it:
$ ruby regult.rb
Regult will now do it's thing.  Check here to see an updated version of this script:
https://gist.github.com/inertia186/cfce5b47fa0f5b1eae0c6dd9c653ca7d

Upgrade

Typically, you can upgrade to the latest version by this command, from the original directory you cloned into:
$ git pull
Usually, this works fine as long as you haven't modified anything.  If you get an error, try this:
$ git stash --all
$ git pull --rebase
$ git stash pop

If you're still having problems, I suggest starting a new clone.

Troubleshooting

Problem: What does this error mean?

regult.yml:1: syntax error, unexpected ':', expecting end-of-input

Solution: You ran ruby regult.yml but you should run ruby regult.rb.

Problem: The node I'm using is down.

Is there a list of nodes?
Solution: Yes, special thanks to @jamzed.

https://geo.steem.pl/


  _{art by randychen}

See my previous Ruby How To posts in: #radiator #ruby
Get in touch!

If you're using Regult, I'd love to hear from you.  Drop me a line and tell me what you think!  I'm @inertia on STEEM and SteemSpeak.
License

I don't believe in intellectual "property".  If you do, consider Regult as licensed under a Creative Commons  License.

  
## authorize-steem-sql.sh
#!/bin/bash

export STEEMSQL_HOST=vip.steemsql.com
export STEEMSQL_USERNAME=
export STEEMSQL_PASSWORD=

## Gemfile
source 'https://rubygems.org'
gem 'radiator'
gem 'steem_api'

## Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    activemodel (5.1.6)
      activesupport (= 5.1.6)
    activerecord (5.1.6)
      activemodel (= 5.1.6)
      activesupport (= 5.1.6)
      arel (~> 8.0)
    activerecord-sqlserver-adapter (5.1.6)
      activerecord (~> 5.1.0)
      tiny_tds
    activesupport (5.1.6)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 0.7, < 2)
      minitest (~> 5.1)
      tzinfo (~> 1.1)
    arel (8.0.0)
    awesome_print (1.8.0)
    bitcoin-ruby (0.0.18)
    concurrent-ruby (1.0.5)
    connection_pool (2.2.2)
    domain_name (0.5.20180417)
      unf (>= 0.0.5, < 1.0.0)
    ffi (1.9.25)
    hashie (3.5.7)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    i18n (1.0.1)
      concurrent-ruby (~> 1.0)
    json (2.1.0)
    little-plugger (1.1.4)
    logging (2.2.2)
      little-plugger (~> 1.1)
      multi_json (~> 1.10)
    mime-types (3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2016.0521)
    mini_portile2 (2.3.0)
    minitest (5.11.3)
    multi_json (1.13.1)
    net-http-persistent (3.0.0)
      connection_pool (~> 2.2)
    netrc (0.11.0)
    nokogiri (1.8.3)
      mini_portile2 (~> 2.3.0)
    radiator (0.4.2)
      awesome_print (~> 1.7, >= 1.7.0)
      bitcoin-ruby (~> 0.0, >= 0.0.11)
      ffi (~> 1.9, >= 1.9.18)
      hashie (~> 3.5, >= 3.5.5)
      json (~> 2.0, >= 2.0.2)
      logging (~> 2.2, >= 2.2.0)
      net-http-persistent (>= 2.5.2)
    rest-client (2.0.2)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    steem_api (1.1.1)
      activerecord (>= 4, < 6)
      activerecord-sqlserver-adapter (>= 4, < 6)
      awesome_print (~> 1.7, >= 1.7.0)
      nokogiri (~> 1.8)
      rest-client (~> 2.0)
      tiny_tds (~> 1.3)
    thread_safe (0.3.6)
    tiny_tds (1.3.0)
      mini_portile2 (~> 2.0)
    tzinfo (1.2.5)
      thread_safe (~> 0.1)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.5)

PLATFORMS
  ruby

DEPENDENCIES
  radiator
  steem_api

BUNDLED WITH
   1.16.1

## regult.rb
# Regult - proxy sentinel

require 'rubygems'
require 'bundler/setup'
require 'yaml'

Bundler.require

def write_last_block_num(block_num)
  File.open(@last_block_num_path, 'w') { |f| f.write(block_num.to_s) }
end

def read_last_block_num
  File.open(@last_block_num_path) { |f| f.read.to_i } rescue 0
end

def append_proxy_account(account_name, options = {temp: false})
  path = if !!options[:temp]
    @temp_proxy_list_path
  else
    @proxy_list_path
  end

  if File.exist? path
    account_names = read_proxy_accounts(options)
    return if account_names.include? account_name
  end

  unless !!options[:temp]
    puts "Appending #{account_name} to #{@proxy_target_account} proxy list."
  end

  File.open(path, 'a') { |f| f.puts(account_name) }
end

def drop_proxy_account(account_name, options = {temp: false})
  path = if !!options[:temp]
    @temp_proxy_list_path
  else
    @proxy_list_path
  end

  return unless File.exist? path

  account_names = read_proxy_accounts(options)

  return unless account_names.include? account_name

  unless !!options[:temp]
    puts "Dropping #{account_name}, no longer proxying to #{@proxy_target_account}."
  end

  # Removing duplicates, if any (being paranoid).
  while account_names.include? account_name
    account_names -= [account_name]
  end

  File.open(path, 'w') do |f|
    account_names.each do |line|
      f.puts line
    end
  end
end

def read_proxy_accounts(options = {temp: false})
  path = if !!options[:temp]
    @temp_proxy_list_path
  else
    @proxy_list_path
  end

  return [] unless File.exists? path

  File.readlines(path).map(&:strip)
end

def api
  @api ||= Radiator::Api.new(@chain_options)
end

# This will read proxy data from Steem SQL so it can get up to date quickly.  In
# theory, as long as the last_block_num isn't too far behind, this should
# usually be redundant.
def sync_from_steem_sql
  if File.exist? @proxy_list_path
    # It can take a while to rebuild this file, so let's use a temporary file
    # until we're done so we can update it more atomically.
    FileUtils.copy_file @proxy_list_path, @temp_proxy_list_path
  end

  begin
    operations = SteemApi::Tx::AccountWitnessProxy.where(proxy: @proxy_target_account)
    operations.find_each do |op|
      append_proxy_account op.account, temp: true
    end

    if operations.any?
      tx_id = operations.maximum(:tx_id)
      transaction = SteemApi::Transaction.find_by(tx_id: tx_id)
      if transaction.block_num > read_last_block_num
        write_last_block_num transaction.block_num # skip to latest block_num
      end
    end

    @steem_sql_enabled = true
  rescue => e
    puts "Cannot look up past account_witness_proxy ops at this time: #{e.inspect}"
    @steem_sql_enabled = false
  end

  begin
    if @steem_sql_enabled
      read_proxy_accounts(temp: true).each do |account_name|
        operations = SteemApi::Tx::AccountWitnessProxy.where(account: account_name)
        proxy_enabled = false
        operations.find_each do |op|
          if !proxy_enabled && op.Proxy == @proxy_target_account
            proxy_enabled = true
          end

          if proxy_enabled && op.Proxy != @proxy_target_account
            proxy_enabled = false
          end
        end

        if proxy_enabled
          append_proxy_account account_name, temp: true
        else
          drop_proxy_account account_name, temp: true
        end

        next if operations.none?

        tx_id = operations.maximum(:tx_id)
        transaction = SteemApi::Transaction.find_by(tx_id: tx_id)
        if transaction.block_num > read_last_block_num
          write_last_block_num transaction.block_num # skip to latest block_num
        end
      end
    end
  rescue => e
    puts "Cannot look up past account_witness_proxy ops at this time: #{e.inspect}"
    @steem_sql_enabled = false
  end

  if !!@steem_sql_enabled && ( !File.exist?(@proxy_list_path) || (read_proxy_accounts(temp: true) & read_proxy_accounts).any?)
    if File.exist?(@temp_proxy_list_path)
      FileUtils.mv @temp_proxy_list_path, @proxy_list_path
    end

    # Seems like everything is working, so let's skip these blocks and only
    # stream from up to one hour ago.

    api.get_dynamic_global_properties do |properties|
      block_num = properties.last_irreversible_block_num - 1200
      write_last_block_num [block_num, read_last_block_num].max
    end
  elsif File.exist?(@temp_proxy_list_path)
    FileUtils.remove_file @temp_proxy_list_path
  end
end

@config_path = __FILE__.sub(/\.rb$/, '.yml')

unless File.exist? @config_path
  puts "Unable to find: #{@config_path}"
  exit
end

@config = YAML.load_file(@config_path)
@proxy_target_account = @config[:proxy_target_account]
@chain_options = @config[:chain_options]
@last_block_num_path = @config[:last_block_num_path]
@proxy_list_path = @config[:proxy_list_path]
@temp_proxy_list_path = @proxy_list_path + '.tmp'
@stream = Radiator::Stream.new(@chain_options)
@last_irreversible_block_num = api.get_dynamic_global_properties do |properties|
  properties.last_irreversible_block_num
end

sync_from_steem_sql

@start_block_num = read_last_block_num

@stream.blocks @start_block_num + 1, :irreversible do |block, block_num|
  remaining_blocks = @last_irreversible_block_num - block_num
  transactions = block.transactions
  height = (remaining_blocks * 3) / 86400.0 # days
  @start ||= Time.now.utc
  elapsed = (Time.now.utc - @start) / 3600 # hours

  if height < 0
    puts "Caught up ... #{block_num}"
    @start = nil
  else
    puts "Catching up ... #{block_num}: height: #{('%.3f' % height)} days; elapsed: #{('%.3f' % elapsed)} hours; blocks remain: #{remaining_blocks}"
  end

  transactions.each do |transaction|
    transaction.operations.each do |type, op|
      if type == 'account_witness_proxy'
        account_names = read_proxy_accounts

        if account_names.include? op.account
          if op.proxy == @proxy_target_account
            puts "Weird, saw a new account_witness_proxy operation, but #{op.account} was already proxying to #{@proxy_target_account}."
          else
            drop_proxy_account op.account
          end
        elsif op.proxy == @proxy_target_account
          append_proxy_account op.account
        end
      end
    end
  end

  write_last_block_num block_num
end

## regult.yml
# Account to monitor.
:proxy_target_account: jesta

# Where this script will look for the last block number when it begins
# streaming.
:last_block_num_path: last-block-num.txt

# List of accounts that proxy the target account.
:proxy_list_path: proxy-list.txt

:chain_options:
  :url: https://api.steemit.com
	#!/bin/bash

	export STEEMSQL_HOST=vip.steemsql.com
	export STEEMSQL_USERNAME=
	export STEEMSQL_PASSWORD=
	GEM
	remote: https://rubygems.org/
	specs:
	activemodel (5.1.6)
	activesupport (= 5.1.6)
	activerecord (5.1.6)
	activemodel (= 5.1.6)
	activesupport (= 5.1.6)
	arel (~> 8.0)
	activerecord-sqlserver-adapter (5.1.6)
	activerecord (~> 5.1.0)
	tiny_tds
	activesupport (5.1.6)
	concurrent-ruby (~> 1.0, >= 1.0.2)
	i18n (>= 0.7, < 2)
	minitest (~> 5.1)
	tzinfo (~> 1.1)
	arel (8.0.0)
	awesome_print (1.8.0)
	bitcoin-ruby (0.0.18)
	concurrent-ruby (1.0.5)
	connection_pool (2.2.2)
	domain_name (0.5.20180417)
	unf (>= 0.0.5, < 1.0.0)
	ffi (1.9.25)
	hashie (3.5.7)
	http-cookie (1.0.3)
	domain_name (~> 0.5)
	i18n (1.0.1)
	concurrent-ruby (~> 1.0)
	json (2.1.0)
	little-plugger (1.1.4)
	logging (2.2.2)
	little-plugger (~> 1.1)
	multi_json (~> 1.10)
	mime-types (3.1)
	mime-types-data (~> 3.2015)
	mime-types-data (3.2016.0521)
	mini_portile2 (2.3.0)
	minitest (5.11.3)
	multi_json (1.13.1)
	net-http-persistent (3.0.0)
	connection_pool (~> 2.2)
	netrc (0.11.0)
	nokogiri (1.8.3)
	mini_portile2 (~> 2.3.0)
	radiator (0.4.2)
	awesome_print (~> 1.7, >= 1.7.0)
	bitcoin-ruby (~> 0.0, >= 0.0.11)
	ffi (~> 1.9, >= 1.9.18)
	hashie (~> 3.5, >= 3.5.5)
	json (~> 2.0, >= 2.0.2)
	logging (~> 2.2, >= 2.2.0)
	net-http-persistent (>= 2.5.2)
	rest-client (2.0.2)
	http-cookie (>= 1.0.2, < 2.0)
	mime-types (>= 1.16, < 4.0)
	netrc (~> 0.8)
	steem_api (1.1.1)
	activerecord (>= 4, < 6)
	activerecord-sqlserver-adapter (>= 4, < 6)
	awesome_print (~> 1.7, >= 1.7.0)
	nokogiri (~> 1.8)
	rest-client (~> 2.0)
	tiny_tds (~> 1.3)
	thread_safe (0.3.6)
	tiny_tds (1.3.0)
	mini_portile2 (~> 2.0)
	tzinfo (1.2.5)
	thread_safe (~> 0.1)
	unf (0.1.4)
	unf_ext
	unf_ext (0.0.7.5)

	PLATFORMS
	ruby

	DEPENDENCIES
	radiator
	steem_api

	BUNDLED WITH
	1.16.1
	# Regult - proxy sentinel

	require 'rubygems'
	require 'bundler/setup'
	require 'yaml'

	Bundler.require

	def write_last_block_num(block_num)
	File.open(@last_block_num_path, 'w') { \|f\| f.write(block_num.to_s) }
	end

	def read_last_block_num
	File.open(@last_block_num_path) { \|f\| f.read.to_i } rescue 0
	end

	def append_proxy_account(account_name, options = {temp: false})
	path = if !!options[:temp]
	@temp_proxy_list_path
	else
	@proxy_list_path
	end

	if File.exist? path
	account_names = read_proxy_accounts(options)
	return if account_names.include? account_name
	end

	unless !!options[:temp]
	puts "Appending #{account_name} to #{@proxy_target_account} proxy list."
	end

	File.open(path, 'a') { \|f\| f.puts(account_name) }
	end

	def drop_proxy_account(account_name, options = {temp: false})
	path = if !!options[:temp]
	@temp_proxy_list_path
	else
	@proxy_list_path
	end

	return unless File.exist? path

	account_names = read_proxy_accounts(options)

	return unless account_names.include? account_name

	unless !!options[:temp]
	puts "Dropping #{account_name}, no longer proxying to #{@proxy_target_account}."
	end

	# Removing duplicates, if any (being paranoid).
	while account_names.include? account_name
	account_names -= [account_name]
	end

	File.open(path, 'w') do \|f\|
	account_names.each do \|line\|
	f.puts line
	end
	end
	end

	def read_proxy_accounts(options = {temp: false})
	path = if !!options[:temp]
	@temp_proxy_list_path
	else
	@proxy_list_path
	end

	return [] unless File.exists? path

	File.readlines(path).map(&:strip)
	end

	def api
	@api \|\|= Radiator::Api.new(@chain_options)
	end

	# This will read proxy data from Steem SQL so it can get up to date quickly. In
	# theory, as long as the last_block_num isn't too far behind, this should
	# usually be redundant.
	def sync_from_steem_sql
	if File.exist? @proxy_list_path
	# It can take a while to rebuild this file, so let's use a temporary file
	# until we're done so we can update it more atomically.
	FileUtils.copy_file @proxy_list_path, @temp_proxy_list_path
	end

	begin
	operations = SteemApi::Tx::AccountWitnessProxy.where(proxy: @proxy_target_account)
	operations.find_each do \|op\|
	append_proxy_account op.account, temp: true
	end

	if operations.any?
	tx_id = operations.maximum(:tx_id)
	transaction = SteemApi::Transaction.find_by(tx_id: tx_id)
	if transaction.block_num > read_last_block_num
	write_last_block_num transaction.block_num # skip to latest block_num
	end
	end

	@steem_sql_enabled = true
	rescue => e
	puts "Cannot look up past account_witness_proxy ops at this time: #{e.inspect}"
	@steem_sql_enabled = false
	end

	begin
	if @steem_sql_enabled
	read_proxy_accounts(temp: true).each do \|account_name\|
	operations = SteemApi::Tx::AccountWitnessProxy.where(account: account_name)
	proxy_enabled = false
	operations.find_each do \|op\|
	if !proxy_enabled && op.Proxy == @proxy_target_account
	proxy_enabled = true
	end

	if proxy_enabled && op.Proxy != @proxy_target_account
	proxy_enabled = false
	end
	end

	if proxy_enabled
	append_proxy_account account_name, temp: true
	else
	drop_proxy_account account_name, temp: true
	end

	next if operations.none?

	tx_id = operations.maximum(:tx_id)
	transaction = SteemApi::Transaction.find_by(tx_id: tx_id)
	if transaction.block_num > read_last_block_num
	write_last_block_num transaction.block_num # skip to latest block_num
	end
	end
	end
	rescue => e
	puts "Cannot look up past account_witness_proxy ops at this time: #{e.inspect}"
	@steem_sql_enabled = false
	end

	if !!@steem_sql_enabled && ( !File.exist?(@proxy_list_path) \|\| (read_proxy_accounts(temp: true) & read_proxy_accounts).any?)
	if File.exist?(@temp_proxy_list_path)
	FileUtils.mv @temp_proxy_list_path, @proxy_list_path
	end

	# Seems like everything is working, so let's skip these blocks and only
	# stream from up to one hour ago.

	api.get_dynamic_global_properties do \|properties\|
	block_num = properties.last_irreversible_block_num - 1200
	write_last_block_num [block_num, read_last_block_num].max
	end
	elsif File.exist?(@temp_proxy_list_path)
	FileUtils.remove_file @temp_proxy_list_path
	end
	end

	@config_path = __FILE__.sub(/\.rb$/, '.yml')

	unless File.exist? @config_path
	puts "Unable to find: #{@config_path}"
	exit
	end

	@config = YAML.load_file(@config_path)
	@proxy_target_account = @config[:proxy_target_account]
	@chain_options = @config[:chain_options]
	@last_block_num_path = @config[:last_block_num_path]
	@proxy_list_path = @config[:proxy_list_path]
	@temp_proxy_list_path = @proxy_list_path + '.tmp'
	@stream = Radiator::Stream.new(@chain_options)
	@last_irreversible_block_num = api.get_dynamic_global_properties do \|properties\|
	properties.last_irreversible_block_num
	end

	sync_from_steem_sql

	@start_block_num = read_last_block_num

	@stream.blocks @start_block_num + 1, :irreversible do \|block, block_num\|
	remaining_blocks = @last_irreversible_block_num - block_num
	transactions = block.transactions
	height = (remaining_blocks * 3) / 86400.0 # days
	@start \|\|= Time.now.utc
	elapsed = (Time.now.utc - @start) / 3600 # hours

	if height < 0
	puts "Caught up ... #{block_num}"
	@start = nil
	else
	puts "Catching up ... #{block_num}: height: #{('%.3f' % height)} days; elapsed: #{('%.3f' % elapsed)} hours; blocks remain: #{remaining_blocks}"
	end

	transactions.each do \|transaction\|
	transaction.operations.each do \|type, op\|
	if type == 'account_witness_proxy'
	account_names = read_proxy_accounts

	if account_names.include? op.account
	if op.proxy == @proxy_target_account
	puts "Weird, saw a new account_witness_proxy operation, but #{op.account} was already proxying to #{@proxy_target_account}."
	else
	drop_proxy_account op.account
	end
	elsif op.proxy == @proxy_target_account
	append_proxy_account op.account
	end
	end
	end
	end

	write_last_block_num block_num
	end
	# Account to monitor.
	:proxy_target_account: jesta

	# Where this script will look for the last block number when it begins
	# streaming.
	:last_block_num_path: last-block-num.txt

	# List of accounts that proxy the target account.
	:proxy_list_path: proxy-list.txt

	:chain_options:
	:url: https://api.steemit.com