Skip to content

Instantly share code, notes, and snippets.

@KentonPrescott

KentonPrescott/2020-10-26_checking_Infura_auction_keeper.md

Last active October 27, 2020 00:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save KentonPrescott/509f5b1b2fec7e54f9cc7637abaddb31 to your computer and use it in GitHub Desktop.
Save KentonPrescott/509f5b1b2fec7e54f9cc7637abaddb31 to your computer and use it in GitHub Desktop.
Download ZIP
2020-10-26 Checking Infura incompatibilities with `auction-keeper`
Raw
2020-10-26_checking_Infura_auction_keeper.md

This gist documents some investigation into Infura's support of filters over HTTPS. Historically, the auction-keeper has been incompatible with infura because of its lack of support in the eth_newBlockFilter.

Below, I uncover some other issues with Infura, such as unsupported parity methods as well as query limits to eth_getLogs. As originally hoped for, however, I have confirmed that the eth_newBlockFilter limitation has been resolved.

Let's begin.

Running on kovan via infura:

macbook-pro:personal-scripts kentonprescott$ ./run-kenton-auction-keeper-kovan.sh
Consider installing rusty-rlp to improve pyrlp performance with a rust based backend
2020-10-26 16:41:14,398 INFO     Keeper connected to RPC connection https://kovan.infura.io/v3/ABC123
2020-10-26 16:41:14,398 INFO     Keeper operating as 0xABC123
2020-10-26 16:41:14,727 INFO     Executing keeper startup logic
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/main.py", line 830, in <module>
    AuctionKeeper(sys.argv[1:]).main()
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/main.py", line 262, in main
    lifecycle.every(self.arguments.return_gem_interval, self.exit_gem)
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/lib/pymaker/pymaker/lifecycle.py", line 149, in __exit__
    self.startup_function()
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/main.py", line 273, in startup
    self.plunge()
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/main.py", line 324, in plunge
    pending_txes = get_pending_transactions(self.web3)
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/lib/pymaker/pymaker/__init__.py", line 383, in get_pending_transactions
    items = web3.manager.request_blocking("parity_pendingTransactions", [])
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/_virtualenv/lib/python3.7/site-packages/web3/manager.py", line 153, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method parity_pendingTransactions does not exist/is not available'}

Looks like infura does not support parity specific methods: INFURA/infura#91

Alright, let's move to mainnet, where Infura uses geth exclusively. Running on mainnet via Infura, kick only:

2020-10-26 17:02:34,507 INFO     Watching for new blocks
2020-10-26 17:02:34,508 INFO     Started 1 timer(s)
2020-10-26 17:03:38,417 ERROR    Error checking for opportunities to start an auction
Traceback (most recent call last):
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/main.py", line 236, in seq_func
    check_func()
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/main.py", line 393, in check_vaults
    urns = self.urn_history.get_urns()
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/urn_history.py", line 58, in get_urns
    return self.get_urns_from_past_frobs()
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/auction_keeper/urn_history.py", line 67, in get_urns_from_past_frobs
    frobs = self.mcd.vat.past_frobs(from_block=from_block, to_block=to_block, ilk=self.ilk)
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/lib/pymaker/pymaker/dss.py", line 562, in past_frobs
    logs = self.web3.eth.getLogs(filter_params)
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/_virtualenv/lib/python3.7/site-packages/web3/eth.py", line 503, in getLogs
    RPC.eth_getLogs, [filter_params],
  File "/Users/kentonprescott/Local_Documents/Github/kenton-auction-keeper/_virtualenv/lib/python3.7/site-packages/web3/manager.py", line 153, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32005, 'message': 'query returned more than 10000 results'}
2020-10-26 17:04:25,948 INFO     Checked auctions 0 to 125 in 47 seconds

Infura imposes a couple limitations on eth_getLogs: https://infura.io/docs/ethereum/json-rpc/eth_getLogs

To prevent queries from consuming too many resources, eth_getLogs requests are currently limited by two constraints:
- A max of 10,000 results can be returned by a single query
- Query duration must not exceed 10 seconds
If a query returns too many results or exceeds the max query duration, one of the following errors is returned:

Okay, fine, let's limit the amount of blocks we get logs from.

Running on mainnet via infura, kick only, --from-block ~= 'latest' - 100 blocks: Great! Looks like infura is working with pymaker's use of eth_newBlockFilter:

2020-10-26 17:18:45,183 INFO     Executing keeper startup logic
2020-10-26 17:18:45,183 INFO     Keeper will not bid on auctions
2020-10-26 17:18:45,183 INFO     Keeper will perform the following operation(s) in parallel:
2020-10-26 17:18:45,183 INFO     --> Check all urns and kick off new auctions if any unsafe urns need to be bitten
2020-10-26 17:18:45,183 INFO     --> Check all auctions and deal for 0xABC123
2020-10-26 17:18:45,183 INFO     *** When Keeper is dealing/bidding, the initial evaluation of auctions will likely take > 45 minutes without setting a lower boundary via '--min-auction' ***
2020-10-26 17:18:45,183 INFO     *** When Keeper is kicking, initializing urn history may take > 30 minutes without using VulcanizeDB via `--vulcanize-endpoint` ***
2020-10-26 17:18:45,289 INFO     Keeper will use Node gas price (currently 18.0 Gwei, changes over time) and will multiply by 1.125 every 42s to a maximum of 2000.0 Gwei for transactions and bids unless model instructs otherwise
2020-10-26 17:18:45,291 INFO     Watching for new blocks
2020-10-26 17:18:45,291 INFO     Started 1 timer(s)
2020-10-26 17:19:28,219 INFO     Checked 9 urns in 4 seconds
2020-10-26 17:20:15,008 INFO     Checked auctions 0 to 125 in 46 seconds
2020-10-26 17:20:20,154 INFO     Checked 9 urns in 3 seconds

And for reference, here's the startup script I used:

#!/bin/bash
/Users/path/to/auction-keeper/bin/auction-keeper \
    --rpc-host 'https://mainnet.infura.io/v3/ABC123' \
    --eth-from '0xABC123' \
    --eth-key 'key_file=/Users/path/to/keystore/file.txt,pass_file=/Users/path/to/keystore/file.txt' \
    --type flip \
    --from-block 11135053 \
    --ilk 'ETH-A' \
    --min-flip-lot 0 \
    --max-auctions 50 \
    --kick-only \
    --model "/Users/path/to/model-eth.sh" \
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment