Comparison of account code

Differences.md

hashMessage

Web3 eth accounts https://github.com/ethereum/web3.js/tree/1.0/packages/web3-eth-accounts

> account.hashMessage("a message")
'0x1b37197897728e876cc5c7413c7fa1bcd7334f8f56ddaee16e725528a665fcd8'

> account.hashMessage("a")
'0x34f291c0b5f0c13c8f43e9d37c04094c22234da43f4040adb36654c98235b4b3'

Python (eth-keys) https://github.com/ethereum/eth-keys

>>> eth_utils.keccak("a message").hex()
'f47606ab5b9ae57f073ca91f67b432293ac184fef0f205ac818d07a85642c736'

>>> eth_utils.keccak(bytearray.fromhex("61")).hex()
'3ac225168df54212a25c1c01fd35bebfea408fdac2e31ddd6f80a4bbf9a5f1cb'

>>> eth_utils.keccak("a").hex()
'3ac225168df54212a25c1c01fd35bebfea408fdac2e31ddd6f80a4bbf9a5f1cb'

Geth console

> web3.sha3("a message")
"0xf47606ab5b9ae57f073ca91f67b432293ac184fef0f205ac818d07a85642c736"

> web3.sha3("a")
"0x3ac225168df54212a25c1c01fd35bebfea408fdac2e31ddd6f80a4bbf9a5f1cb"

Encryption.md

Encrypt

KDF Differences

• web3.js uses scrypt(dklen:32, n:8192, r: 8, p: 1) as kdf, whereas
• web3.py uses pbkdf2(dklen:32, c:1000000, prf: hmac-sha256).
• Geth uses scrypt (dklen:32,n:262144,p:1,r:8

Times to encrypt and decrypt:

• web3.py: 1.8s
• web3.js: 40ms
• web3.js decrypting a python-encrypted keystore: 7.6s
• web3.py decrypting a js-encrypted keystore: 36ms

JS

> account.encrypt("0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7","pw")
{ version: 3,
  id: '46c46b7f-0d89-496e-87c4-2e45c06c3fae',
  address: '39e58ee5c267683ac7694e5bfdb69eb868e09985',
  crypto: 
   { ciphertext: '384cea39da9c79155e51d8286944dc4a34a89f6afae2e732b2650e00e3fc3404',
     cipherparams: { iv: '2b5d91f790fda75160cd3d1810a9a20e' },
     cipher: 'aes-128-ctr',
     kdf: 'scrypt',
     kdfparams: 
      { dklen: 32,
        salt: '7a13b2c6e8bb49ef37a599d34396c548f4baf36698940b81af9b8debe81db878',
        n: 8192,
        r: 8,
        p: 1 },
     mac: '14aebfcc6e97398a8d0d5d94708895f02138c529a9f80e98b0913d97e88b0d1c' } }

Python

>>> la.encrypt("pw")
{'address': '39e58ee5c267683ac7694e5bfdb69eb868e09985', 'crypto': {'cipher': 'aes-128-ctr', 'cipherparams': {'iv': 'c2b6bdd46f06ac228d6eb5c084b18e57'}, 'ciphertext': 'd00fdd771567ee81c48020dc1d18a45ebb47899293f5259b3d6d4ae8ca87b743', 'kdf': 'pbkdf2', 'kdfparams': {'c': 1000000, 'dklen': 32, 'prf': 'hmac-sha256', 'salt': '3ac928757d88274877acfc68aa6113a3'}, 'mac': 'e309f7dadfaa76d3919a7db788ae19112078c0c79557c3d329860fd0c8d694a3'}, 'id': 'a1a80512-bb20-4d6d-8e7f-ced390e999b1', 'version': 3}

Moar transaction signing.md

Transaction signing, continued

JS

Verification of addresses

Too short:

> account.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1, "to":"0xff"},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
Error: Provided address "0xff" is invalid, the capitalization checksum test failed, or its an indrect IBAN address which can't be converted.
    at Object.inputAddressFormatter (/node_modules/web3-core-helpers/src/formatters.js:383:11)
    at signed (/node_modules/web3-eth-accounts/src/index.js:143:44)
    at Accounts.signTransaction (/node_modules/web3-eth-accounts/src/index.js:189:16)
    at repl:1:9

Invalid checksum:

> account.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1, "to":"0x39e58Ee5C267683aC7694E5BFdb69Eb868e09985"},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
Error: Provided address "0x39e58Ee5C267683aC7694E5BFdb69Eb868e09985" is invalid, the capitalization checksum test failed, or its an indrect IBAN address which can't be converted.
    at Object.inputAddressFormatter (/node_modules/web3-core-helpers/src/formatters.js:383:11)

Other verifications

Allows -1 as nonce:

> account.signTransaction({'nonce':-1, "value":0,"gas":21000,"gasPrice":0, "chainId":1},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ messageHash: '0x71488a00b0ac6c3f13f230787de8e2cde99e018a9037209ae89f81baf7e7f471',
  v: '0x26',
  r: '0xc0d893f83137b9c82d3b1a602edda9a2d9c92f6714f421f96be08fa200a0b7d8',
  s: '0x3fd99522cf6ea496c7bc4ae2de4807e1d9fb136919462bbfb99012979664ff34',
  rawTransaction: '0xf84c81x18082520880808026a0c0d893f83137b9c82d3b1a602edda9a2d9c92f6714f421f96be08fa200a0b7d8a03fd99522cf6ea496c7bc4ae2de4807e1d9fb136919462bbfb99012979664ff34' }

But in fact, it produces an erroneous rawTransaction: 0xf84c81x1... with an x in the middle of the data payload.

Python

Verification of addresses

Too short

>>> la.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"to":"0x00"})
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rlp/sedes/lists.py", line 59, in serialize
    result.append(sedes.serialize(element))
  File "/usr/local/lib/python3.6/site-packages/rlp/sedes/binary.py", line 46, in serialize
    raise SerializationError('Object has invalid length', serial)
rlp.exceptions.SerializationError: Object has invalid length

Invalid checksum is not checked:

>>> la.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"to":"0x39e58Ee5C267683aC7694E5BFdb69Eb868e09985".lower()})
AttributeDict({'rawTransaction': HexBytes('0xf85f80808252089439e58ee5c267683ac7694e5bfdb69eb868e09985808025a0cff43abbdc1ed34c7b6f936bdd0a1edad1a202df04e3a5b2b19c5437e89604cca03871f2c8badd9f2c1a916cf96cf54d2923e92f06a34510b80ed492d968541f0b'), 'hash': HexBytes('0xb558662444bb2a38c7c874b84baf76240b653198265f5182c10a0c8e6b8d3376'), 'r': 94060275706521593774841937292426879477319421971470747935013613432719092876492, 's': 25530848873267388081850177786638222967045949825289717041283658277571286802187, 'v': 37})

Other verifications

Does not allow negative nonce

>>> la.signTransaction({'nonce':-1, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"to":""})
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rlp/sedes/lists.py", line 59, in serialize
    result.append(sedes.serialize(element))
  File "/usr/local/lib/python3.6/site-packages/rlp/sedes/big_endian_int.py", line 22, in serialize
    raise SerializationError('Cannot serialize negative integers', obj)
rlp.exceptions.SerializationError: Cannot serialize negative integers

Transaction signing.md

Sign transaction

Neither JS nor PY cares about extra fields on tranasction objects, see ethereum/go-ethereum#15628 for a scenario where this can lead to problems. Particularly if there is a field input which is used in place of data. Parity does not allow extra fields, Geth explicitly checks if input is present, and of so, uses that as data. If both are present, an error is prodiced.

Javascript

// main chainid
> account.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ messageHash: '0xc1cdfb8128c3ca12f0448617d20222313b052291d2f2a1b1cc492b3994ebfd78',
  v: '0x26',
  r: '0x7b1dcad9fe49d3f88cd3c8bd76d8b9e6ef657da10310446f14dcb8a017aa79ea',
  s: '0x35908414bccc859ecd4691c0c37dea7d72c71d44f4125051f12f9d54436fc753',
  rawTransaction: '0xf84b808082520880808026a07b1dcad9fe49d3f88cd3c8bd76d8b9e6ef657da10310446f14dcb8a017aa79eaa035908414bccc859ecd4691c0c37dea7d72c71d44f4125051f12f9d54436fc753' }
> 

# V is set to  `0x26` == 38

//test chainid
account.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0,"chainId":14},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ messageHash: '0xb28085cdc25f6ecd1a90f511f041d0b93d58af430f337da9072113dd0a5a7148',
  v: '0x40',
  r: '0x7a90a62179d39a7241af2212908dce6c89bc6f1da6bef93716d44e9dcc3bdace',
  s: '0x91433fad2893a8f7d18f4daf2b0f8370f09d681e62d30016f995af9c4ca0a77',
  rawTransaction: '0xf84b808082520880808040a07a90a62179d39a7241af2212908dce6c89bc6f1da6bef93716d44e9dcc3bdacea0091433fad2893a8f7d18f4daf2b0f8370f09d681e62d30016f995af9c4ca0a77' }

#JS uses `data`
> account.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"data":"0xff"},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ messageHash: '0xa807646cb0d1811a3f4a8055d1137e12e0b21b0b1608e27ef54c4962406ab5e3',
  v: '0x26',
  r: '0x7d4824225b3ceff1bac6d2ad5a2fb71cdeb5681936b67ff33321e49dcf4fe94f',
  s: '0x6c9835d05bed94b33f96b3eaba2e3f51666fa2fa5c3ef5fc6a199444a6d85605',
  rawTransaction: '0xf84c8080825208808081ff26a07d4824225b3ceff1bac6d2ad5a2fb71cdeb5681936b67ff33321e49dcf4fe94fa06c9835d05bed94b33f96b3eaba2e3f51666fa2fa5c3ef5fc6a199444a6d85605' }

# JS silently ignores `input`

> account.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"input":"0xff"},"0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ messageHash: '0xc1cdfb8128c3ca12f0448617d20222313b052291d2f2a1b1cc492b3994ebfd78',
  v: '0x26',
  r: '0x7b1dcad9fe49d3f88cd3c8bd76d8b9e6ef657da10310446f14dcb8a017aa79ea',
  s: '0x35908414bccc859ecd4691c0c37dea7d72c71d44f4125051f12f9d54436fc753',
  rawTransaction: '0xf84b808082520880808026a07b1dcad9fe49d3f88cd3c8bd76d8b9e6ef657da10310446f14dcb8a017aa79eaa035908414bccc859ecd4691c0c37dea7d72c71d44f4125051f12f9d54436fc753' }

Python web3 api

# Requires the to-field to be set:

>>> la.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/src/app/web3/utils/signing.py", line 173, in signTransaction
    return self._publicapi.signTransaction(transaction_dict, self.privateKey)
  File "/usr/src/app/web3/utils/decorators.py", line 14, in _wrapper
    return self.method(obj, *args, **kwargs)
  File "/usr/src/app/web3/account.py", line 167, in signTransaction
    ) = sign_transaction_dict(account._key_obj, transaction_dict)
  File "/usr/src/app/web3/utils/signing.py", line 28, in sign_transaction_dict
    unsigned_transaction = serializable_unsigned_transaction_from_dict(web3, transaction_dict)
  File "/usr/src/app/web3/utils/transactions.py", line 51, in serializable_unsigned_transaction_from_dict
    return serializer.from_dict(filled_transaction)
  File "/usr/src/app/web3/utils/encoding.py", line 276, in from_dict
    return cls(**field_dict)
  File "/usr/local/lib/python3.6/site-packages/rlp/sedes/lists.py", line 181, in __init__
    raise TypeError('Not all fields initialized')
TypeError: Not all fields initialized
>>> 

# V is set to  `0x26` == 38
>>> la.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"to":""})
AttributeDict({'rawTransaction': HexBytes('0xf84b808082520880808026a07b1dcad9fe49d3f88cd3c8bd76d8b9e6ef657da10310446f14dcb8a017aa79eaa035908414bccc859ecd4691c0c37dea7d72c71d44f4125051f12f9d54436fc753'), 'hash': HexBytes('0xdbe65aac443ad5f749939739ed752ab0280a0fd6a1ea0a00077a884957397ae8'), 'r': 55687118970470510547136965950997387526335795195977147650950269035342895675882, 's': 24227918541840761910366350430158278426875887423795240935429470879338619455315, 'v': 38})

# Py uses `data`
>>> la.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"to":"", "data":"0xff"})
AttributeDict({'rawTransaction': HexBytes('0xf84c8080825208808081ff26a07d4824225b3ceff1bac6d2ad5a2fb71cdeb5681936b67ff33321e49dcf4fe94fa06c9835d05bed94b33f96b3eaba2e3f51666fa2fa5c3ef5fc6a199444a6d85605'), 'hash': HexBytes('0x768b5c26b02591909a3047433e258065243d227c217bee694637361676d7b1ab'), 'r': 56666568450687500047565986506343001281974936404822881852058783468469162600783, 's': 49118719810745524668480418231531582400137060782954719179938068568634960664069, 'v': 38})

# Py silently ignores `input`
>>> la.signTransaction({'nonce':0, "value":0,"gas":21000,"gasPrice":0, "chainId":1,"to":"", "input":"0xff"})
AttributeDict({'rawTransaction': HexBytes('0xf84b808082520880808026a07b1dcad9fe49d3f88cd3c8bd76d8b9e6ef657da10310446f14dcb8a017aa79eaa035908414bccc859ecd4691c0c37dea7d72c71d44f4125051f12f9d54436fc753'), 'hash': HexBytes('0xdbe65aac443ad5f749939739ed752ab0280a0fd6a1ea0a00077a884957397ae8'), 'r': 55687118970470510547136965950997387526335795195977147650950269035342895675882, 's': 24227918541840761910366350430158278426875887423795240935429470879338619455315, 'v': 38})

python will use scrypt if it's installed, but it falls back to pbkdf2 when it's not.

What are the parameters used for scrypt?

But wait. How could it decrypt the js-encrypted (scrypt) keystore if scrypt wasn't installed?

Hash prefix

we have an equivalent to hashMessage which does the prepending laying around somewhere.

Yes, Account.sign() does prepending.

Address checksum

Invalid checksum is not checked:

Oops, I will go add a test and fix it.

Text encoding

We should take another look at how unicode characters are treated.

• In its current state, eth_utils.keccak() should only be passed a bytes type. Let's test a character that encodes differently in latin-1 and UTF-8, and another that has no encoding in latin-1
• It looks like there is an unexplained difference in message hashes for a unicode message.

Keccak

Python eth_utils

In [1]: import eth_utils

In [2]: eth_utils.keccak('ö')
Out[2]: b'\x9b\x13\x1f.8\xa9\x7f\xf2\xbcG7>\x06\xa0\x7f\x8c\x13\xdf+\t\xe9\xff\x9d\x7f\xa8\xb09o2\x96e\x9a'

In [3]: eth_utils.keccak('☢')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2622' in position 0: ordinal not in range(256)

In [4]: eth_utils.keccak('ö'.encode('utf-8'))
Out[4]: b'\xd6y\x9c\x9f\x9aW\x89(]\xe4\x9d\x9e\x05}e>\xdf\xc7\x94\x8bD\x01/Eb\xda\tO\x0b\xa3\xeeJ'

In [5]: eth_utils.keccak('☢'.encode('utf-8'))
Out[5]: b'\x85\xe8\x07"\xeb\x93\r\xe9;\xcc\xa8{\xa5\xdf\xda\x89\n\xa12\x95\xae\xad.\xec\xc9\x0b\xb2\xd9z\x14\x93\x16'

web3.js

> web3.sha3('ö')
"0xd6799c9f9a5789285de49d9e057d653edfc7948b44012f4562da094f0ba3ee4a"
> web3.sha3('☢')
"0x85e80722eb930de93bcca87ba5dfda890aa13295aead2eecc90bb2d97a149316"

Message Signing

Python web3.Account

In [1]: from web3 import Account

In [2]: Account.sign(message_text='Some data', private_key='0x4c0883a69102937d6231471b5dbb6204fe5129617082792ae468d01a3f362318')
Out[2]: 
AttributeDict({'message': HexBytes('0x536f6d652064617461'),
 'messageHash': HexBytes('0x1da44b586eb0729ff70a73c326926f6ed5a25f5b056e7f47fbc6e58d86871655'),
 'r': 83713930994764734002432606962255364472443135907807238282514898577139886061053,  # hex-encoded: 0xb91467e570a6466aa9e9876cbcd013baba02900b8979d43fe208a4a4f339f5fd
 's': 43435997768575461196683613590576722655951133545204789519877940758262837256233,  # hex-encoded: 0x6007e74cd82e037b800186422fc2da167c747ef045e5d18a5f5d4300f8e1a029
 'signature': HexBytes('0xb91467e570a6466aa9e9876cbcd013baba02900b8979d43fe208a4a4f339f5fd6007e74cd82e037b800186422fc2da167c747ef045e5d18a5f5d4300f8e1a0291c'),
 'v': 28,  # hex-encoded: 0x1c
})

In [3]: Account.sign(message_text='ö', private_key="0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
Out[3]: 
AttributeDict({'message': HexBytes('0xc3b6'),
 'messageHash': HexBytes('0xb47d5d86fe7a550fdf31a5d921cdb31b7929e50ba357a02de2bc65641c5694ea'),
 'r': 47105596547189414011492542599151821567086936402617731047376442508702236285701,  # hex-encoded: 0x6824d2a4724414dd0074e1eb0b451509c140e089f9adfb2f5727a72214101b05
 's': 32389960013298448456226415041162031803593056414264733415065824281391218193028,  # hex-encoded: 0x479c115504b9f3561c2c437d4fb1f7a10d87359687896205ce4fd8b9014a7284
 'signature': HexBytes('0x6824d2a4724414dd0074e1eb0b451509c140e089f9adfb2f5727a72214101b05479c115504b9f3561c2c437d4fb1f7a10d87359687896205ce4fd8b9014a72841b'),
 'v': 27,  # hex-encoded: 0x1b
})

In [4]: Account.sign(message_text='☢', private_key="0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
Out[4]: 
AttributeDict({'message': HexBytes('0xe298a2'),
 'messageHash': HexBytes('0x884698b13df21cce14a0feef2c7109aa5c0909cb7ff70b35e72da6be0b6de77d'),
 'r': 40608581733999862361551872008024969063058550762674618727283784812387661315659,  # hex-encoded: 0x59c7a48b626b6bb8169cf86cbc57976d5d66a06fa2895b601fc61c1a9bccfe4b
 's': 45909177089447279945965695051015775713173772161173808215973817421953530722904,  # hex-encoded: 0x657fac6454ee28927d2e39eb2427d014fa0d5fb27ef963a2f51910e7f341fa58
 'signature': HexBytes('0x59c7a48b626b6bb8169cf86cbc57976d5d66a06fa2895b601fc61c1a9bccfe4b657fac6454ee28927d2e39eb2427d014fa0d5fb27ef963a2f51910e7f341fa581b'),
 'v': 27,  # hex-encoded: 0x1c
})

web3js, 1.0 branch

> Account.sign('Some data', '0x4c0883a69102937d6231471b5dbb6204fe5129617082792ae468d01a3f362318')
{ message: 'Some data',
  messageHash: '0x1da44b586eb0729ff70a73c326926f6ed5a25f5b056e7f47fbc6e58d86871655',
  v: '0x1c',
  r: '0xb91467e570a6466aa9e9876cbcd013baba02900b8979d43fe208a4a4f339f5fd',
  s: '0x6007e74cd82e037b800186422fc2da167c747ef045e5d18a5f5d4300f8e1a029',
  signature: '0xb91467e570a6466aa9e9876cbcd013baba02900b8979d43fe208a4a4f339f5fd6007e74cd82e037b800186422fc2da167c747ef045e5d18a5f5d4300f8e1a0291c' }

> accounts.sign('ö', "0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ message: 'ö',
  messageHash: '0x2ed1ff902ecf61c9b2735bfa706fec8c0cc5cd48bf9aeaeb98a3f50861b92ca9',
  v: '0x1b',
  r: '0x5712358e8dcb4d34e9e1f0508f8a502f3551d26b290ae80f245be455e49193f4',
  s: '0x4554d57d812fe9b9c44ab0490fb8bfc5812b600732fc210972e256ced5292cc3',
  signature: '0x5712358e8dcb4d34e9e1f0508f8a502f3551d26b290ae80f245be455e49193f44554d57d812fe9b9c44ab0490fb8bfc5812b600732fc210972e256ced5292cc31b' }

> accounts.sign('☢', "0x5384837ece911af85b8b2f2a0e6fe73f5f2add257fc25e1efbeebd4ee72226b7")
{ message: '☢',
  messageHash: '0xd899d2d75a705648731494feff1738a70c9f0a9f91e86b61ff11c340217bd45c',
  v: '0x1c',
  r: '0x1558e9acb5941d3bd832bbb892cc080ccc97b49782e2b34db92e003be892246a',
  s: '0x1620031241d22d1e2fd04b40a7b3bae71f9946d6e7b88d07e81927e292fc91ac',
  signature: '0x1558e9acb5941d3bd832bbb892cc080ccc97b49782e2b34db92e003be892246a1620031241d22d1e2fd04b40a7b3bae71f9946d6e7b88d07e81927e292fc91ac1c' }

Todo

• Convert text using utf-8 instead of latin-1 in eth_utils.keccak (and a major version bump, I suppose), or at least reject str input
• Verify address checksum in to field when signing transaction
• Investigate why message hashes are different in js and python. First guess: one of the libraries calculates message length in unicode character points instead of number of bytes. Since the message length is embedded in the prefix, it would change the message and its hash.
• release eth-utils b2
• upgrade eth-utils in eth-account