The following records cover the different data inputs available at the time that the erroneous trades ocurred.
The bot failed to take down all of the buy walls before shifting the sell walls to a new NBT/PPC exchange rate.
Ok, I figured out how it happened, though I'm still not clear why it happened, but here's the timeline of how the trades occurred and where the breakdown happened.
Starting at a known point in time, Kiara's CCEDK transaction records show the creation of the two walls, each with two orders in them for approximately the same amount of funds. The pricing data used for these orders was:
Fri Oct 03 19:58:43 GMT 2014 btce ppc 0.797 USD 1.25721455 1.25219573 { feed : btce - price : 0.797} { feed : coinmarketcap_ne - price : 0.816319} { feed : coinmarketcap_ne - price : 0.816319} { feed : coinmarketcap_no - price : 0.81885}
The bot creates and submits the orders...
StrategySecondaryPegTask,reInitiateOrders,Oct_03_19_59_15
Strategy : Submit order : sell 217235.6497 nbt @ 1.25721455 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_21
Strategy : Sell Response1 = 30150,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_21
Strategy : Submit order : sell 217235.64978751 nbt @ 1.25721455 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_21
Strategy : Sell Response2 = 30151,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_22
Strategy : Submit order : buy 19908.60494309464 nbt @ 1.25219573 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_22
Strategy : Buy Response1 = 30152,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_23
Strategy : Submit order : buy 19908.604997686743 nbt @ 1.25219573 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_19_59_23
Strategy : Buy Response2 = 30153,WARNING,com.nubits.nubot.tasks.
...which appear on CCEDK:
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65894 | Create order | 217235.6497 | NBT | 03.10.2014 19:59 |
| 65895 | Create order | 217235.64978751 | NBT | 03.10.2014 19:59 |
| 65896 | Create order | 24929.47009999 | PPC | 03.10.2014 19:59 |
| 65897 | Create order | 24929.47016836 | PPC | 03.10.2014 19:59 |
This gives us a starting point to work from.
SELL:
217235.6497 nbt @ 1.25721455 ppc
217235.64978751 nbt @ 1.25721455 ppc
BUY:
19908.60494309464 nbt @ 1.25219573 ppc (Total: 24929.47009999 ppc)
19908.604997686743 nbt @ 1.25219573 ppc (Total: 24929.47016836 ppc)
At 20:05:28, the bot receives a notification from the price tracking function that a shift in the walls was needed:
We should move the walls now!,SEVERE,com.nubits.nubot.tasks.PriceMonitorTriggerTask,verifyPegPrices,Oct_03_20_05_28
Sell Price (sell-side custodians) 1.25879397
Buy Price 1.25376884,INFO,com.nubits.nubot.tasks.PriceMonitorTriggerTask,computeNewPrices,Oct_03_20_05_28
Strategy received a price change notification.,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,notifyPriceChanged,Oct_03_20_05_28
Fri Oct 03 20:05:28 GMT 2014 btce ppc 0.796 USD 1.25879397 1.25376884 { feed : btce - price : 0.796} { feed : coinmarketcap_ne - price : 0.819271} { feed : coinmarketcap_ne - price : 0.819271} { feed : coinmarketcap_no - price : 0.816319}
,WARNING,com.nubits.nubot.tasks.PriceMonitorTriggerTask,computeNewPrices,Oct_03_20_05_28
Order 30153 deleted succesfully,WARNING,com.nubits.nubot.trading.wrappers.CcedkWrapper,clearOrders,Oct_03_20_05_42
Order 30152 deleted succesfully,WARNING,com.nubits.nubot.trading.wrappers.CcedkWrapper,clearOrders,Oct_03_20_05_43
Order 30151 deleted succesfully,WARNING,com.nubits.nubot.trading.wrappers.CcedkWrapper,clearOrders,Oct_03_20_05_43
Order 30150 deleted succesfully,WARNING,com.nubits.nubot.trading.wrappers.CcedkWrapper,clearOrders,Oct_03_20_05_44
Clear all orders request succesfully,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,reInitiateOrders,Oct_03_20_05_45
Strategy : Submit order : sell 217235.6497 nbt @ 1.25879397 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_51
Strategy : Sell Response1 = 30177,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_52
Strategy : Submit order : sell 217235.64978751 nbt @ 1.25879397 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_52
Strategy : Sell Response2 = 30178,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_53
Strategy : Submit order : buy 19883.62551744387 nbt @ 1.25376884 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_53
Strategy : Buy Response1 = 30179,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_53
Strategy : Submit order : buy 19883.625571967477 nbt @ 1.25376884 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_05_53
Strategy : Buy Response2 = 30180,WARNING,com.nubits.nubot.tasks.
This was reflected in the orders that appeared on CCEDK. First the previous walls were taken down...
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65946 | Delete order | 24929.47016836 | PPC | 03.10.2014 20:05 |
| 65947 | Delete order | 24929.47009999 | PPC | 03.10.2014 20:05 |
| 65948 | Delete order | 217235.64978751 | NBT | 03.10.2014 20:05 |
| 65949 | Delete order | 217235.6497 | NBT | 03.10.2014 20:05 |
...and then new orders were placed:
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65950 | Create order | 217235.6497 | NBT | 03.10.2014 20:05 |
| 65951 | Create order | 217235.64978751 | NBT | 03.10.2014 20:05 |
| 65952 | Create order | 24929.4701 | PPC | 03.10.2014 20:05 |
| 65953 | Create order | 24929.47016836 | PPC | 03.10.2014 20:05 |
After these orders are placed, the walls look like this:
SELL:
217235.6497 nbt @ 1.25879397 ppc
217235.64978751 nbt @ 1.25879397 ppc
BUY:
19883.62551744387 nbt @ 1.25376884 ppc (Total: 24929.4701 ppc)
19883.625571967477 nbt @ 1.25376884 ppc (Total: 24929.47016836 ppc)
At 20:07 someone completes a small trade and buys 173 NBT from one of Kiara's sell orders. This trade is captured in CCEDK's transaction logs:
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65966 | Trade | 173.33986622 | NBT | 03.10.2014 20:07 |
| 65967 | Fee | 0.34737448 | NBT | 03.10.2014 20:07 |
Immediately the bot notices that the current sell wall orders are no longer "even" in amount, and initiates a order placement to normalize the two. Order 30177 (which corresponds to CCEDK order # 65950) is deleted.
Sellside : Taking down smaller order to aggregate it with new balance,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,sellSide,Oct_03_20_07_42
Order 30177 delete request submitted,WARNING,com.nubits.nubot.utils.TradeUtils,takeDownAndWait,Oct_03_20_07_42
From this point forward, operations do not go as planned. As of 20:07, the state of the walls and their pricing looks like this:
SELL:
217235.64978751 nbt @ 1.25879397 ppc
BUY:
19883.62551744387 nbt @ 1.25376884 ppc (Total: 24929.4701 ppc)
19883.625571967477 nbt @ 1.25376884 ppc (Total: 24929.47016836 ppc)
One second after taking down order #65950, the bot receives this notification:
We should move the walls now!,SEVERE,com.nubits.nubot.tasks.PriceMonitorTriggerTask,verifyPegPrices,Oct_03_20_07_43
Sell Price (sell-side custodians) 1.2525
Buy Price 1.2475,INFO,com.nubits.nubot.tasks.PriceMonitorTriggerTask,computeNewPrices,Oct_03_20_07_43
Strategy received a price change notification.,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,notifyPriceChanged,Oct_03_20_07_43
Fri Oct 03 20:07:43 EDT 2014 btce ppc 0.8 USD 1.2525 1.2475 { feed : btce - price : 0.8} { feed : coinmarketcap_ne - price : 0.815916} { feed : coinmarketcap_ne - price : 0.815916} { feed : coinmarketcap_no - price : 0.819271}
,WARNING,com.nubits.nubot.tasks.PriceMonitorTriggerTask,computeNewPrices,Oct_03_20_07_43
The first action that it tries is to clear the existing walls...but runs into a problem when it cannot locate the order that it just submitted the cancellation request for a second before:
The order 30177 does not exist,SEVERE,com.nubits.nubot.trading.wrappers.CcedkWrapper,isOrderActive,Oct_03_20_07_48
Quickly followed by a placement of a SELL order that is less than the previous buy order:
Strategy : Submit order : sell 217408.98956622 nbt @ 1.2525 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,sellSide,Oct_03_20_07_48
Strategy : Sell Response = 30187,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,sellSide,Oct_03_20_07_49
The ppc balance is not zero (49491.75683218 ). If the balance represent proceedings from a sale the bot will notice. On the other hand If you keep seying this message repeatedly over and over you should restart the bot. ,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,recount,Oct_03_20_08_11
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65973 | Create order | 217408.98956622 | NBT | 03.10.2014 20:07 |
That leaves the walls in this state, for a very short time...
SELL:
217235.64978751 nbt @ 1.25879397 ppc
217408.98956622 nbt @ 1.2525 ppc
BUY:
19883.62551744387 nbt @ 1.25376884 ppc (Total: 24929.4701 ppc)
19883.625571967477 nbt @ 1.25376884 ppc (Total: 24929.47016836 ppc)
"Oh, look, there are buy orders that can be fulfilled by the new sell order that I placed," says NuBot, "I'll just take care of that now! Yeah, happy buyers!.."
Buy Wall #1 is consumed. PPC is traded for NBT from one side of the account to the other:
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65975 | Trade | 24686.69772075 | PPC | 03.10.2014 20:07 |
| 65976 | Trade | 19670.51845461 | NBT | 03.10.2014 20:07 |
| 65977 | Fee | 39.41987666 | NBT | 03.10.2014 20:07 |
I'm not totally clear what this next transaction is, and may be an issue with the exchange, I have to look into it more. If anyone has an idea, let me know.
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65978 | Trade | 24 637.32432524 | PPC | 03.10.2014 20:07 |
| 65979 | Fee | 49.37339551 | PPC | 03.10.2014 20:07 |
Buy Wall #2 is consumed. Same dual-side transfer so it shows up as two trades and one fee:
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65982 | Trade | 24 904.24093875 | PPC | 03.10.2014 20:07 |
| 65983 | Trade | 19 843.85826642 | NBT | 03.10.2014 20:07 |
| 65984 | Fee | 39.76725102 | NBT | 03.10.2014 20:07 |
And another anomolous PPC trade:
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65985 | Trade | 24 854.43245685 | PPC | 03.10.2014 20:07 |
| 65986 | Fee | 49.8084819 | PPC | 03.10.2014 20:07 |
At 20:08, the bot notices that there's a mismatch in the PPC balance. It then deletes the remaining trades that were in the walls:
The ppc balance is not zero (49491.75683218 ). If the balance represent proceedings from a sale the bot will notice. On the other hand If you keep seying this message repeatedly over and over you should restart the bot. ,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,recount,Oct_03_20_08_11
Order 30187 deleted succesfully,WARNING,com.nubits.nubot.trading.wrappers.CcedkWrapper,clearOrders,Oct_03_20_08_12
Order 30178 deleted succesfully,WARNING,com.nubits.nubot.trading.wrappers.CcedkWrapper,clearOrders,Oct_03_20_08_13
Clear all orders request succesfully,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,reInitiateOrders,Oct_03_20_08_13
Finally, is able to get back on track and resubmits a correct number of orders:
Strategy : Submit order : sell 217282.7261 nbt @ 1.2525 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_20
Strategy : Sell Response1 = 30194,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_20
Strategy : Submit order : sell 217282.72612605002 nbt @ 1.2525 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_20
Strategy : Sell Response2 = 30195,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_21
Strategy : Submit order : buy 19836.375470941883 nbt @ 1.2475 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_21
Strategy : Buy Response1 = 30196,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_22
Strategy : Submit order : buy 19836.375496737473 nbt @ 1.2475 ppc,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_22
Strategy : Buy Response2 = 30197,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,placeInitialWalls,Oct_03_20_08_22
| Order | Type | Amount | Currency | Timestamp |
|---|---|---|---|---|
| 65994 | Create order | 217282.7261 | NBT | 03.10.2014 20:08 |
| 65995 | Create order | 217282.72612605 | NBT | 03.10.2014 20:08 |
| 65996 | Create order | 24745.8784 | PPC | 03.10.2014 20:08 |
| 65997 | Create order | 24745.87843218 | PPC | 03.10.2014 20:08 |
SELL:
217282.7261 nbt @ 1.2525 ppc
217282.72612605002 nbt @ 1.2525 ppc
BUY:
19836.375470941883 nbt @ 1.2475 (Total: 24745.8784 ppc)
19836.375496737473 nbt @ 1.2475 ppc (Total: 24745.87843218 ppc)
But all is not well, because the bot detects that things aren't correct (but they appear to be...):
Detected a number of active orders not in line with strategy. Will try to aggregate soon,SEVERE,com.nubits.nubot.tasks.StrategySecondaryPegTask,run,Oct_03_20_08_26
Apparently it then figures out that things really are OK.
No need to init new orders since current orders seems correct,WARNING,com.nubits.nubot.tasks.StrategySecondaryPegTask,run,Oct_03_20_08_41
This collision is not the only one that I've noticed while reviewing Kiara's logs that she sent me after this occurred yesterday. I'm trying to look for a common pattern of execution that leads to the wall shifting without first being cleared fully. It will take me a bit more time to review the records to see if I can reliably identify the occurances so she can apply a recalculation to this period's sales to drop any of these "internal trades".
Thankfully the end impact of these failures only affects the volume shown on the exchange, and the cost is minimal (the exchange commission fees on the trades). This is something that needs to be fixed though, because I'm concerned that there are scenarios that could exist that result in arbitrage opportunties once the trading volume increases.
Please review my findings and if you see anything that I missed, or if you feel that any of my conclusions were draw incorrectly from the data.
Excerpt from
wall_shifts1412266898704.csv: