Coinos Incident Report
On Tuesday January 12 from the hours of 6:25pm to 8:20pm PST, an attacker exploited two flaws in coinos' application server code that enabled them to withdraw funds from the server's lightning node that they were not meant to be able to access, to the tune of 0.24551673 BTC.
Both flaws were due to improperly implemented database transaction locks resulting in improper handling of simultaneous payment requests.
The first flaw allowed them to send a self-payment to their own username and see their balance increase by whatever amount was sent. This happened because during a self-payment, two requests would be generated, one debiting the account and the other incrementing it for the same amount, but the request that incremented the account was reading the balance of the account before the debiting request had been completed.
I happened to notice in the server logs an account sending itself payments and doubling their value each time. An emergency patch was comitted and deployed at 6:51pm to fix the problem: https://github.com/asoltys/coinos-server/commit/8afa3af7f790bd1d78b2105e7541eed462fbb825
This did appear to solve the issue with self payments and the attack subsided for about 20 minutes but at 7:10pm the attacker began to exploit a second flaw that also involved issuing two or more requests to the server in rapid succession. In this case, they were able to create a redeemable voucher payment and redeem it with another coinos account under their control while also sending a lightning payment for the same amount to an external lightning wallet.
Again the server did not properly read their debited account balance from the voucher payment when the lightning withdrawal was processed, so when the code checked their account balance before authorizing the lightning payment it still showed that they had funds in their account even though they had just been transferred to another coinos account. They performed this same procedure several dozen times, sending an internal payment to another account under their control and a lightning payment of typically 400000-500000 satoshis to an external lightning wallet each time.
Here is an excerpt of the server logs during the incident: https://gist.github.com/5c6255d9ab858df8cf521ae577008c32
And here is an excerpt of the relevant payment transactions from the database: https://gist.github.com/0d25087504af33669d749aea38edb7d6
In the aftermath of this incident, the coinos server was turned off for a period of about 24 hours while I took time to investigate. As of writing I've not yet been able to reliably reproduce the second exploit in my local development environment so I've disabled the internal transfer and voucher features altogether for now until I have more time to ensure that they are properly secured. I've also disabled the ability to place orders on the exchange or fund liquid faucets because of a report by an anonymous Telegram user that tipped me off to potential vulnerabilities with them as well.
I've deployed a new feature that allows me to lock user accounts and I've locked all the ones involved in the incident. Lastly I've put in place a crude script to monitor the hot wallet balance and shut the server down if the balance goes below a certain threshold to give me time to manually double check any large withdrawal activity and re-enable the server if it appears to be legitimate.
My apologies to anyone who was adversely affected by this incident and I thank you for your patience and understanding.
Adam Soltys