Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
We pulled a host from the pool for upgrades, and in the process seems to have gotten a virtual router in an odd state. It's showing as destroyed in the UI, but cloudmonkey says it's still expunging.
This host has been completely rebuild including completely redisked. On the management node I found:
mysql> select * from op_ha_work
-> ;
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+
| id | instance_id | type | vm_type | state | mgmt_server_id | host_id | created | tried | taken | step | time_to_try | updated |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+
| 1 | 13 | Migration | DomainRouter | Expunging | NULL | 24 | 2016-07-01 14:34:17 | 0 | NULL | Migrating | 1433332034 | 205 |
| 4 | 78 | Migration | DomainRouter | Destroyed | NULL | 24 | 2016-07-01 14:34:17 | 0 | NULL | Migrating | 1433332092 | 68 |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+
I removed those entries, but when the hosts persist. Via cloudmonkey it shows expunging:
{
"count": 1,
"router": [
{
"account": "engineering",
"created": "2014-09-05T03:56:07+0200",
"dns1": "172.16.8.46",
"dns2": "172.16.8.47",
"domain": "engineering",
"domainid": "1da498ba-5646-4cc3-a704-a20ebe12f518",
"id": "dc48a402-41d8-4e93-b441-4b34eb83a4c8",
"isredundantrouter": true,
"name": "r-78-VM",
"nic": [],
"podid": "f53afa8d-51ff-484d-9a88-52e979aeb688",
"redundantstate": "UNKNOWN",
"requiresupgrade": false,
"role": "VIRTUAL_ROUTER",
"serviceofferingid": "ed6b13d0-3e74-4aa5-a6b7-a5d2ac6c4a6c",
"serviceofferingname": "System Offering For Software Router",
"state": "Expunging",
"templateid": "bb3f7e4e-d7f6-4a72-a752-12c3221e43e9",
"version": "4.4.1",
"zoneid": "3467ff63-b582-4ace-9fda-8d5851bd8753",
"zonename": "Oakland"
}
]
}
If I try to destroy the host from the api I get:
Async job cf08d7fa-1609-4d0e-b33c-63cc38f7e897 failed
Error 530, Unable to locate datastore with id 18
{
"accountid": "e3389462-6020-425a-9b9e-57141d58e1ab",
"cmd": "org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd",
"created": "2016-07-05T17:23:53+0200",
"jobid": "cf08d7fa-1609-4d0e-b33c-63cc38f7e897",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Unable to locate datastore with id 18"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "xxx"
}
I'm guessing I need to remove all references for the routers from the database. Does anyone know what table(s) that's stored in?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.