Created
July 5, 2016 16:17
-
-
Save looprock/687b0678473849ae14f4f4307ac2395d to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We pulled a host from the pool for upgrades, and in the process seems to have gotten a virtual router in an odd state. It's showing as destroyed in the UI, but cloudmonkey says it's still expunging. | |
This host has been completely rebuild including completely redisked. On the management node I found: | |
mysql> select * from op_ha_work | |
-> ; | |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+ | |
| id | instance_id | type | vm_type | state | mgmt_server_id | host_id | created | tried | taken | step | time_to_try | updated | | |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+ | |
| 1 | 13 | Migration | DomainRouter | Expunging | NULL | 24 | 2016-07-01 14:34:17 | 0 | NULL | Migrating | 1433332034 | 205 | | |
| 4 | 78 | Migration | DomainRouter | Destroyed | NULL | 24 | 2016-07-01 14:34:17 | 0 | NULL | Migrating | 1433332092 | 68 | | |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+ | |
I removed those entries, but when the hosts persist. Via cloudmonkey it shows expunging: | |
{ | |
"count": 1, | |
"router": [ | |
{ | |
"account": "engineering", | |
"created": "2014-09-05T03:56:07+0200", | |
"dns1": "172.16.8.46", | |
"dns2": "172.16.8.47", | |
"domain": "engineering", | |
"domainid": "1da498ba-5646-4cc3-a704-a20ebe12f518", | |
"id": "dc48a402-41d8-4e93-b441-4b34eb83a4c8", | |
"isredundantrouter": true, | |
"name": "r-78-VM", | |
"nic": [], | |
"podid": "f53afa8d-51ff-484d-9a88-52e979aeb688", | |
"redundantstate": "UNKNOWN", | |
"requiresupgrade": false, | |
"role": "VIRTUAL_ROUTER", | |
"serviceofferingid": "ed6b13d0-3e74-4aa5-a6b7-a5d2ac6c4a6c", | |
"serviceofferingname": "System Offering For Software Router", | |
"state": "Expunging", | |
"templateid": "bb3f7e4e-d7f6-4a72-a752-12c3221e43e9", | |
"version": "4.4.1", | |
"zoneid": "3467ff63-b582-4ace-9fda-8d5851bd8753", | |
"zonename": "Oakland" | |
} | |
] | |
} | |
If I try to destroy the host from the api I get: | |
Async job cf08d7fa-1609-4d0e-b33c-63cc38f7e897 failed | |
Error 530, Unable to locate datastore with id 18 | |
{ | |
"accountid": "e3389462-6020-425a-9b9e-57141d58e1ab", | |
"cmd": "org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd", | |
"created": "2016-07-05T17:23:53+0200", | |
"jobid": "cf08d7fa-1609-4d0e-b33c-63cc38f7e897", | |
"jobprocstatus": 0, | |
"jobresult": { | |
"errorcode": 530, | |
"errortext": "Unable to locate datastore with id 18" | |
}, | |
"jobresultcode": 530, | |
"jobresulttype": "object", | |
"jobstatus": 2, | |
"userid": "xxx" | |
} | |
I'm guessing I need to remove all references for the routers from the database. Does anyone know what table(s) that's stored in? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment