Skip to content

Instantly share code, notes, and snippets.

@luqmana

luqmana/migration-protocol-flow.svg

Created October 29, 2021 06:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save luqmana/77b7db59e6b818642c8ba2af95c2a941 to your computer and use it in GitHub Desktop.
Save luqmana/77b7db59e6b818642c8ba2af95c2a941 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
migration-protocol-flow.svg
<?xml version="1.0" encoding="UTF-8" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="3001px" preserveAspectRatio="none" style="width:1374px;height:3001px;background:#FFFFFF;" version="1.1" viewBox="0 0 1374 3001" width="1374px" zoomAndPan="magnify"><defs/><g><text fill="#000000" font-family="sans-serif" font-size="18" lengthAdjust="spacing" textLength="281" x="543" y="29.4023">Propolis Migration Protocol Flow</text><rect fill="#FFFFFF" height="146.375" style="stroke:#23684A;stroke-width:1.0;" width="10" x="520" y="1314.127"/><rect fill="#FFFFFF" height="204.9961" style="stroke:#23684A;stroke-width:1.0;" width="10" x="520" y="1810.5625"/><rect fill="#FFFFFF" height="58.6211" style="stroke:#23684A;stroke-width:1.0;" width="10" x="1000" y="1314.127"/><rect fill="#FFFFFF" height="58.6211" style="stroke:#23684A;stroke-width:1.0;" width="10" x="1000" y="1810.5625"/><rect fill="#FFFFFF" height="29.3105" style="stroke:#23684A;stroke-width:1.0;" width="10" x="1000" y="1957.1152"/><rect fill="#FFFFFF" height="141.8633" style="stroke:#000000;stroke-width:2.0;" width="1065.5" x="10" y="406.1465"/><rect fill="#FFFFFF" height="178.4844" style="stroke:#000000;stroke-width:2.0;" width="610" x="10" y="873.1367"/><rect fill="#FFFFFF" height="192.9961" style="stroke:#000000;stroke-width:2.0;" width="621" x="454.5" y="1275.5059"/><rect fill="#FFFFFF" height="251.6172" style="stroke:#000000;stroke-width:2.0;" width="621" x="454.5" y="1771.9414"/><rect fill="#FFFFFF" height="172.4844" style="stroke:#000000;stroke-width:2.0;" width="1027.5" x="48" y="2341.3535"/><rect fill="#FFFFFF" height="143.1738" style="stroke:#000000;stroke-width:2.0;" width="635" x="48" y="2680.0352"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="94" x2="94" y1="88.1758" y2="2949.4512"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="192" x2="192" y1="1148.2305" y2="2949.4512"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="318" x2="318" y1="1206.207" y2="2949.4512"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="524.5" x2="524.5" y1="137.1641" y2="2949.4512"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="1004.5" x2="1004.5" y1="88.1758" y2="2949.4512"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="1180" x2="1180" y1="738.5508" y2="2949.4512"/><line style="stroke:#23684A;stroke-width:1.0;stroke-dasharray:5.0,5.0;" x1="1296" x2="1296" y1="796.5273" y2="2949.4512"/><rect fill="#F6FDFA" height="30.4883" style="stroke:#23684A;stroke-width:1.5;" width="73" x="58" y="56.6875"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="59" x="65" y="77.2227">Omicron</text><rect fill="#F6FDFA" height="30.4883" style="stroke:#23684A;stroke-width:1.5;" width="73" x="58" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="59" x="65" y="2968.9863">Omicron</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="102" x="141" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="68" x="158" y="2968.9863">OPTE Port</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="88" x="148" y="2985.4746">(Destination)</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="131" x="253" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="117" x="260" y="2968.9863">Crucible Upstairs</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="88" x="274.5" y="2985.4746">(Destination)</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="121" x="464.5" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="107" x="471.5" y="2968.9863">propolis-server</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="88" x="481" y="2985.4746">(Destination)</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="121" x="944.5" y="40.1992"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="107" x="951.5" y="60.7344">propolis-server</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="57" x="976.5" y="77.2227">(Source)</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="121" x="944.5" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="107" x="951.5" y="2968.9863">propolis-server</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="57" x="976.5" y="2985.4746">(Source)</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="82" x="1139" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="68" x="1146" y="2968.9863">OPTE Port</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="57" x="1151.5" y="2985.4746">(Source)</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="131" x="1231" y="2948.4512"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="117" x="1238" y="2968.9863">Crucible Upstairs</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="57" x="1268" y="2985.4746">(Source)</text><rect fill="#FFFFFF" height="146.375" style="stroke:#23684A;stroke-width:1.0;" width="10" x="520" y="1314.127"/><rect fill="#FFFFFF" height="204.9961" style="stroke:#23684A;stroke-width:1.0;" width="10" x="520" y="1810.5625"/><rect fill="#FFFFFF" height="58.6211" style="stroke:#23684A;stroke-width:1.0;" width="10" x="1000" y="1314.127"/><rect fill="#FFFFFF" height="58.6211" style="stroke:#23684A;stroke-width:1.0;" width="10" x="1000" y="1810.5625"/><rect fill="#FFFFFF" height="29.3105" style="stroke:#23684A;stroke-width:1.0;" width="10" x="1000" y="1957.1152"/><polygon fill="#23684A" points="452.5,130.7969,462.5,134.7969,452.5,138.7969,456.5,134.7969" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="94.5" x2="458.5" y1="134.7969" y2="134.7969"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="171" x="194" y="114.7441">Create new propolis-server</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="173" x="193" y="130.0547">instance for destination VM</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="121" x="464.5" y="98.1758"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="107" x="471.5" y="118.7109">propolis-server</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="88" x="481" y="135.1992">(Destination)</text><polygon fill="#23684A" points="513,218.6836,523,222.6836,513,226.6836,517,222.6836" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="94.5" x2="519" y1="222.6836" y2="222.6836"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="224" x="197.75" y="172.2192">PUT /instances/{dst}/migrate</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="144" x="237.75" y="187.3521">role = Destination</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="112" x="253.75" y="202.4849">src-addr = ...</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="112" x="253.75" y="217.6177">src-uuid = ...</text><path d="M7,173.1074 L7,213.1074 L89,213.1074 L89,183.1074 L79,173.1074 L7,173.1074 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M79,173.1074 L79,183.1074 L89,183.1074 L79,173.1074 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="44" x="13" y="190.6758">Initiate</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="61" x="13" y="205.9863">migration</text><path d="M27,235.6836 L27,275.6836 L161,275.6836 L161,245.6836 L151,235.6836 L27,235.6836 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M151,235.6836 L151,245.6836 L161,245.6836 L151,235.6836 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="113" x="33" y="253.252">Omicron waits for</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="94" x="33" y="268.5625">HTTP response</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="301.96"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="301.96" y2="301.96"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="304.96" y2="304.96"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="154" x="606.5" y="291.3047"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="136" x="612.5" y="307.873">Protocol Negotiation</text><polygon fill="#23684A" points="993,387.1465,1003,391.1465,993,395.1465,997,391.1465" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="525" x2="999" y1="391.1465" y2="391.1465"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="224" x="653" y="340.6821">PUT /instances/{src}/migrate</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="160" x="685" y="355.8149">Connection = Upgrade</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="456" x="537" y="370.9478">Upgrade = propolis-migrate-ron/m, propolis-migrate-bson/n</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="104" x="713" y="386.0806">role = Source</text><path d="M10,406.1465 L183,406.1465 L183,413.1465 L173,423.1465 L10,423.1465 L10,406.1465 " fill="#EEEEEE" style="stroke:#000000;stroke-width:1.0;"/><rect fill="none" height="141.8633" style="stroke:#000000;stroke-width:2.0;" width="1065.5" x="10" y="406.1465"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="128" x="25" y="419.7148">Negotiation Failure</text><polygon fill="#FF0000" points="536,440.7676,526,444.7676,536,448.7676,532,444.7676" style="stroke:#FF0000;stroke-width:1.0;"/><line style="stroke:#FF0000;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="530" x2="1004" y1="444.7676" y2="444.7676"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="56" x="576" y="439.7017">Upgrade</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="318" x="636" y="440.0254">header didn't list a supported version or encoding.</text><polygon fill="#FF0000" points="105.5,470.0781,95.5,474.0781,105.5,478.0781,101.5,474.0781" style="stroke:#FF0000;stroke-width:1.0;"/><line style="stroke:#FF0000;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="99.5" x2="524" y1="474.0781" y2="474.0781"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="103" x="258.25" y="469.3359">Migration failed.</text><path d="M20,487.0781 L20,542.0781 L169,542.0781 L169,497.0781 L159,487.0781 L20,487.0781 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><path d="M159,487.0781 L159,497.0781 L169,497.0781 L159,487.0781 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="126" x="26" y="504.6465">Cleanup destination</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="128" x="26" y="519.957">propolis. Nothing to</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="88" x="26" y="535.2676">do for source.</text><polygon fill="#23684A" points="536,590.4531,526,594.4531,536,598.4531,532,594.4531" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="1004" y1="594.4531" y2="594.4531"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="184" x="673" y="574.2544">101 Switching Protocols</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="248" x="641" y="589.3872">Upgrade: propolis-migrate-ron/m</text><path d="M1010,560.0098 L1010,600.0098 L1184,600.0098 L1184,570.0098 L1174,560.0098 L1010,560.0098 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1174,560.0098 L1174,570.0098 L1184,570.0098 L1174,560.0098 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="117" x="1016" y="577.5781">Migration protocol</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="153" x="1016" y="592.8887">successfully negotiated.</text><path d="M467,610.6309 L467,650.6309 L1063,650.6309 L1063,620.6309 L1053,610.6309 L467,610.6309 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1053,610.6309 L1053,620.6309 L1063,620.6309 L1053,610.6309 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="336" x="592.5" y="628.1992">From this point on, the underlying TCP socket for the</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="336" x="592.5" y="643.5098">HTTP request is now used as the migration transport.</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="676.9072"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="676.9072" y2="676.9072"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="679.9072" y2="679.9072"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="49" x="659" y="666.252"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="31" x="665" y="682.8203">Sync</text><polygon fill="#23684A" points="1127,716.873,1137,720.873,1127,724.873,1131,720.873" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="1005" x2="1133" y1="720.873" y2="720.873"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="110" x="1017" y="716.1309">Get backend blob</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="82" x="1139" y="699.5625"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="68" x="1146" y="720.0977">OPTE Port</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="57" x="1151.5" y="736.5859">(Source)</text><polygon fill="#23684A" points="1219,774.8496,1229,778.8496,1219,782.8496,1223,778.8496" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="1005" x2="1225" y1="778.8496" y2="778.8496"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="110" x="1063" y="774.1074">Get backend blob</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="131" x="1231" y="757.5391"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="117" x="1238" y="778.0742">Crucible Upstairs</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="57" x="1268" y="794.5625">(Source)</text><polygon fill="#23684A" points="536,843.4814,526,847.4814,536,851.4814,532,847.4814" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="1004" y1="847.4814" y2="847.4814"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="96" x="717" y="842.7393">VM Description</text><path d="M1010,820.5156 L1010,860.5156 L1161,860.5156 L1161,830.5156 L1151,820.5156 L1010,820.5156 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1151,820.5156 L1151,830.5156 L1161,830.5156 L1151,820.5156 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="130" x="1016" y="838.084">Num of vCPUs, RAM,</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="119" x="1016" y="853.3945">backend blobs, etc</text><path d="M10,873.1367 L183,873.1367 L183,880.1367 L173,890.1367 L10,890.1367 L10,873.1367 " fill="#EEEEEE" style="stroke:#000000;stroke-width:1.0;"/><rect fill="none" height="178.4844" style="stroke:#000000;stroke-width:2.0;" width="610" x="10" y="873.1367"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="128" x="25" y="886.7051">Negotiation Failure</text><path d="M439,895.4473 L439,935.4473 L610,935.4473 L610,905.4473 L600,895.4473 L439,895.4473 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><path d="M600,895.4473 L600,905.4473 L610,905.4473 L600,895.4473 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="128" x="445" y="913.0156">Destination couldn't</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="150" x="445" y="928.3262">proceed with migration.</text><polygon fill="#FF0000" points="105.5,973.6895,95.5,977.6895,105.5,981.6895,101.5,977.6895" style="stroke:#FF0000;stroke-width:1.0;"/><line style="stroke:#FF0000;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="99.5" x2="524" y1="977.6895" y2="977.6895"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="0" x="311.75" y="957.6367"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="103" x="258.25" y="972.9473">Migration failed.</text><path d="M20,990.6895 L20,1045.6895 L169,1045.6895 L169,1000.6895 L159,990.6895 L20,990.6895 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><path d="M159,990.6895 L159,1000.6895 L169,1000.6895 L159,990.6895 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="126" x="26" y="1008.2578">Cleanup destination</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="128" x="26" y="1023.5684">propolis. Nothing to</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="88" x="26" y="1038.8789">do for source.</text><path d="M396,1063.6211 L396,1103.6211 L654,1103.6211 L654,1073.6211 L644,1063.6211 L396,1063.6211 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M644,1063.6211 L644,1073.6211 L654,1073.6211 L644,1063.6211 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="237" x="402" y="1081.1895">Allocate new VM with given resources</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="212" x="402" y="1096.5">and descriptions. State is paused.</text><polygon fill="#23684A" points="254,1126.5527,244,1130.5527,254,1134.5527,250,1130.5527" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="248" x2="524" y1="1130.5527" y2="1130.5527"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="117" x="325.5" y="1125.8105">Pass backend blob</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="102" x="141" y="1109.2422"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="68" x="158" y="1129.7773">OPTE Port</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="88" x="148" y="1146.2656">(Destination)</text><polygon fill="#23684A" points="395,1184.5293,385,1188.5293,395,1192.5293,391,1188.5293" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="389" x2="524" y1="1188.5293" y2="1188.5293"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="117" x="396" y="1183.7871">Pass backend blob</text><rect fill="#F6FDFA" height="46.9766" style="stroke:#23684A;stroke-width:1.5;" width="131" x="253" y="1167.2188"/><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="117" x="260" y="1187.7539">Crucible Upstairs</text><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="88" x="274.5" y="1204.2422">(Destination)</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="1245.8506"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="1245.8506" y2="1245.8506"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="1248.8506" y2="1248.8506"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="86" x="640.5" y="1235.1953"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="68" x="646.5" y="1251.7637">RAM Push</text><path d="M454.5,1275.5059 L611.5,1275.5059 L611.5,1282.5059 L601.5,1292.5059 L454.5,1292.5059 L454.5,1275.5059 " fill="#EEEEEE" style="stroke:#000000;stroke-width:1.0;"/><rect fill="none" height="192.9961" style="stroke:#000000;stroke-width:2.0;" width="621" x="454.5" y="1275.5059"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="108" x="469.5" y="1289.0742">Initial RAM Sync</text><text fill="#000000" font-family="sans-serif" font-size="11" font-weight="bold" lengthAdjust="spacing" textLength="239" x="626.5" y="1288.1406">[Repeat twice to whittle down working set]</text><polygon fill="#23684A" points="988,1310.127,998,1314.127,988,1318.127,992,1314.127" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="994" y1="1314.127" y2="1314.127"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="675" y="1309.061">Query</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="136" x="719" y="1309.3848">for full address space</text><polygon fill="#23684A" points="541,1339.4375,531,1343.4375,541,1347.4375,537,1343.4375" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="999" y1="1343.4375" y2="1343.4375"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="689" y="1338.3716">Offer</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="108" x="733" y="1338.6953">some dirty pages</text><polygon fill="#23684A" points="541,1368.748,531,1372.748,541,1376.748,537,1372.748" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="1004" y1="1372.748" y2="1372.748"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="701" y="1367.6821">End</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="105" x="729" y="1368.0059">(query complete)</text><polygon fill="#23684A" points="993,1398.0586,1003,1402.0586,993,1406.0586,997,1402.0586" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="999" y1="1402.0586" y2="1402.0586"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="688.5" y="1396.9927">Fetch</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="114" x="732.5" y="1397.3164">full address space</text><polygon fill="#23684A" points="541,1427.3691,531,1431.3691,541,1435.3691,537,1431.3691" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="1004" y1="1431.3691" y2="1431.3691"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="32" x="692.5" y="1426.3032">Xfer</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="114" x="728.5" y="1426.627">full address space</text><polygon fill="#23684A" points="993,1456.502,1003,1460.502,993,1464.502,997,1460.502" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="525" x2="999" y1="1460.502" y2="1460.502"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="32" x="749" y="1455.436">Done</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="1496.1572"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="1496.1572" y2="1496.1572"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="1499.1572" y2="1499.1572"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="107" x="630" y="1485.502"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="89" x="636" y="1502.0703">Source Pause</text><path d="M881,1523.8125 L881,1563.8125 L1128,1563.8125 L1128,1533.8125 L1118,1523.8125 L881,1523.8125 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1118,1523.8125 L1118,1533.8125 L1128,1533.8125 L1118,1523.8125 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="99" x="887" y="1541.3809">Begin migrating</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="226" x="887" y="1556.6914">e.g. stop answering NVMe doorbells</text><polygon fill="#23684A" points="1168,1586.7441,1178,1590.7441,1168,1594.7441,1172,1590.7441" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="1005" x2="1174" y1="1590.7441" y2="1590.7441"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="99" x="1043" y="1586.002">Begin migrating</text><polygon fill="#23684A" points="1016,1615.877,1006,1619.877,1016,1623.877,1012,1619.877" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="1010" x2="1179" y1="1619.877" y2="1619.877"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="1080.5" y="1614.811">ACK</text><polygon fill="#23684A" points="1284.5,1645.1875,1294.5,1649.1875,1284.5,1653.1875,1288.5,1649.1875" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="1005" x2="1290.5" y1="1649.1875" y2="1649.1875"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="99" x="1101.25" y="1644.4453">Begin migrating</text><polygon fill="#23684A" points="1016,1674.3203,1006,1678.3203,1016,1682.3203,1012,1678.3203" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="1010" x2="1295.5" y1="1678.3203" y2="1678.3203"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="1138.75" y="1673.2544">ACK</text><path d="M955,1691.3203 L955,1716.3203 L1055,1716.3203 L1055,1701.3203 L1045,1691.3203 L955,1691.3203 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1045,1691.3203 L1045,1701.3203 L1055,1701.3203 L1045,1691.3203 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="79" x="961" y="1708.8887">Pause vCPUs</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="1742.2861"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="1742.2861" y2="1742.2861"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="1745.2861" y2="1745.2861"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="133" x="617" y="1731.6309"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="115" x="623" y="1748.1992">RAM Push (Dirty)</text><path d="M454.5,1771.9414 L649.5,1771.9414 L649.5,1778.9414 L639.5,1788.9414 L454.5,1788.9414 L454.5,1771.9414 " fill="#EEEEEE" style="stroke:#000000;stroke-width:1.0;"/><rect fill="none" height="251.6172" style="stroke:#000000;stroke-width:2.0;" width="621" x="454.5" y="1771.9414"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="150" x="469.5" y="1785.5098">Working Set RAM Sync</text><polygon fill="#23684A" points="988,1806.5625,998,1810.5625,988,1814.5625,992,1810.5625" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="994" y1="1810.5625" y2="1810.5625"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="697" y="1805.4966">Query</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="92" x="741" y="1805.8203">for dirty pages</text><polygon fill="#23684A" points="541,1835.873,531,1839.873,541,1843.873,537,1839.873" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="999" y1="1839.873" y2="1839.873"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="689" y="1834.8071">Offer</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="108" x="733" y="1835.1309">some dirty pages</text><polygon fill="#23684A" points="541,1865.1836,531,1869.1836,541,1873.1836,537,1869.1836" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="1004" y1="1869.1836" y2="1869.1836"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="701" y="1864.1177">End</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="105" x="729" y="1864.4414">(query complete)</text><polygon fill="#23684A" points="993,1894.4941,1003,1898.4941,993,1902.4941,997,1898.4941" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="999" y1="1898.4941" y2="1898.4941"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="710.5" y="1893.4282">Fetch</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="70" x="754.5" y="1893.752">dirty pages</text><polygon fill="#23684A" points="541,1923.8047,531,1927.8047,541,1931.8047,537,1927.8047" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="1004" y1="1927.8047" y2="1927.8047"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="32" x="714.5" y="1922.7388">Xfer</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="70" x="750.5" y="1923.0625">dirty pages</text><polygon fill="#23684A" points="988,1953.1152,998,1957.1152,988,1961.1152,992,1957.1152" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="994" y1="1957.1152" y2="1957.1152"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="590" y="1952.0493">Query</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="306" x="634" y="1952.373">for dirty pages (should be none as VM is paused)</text><polygon fill="#23684A" points="541,1982.4258,531,1986.4258,541,1990.4258,537,1986.4258" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="535" x2="1004" y1="1986.4258" y2="1986.4258"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="658" y="1981.3599">End</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="133" x="686" y="1981.6836">(query complete - no</text><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="40" x="823" y="1981.3599">Offer</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="14" x="863" y="1981.6836">'s)</text><polygon fill="#23684A" points="993,2011.5586,1003,2015.5586,993,2019.5586,997,2015.5586" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="525" x2="999" y1="2015.5586" y2="2015.5586"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="32" x="749" y="2010.4927">Done</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="2051.2139"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="2051.2139" y2="2051.2139"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="2054.2139" y2="2054.2139"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="137" x="615" y="2040.5586"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="119" x="621" y="2057.127">Device State Push</text><path d="M928,2078.8691 L928,2103.8691 L1082,2103.8691 L1082,2088.8691 L1072,2078.8691 L928,2078.8691 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1072,2078.8691 L1072,2088.8691 L1082,2088.8691 L1072,2078.8691 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="133" x="934" y="2096.4375">Serialize Device State</text><polygon fill="#23684A" points="536,2126.4902,526,2130.4902,536,2134.4902,532,2130.4902" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="1004" y1="2130.4902" y2="2130.4902"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="109" x="710.5" y="2125.748">Send device state</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="2159.1455"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="2159.1455" y2="2159.1455"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="2162.1455" y2="2162.1455"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="180" x="593.5" y="2148.4902"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="162" x="599.5" y="2165.0586">Architectural State Push</text><path d="M929,2186.8008 L929,2226.8008 L1081,2226.8008 L1081,2196.8008 L1071,2186.8008 L929,2186.8008 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1071,2186.8008 L1071,2196.8008 L1081,2196.8008 L1071,2186.8008 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="120" x="935" y="2204.3691">Serialize Arch State</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="131" x="935" y="2219.6797">(e.g. registers, MSRs)</text><polygon fill="#23684A" points="536,2249.7324,526,2253.7324,536,2257.7324,532,2253.7324" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="530" x2="1004" y1="2253.7324" y2="2253.7324"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="96" x="717" y="2248.9902">Send arch state</text><rect fill="#EEEEEE" height="3" style="stroke:#EEEEEE;stroke-width:1.0;" width="1367" x="0" y="2282.3877"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="2282.3877" y2="2282.3877"/><line style="stroke:#000000;stroke-width:1.0;" x1="0" x2="1367" y1="2285.3877" y2="2285.3877"/><rect fill="#EEEEEE" height="23.3105" style="stroke:#000000;stroke-width:2.0;" width="153" x="607" y="2271.7324"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="135" x="613" y="2288.3008">Destination Resume</text><polygon fill="#23684A" points="993,2322.3535,1003,2326.3535,993,2330.3535,997,2326.3535" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="525" x2="999" y1="2326.3535" y2="2326.3535"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="48" x="631" y="2321.2876">Resume</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="216" x="683" y="2321.6113">Acknowledge Dst will resume now</text><path d="M48,2341.3535 L214,2341.3535 L214,2348.3535 L204,2358.3535 L48,2358.3535 L48,2341.3535 " fill="#EEEEEE" style="stroke:#000000;stroke-width:1.0;"/><rect fill="none" height="172.4844" style="stroke:#000000;stroke-width:2.0;" width="1027.5" x="48" y="2341.3535"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="121" x="63" y="2354.9219">Recoverable Error</text><path d="M382,2363.6641 L382,2434.6641 L667,2434.6641 L667,2373.6641 L657,2363.6641 L382,2363.6641 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><path d="M657,2363.6641 L657,2373.6641 L667,2373.6641 L657,2363.6641 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="264" x="388" y="2381.2324">The destination can't resume for whatever</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="261" x="388" y="2396.543">reason and it has yet to modify any guest</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="241" x="388" y="2411.8535">state nor attach to any backends. Give</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="223" x="388" y="2427.1641">source a change to resume instead.</text><polygon fill="#FF0000" points="993,2457.2168,1003,2461.2168,993,2465.2168,997,2461.2168" style="stroke:#FF0000;stroke-width:1.0;"/><line style="stroke:#FF0000;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="525" x2="999" y1="2461.2168" y2="2461.2168"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="150" x="690" y="2456.4746">Error, let source resume</text><polygon fill="#FF0000" points="105.5,2501.8379,95.5,2505.8379,105.5,2509.8379,101.5,2505.8379" style="stroke:#FF0000;stroke-width:1.0;"/><line style="stroke:#FF0000;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="99.5" x2="524" y1="2505.8379" y2="2505.8379"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="0" x="311.75" y="2485.7852"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="103" x="258.25" y="2501.0957">Migration failed.</text><polygon fill="#23684A" points="203,2538.1484,193,2542.1484,203,2546.1484,199,2542.1484" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="197" x2="524" y1="2542.1484" y2="2542.1484"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="48" x="316.5" y="2537.0825">Resume</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="32" x="368.5" y="2537.4063">OPTE</text><polygon fill="#23684A" points="513,2567.2813,523,2571.2813,513,2575.2813,517,2571.2813" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="192" x2="519" y1="2571.2813" y2="2571.2813"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="346.5" y="2566.2153">ACK</text><polygon fill="#23684A" points="329.5,2596.5918,319.5,2600.5918,329.5,2604.5918,325.5,2600.5918" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="323.5" x2="524" y1="2600.5918" y2="2600.5918"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="48" x="369.75" y="2595.5259">Resume</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="52" x="421.75" y="2595.8496">Crucible</text><polygon fill="#23684A" points="513,2625.7246,523,2629.7246,513,2633.7246,517,2629.7246" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="318.5" x2="519" y1="2629.7246" y2="2629.7246"/><text fill="#000000" font-family="monospace" font-size="13" lengthAdjust="spacing" textLength="24" x="409.75" y="2624.6587">ACK</text><path d="M468,2642.7246 L468,2667.7246 L581,2667.7246 L581,2652.7246 L571,2642.7246 L468,2642.7246 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M571,2642.7246 L571,2652.7246 L581,2652.7246 L571,2642.7246 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="92" x="474" y="2660.293">Resume vCPUs</text><path d="M48,2680.0352 L230,2680.0352 L230,2687.0352 L220,2697.0352 L48,2697.0352 L48,2680.0352 " fill="#EEEEEE" style="stroke:#000000;stroke-width:1.0;"/><rect fill="none" height="143.1738" style="stroke:#000000;stroke-width:2.0;" width="635" x="48" y="2680.0352"/><text fill="#000000" font-family="sans-serif" font-size="13" font-weight="bold" lengthAdjust="spacing" textLength="137" x="63" y="2693.6035">Unrecoverable Error</text><path d="M376,2702.3457 L376,2773.3457 L673,2773.3457 L673,2712.3457 L663,2702.3457 L376,2702.3457 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><path d="M663,2702.3457 L663,2712.3457 L673,2712.3457 L663,2702.3457 " fill="#FFAAAA" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="264" x="382" y="2719.9141">The destination can't resume for whatever</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="272" x="382" y="2735.2246">reason but it has already possibly modified</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="276" x="382" y="2750.5352">guest state or attached to backends. Source</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="270" x="382" y="2765.8457">cannot easily resume from this so just fail.</text><polygon fill="#FF0000" points="105.5,2811.209,95.5,2815.209,105.5,2819.209,101.5,2815.209" style="stroke:#FF0000;stroke-width:1.0;"/><line style="stroke:#FF0000;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="99.5" x2="524" y1="2815.209" y2="2815.209"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="0" x="311.75" y="2795.1563"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="103" x="258.25" y="2810.4668">Migration failed.</text><polygon fill="#23684A" points="993,2847.5195,1003,2851.5195,993,2855.5195,997,2851.5195" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="525" x2="999" y1="2851.5195" y2="2851.5195"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="140" x="695" y="2846.7773">OK, resume succesful.</text><path d="M898,2864.5195 L898,2904.5195 L1112,2904.5195 L1112,2874.5195 L1102,2864.5195 L898,2864.5195 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><path d="M1102,2864.5195 L1102,2874.5195 L1112,2874.5195 L1102,2864.5195 " fill="#D1F5E5" style="stroke:#23684A;stroke-width:1.0;"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="191" x="904" y="2882.0879">The migration was successful.</text><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="193" x="904" y="2897.3984">Clean up source VM resources.</text><polygon fill="#23684A" points="105.5,2927.4512,95.5,2931.4512,105.5,2935.4512,101.5,2931.4512" style="stroke:#23684A;stroke-width:1.0;"/><line style="stroke:#23684A;stroke-width:1.0;" x1="99.5" x2="524" y1="2931.4512" y2="2931.4512"/><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="134" x="242.75" y="2926.709">Migration succeeded.</text><!--MD5=[7ca2938a53a81042d900a40b55d7bc47]
@startuml migration-protocol-flow
!include theme.pml
title "Propolis Migration Protocol Flow"
skinparam sequenceMessageAlign center
participant Omicron order 10
participant Src as "propolis-server\n(Source)" order 30
create Dst as "propolis-server\n(Destination)" order 20
Omicron -> Dst: Create new propolis-server\ninstance for destination VM
Omicron -> Dst: ""PUT /instances/{dst}/migrate""\n""role = Destination""\n""src-addr = ...""\n""src-uuid = ...""
note left: Initiate\nmigration
note over Omicron
Omicron waits for
HTTP response
end note
== Protocol Negotiation ==
Dst -> Src: ""PUT /instances/{src}/migrate""\n""Connection = Upgrade""\n""Upgrade = propolis-migrate-ron/m, propolis-migrate-bson/n""\n""role = Source""
group Negotiation Failure
Src -[#red]-> Dst: ""Upgrade"" header didn't list a supported version or encoding.
Dst -[#red]-> Omicron: Migration failed.
note over Omicron #FFAAAA
Cleanup destination
propolis. Nothing to
do for source.
end note
end
Src -> Dst: ""101 Switching Protocols""\n""Upgrade: propolis-migrate-ron/m""
note right: Migration protocol\nsuccessfully negotiated.
note over Dst, Src
From this point on, the underlying TCP socket for the
HTTP request is now used as the migration transport.
end note
== Sync ==
create SrcOpte as "OPTE Port\n(Source)" order 31
Src -> SrcOpte: Get backend blob
create SrcCrucible as "Crucible Upstairs\n(Source)" order 32
Src -> SrcCrucible: Get backend blob
Src -> Dst: VM Description
note right: Num of vCPUs, RAM,\nbackend blobs, etc
group Negotiation Failure
note over Dst #FFAAAA
Destination couldn't
proceed with migration.
end note
Dst -[#red]-> Omicron: \nMigration failed.
note over Omicron #FFAAAA
Cleanup destination
propolis. Nothing to
do for source.
end note
end
note over Dst
Allocate new VM with given resources
and descriptions. State is paused.
end note
create DstOpte as "OPTE Port\n(Destination)" Order 18
Dst -> DstOpte: Pass backend blob
create DstCrucible as "Crucible Upstairs\n(Destination)" order 19
Dst -> DstCrucible: Pass backend blob
== RAM Push ==
group Initial RAM Sync [Repeat twice to whittle down working set]
Dst -> Src: ""Query"" for full address space
activate Dst
activate Src
Src -> Dst: ""Offer"" some dirty pages
Src -> Dst: ""End"" (query complete)
deactivate Src
Dst -> Src: ""Fetch"" full address space
Src -> Dst: ""Xfer"" full address space
Dst -> Src: ""Done""
deactivate Dst
end
== Source Pause ==
note over Src
Begin migrating
e.g. stop answering NVMe doorbells
end note
Src -> SrcOpte: Begin migrating
SrcOpte -> Src: ""ACK""
Src -> SrcCrucible: Begin migrating
SrcCrucible -> Src: ""ACK""
note over Src
Pause vCPUs
end note
== RAM Push (Dirty) ==
group Working Set RAM Sync
Dst -> Src: ""Query"" for dirty pages
activate Dst
activate Src
Src -> Dst: ""Offer"" some dirty pages
Src -> Dst: ""End"" (query complete)
deactivate Src
Dst -> Src: ""Fetch"" dirty pages
Src -> Dst: ""Xfer"" dirty pages
Dst -> Src: ""Query"" for dirty pages (should be none as VM is paused)
activate Src
Src -> Dst: ""End"" (query complete - no ""Offer""'s)
deactivate Src
Dst -> Src: ""Done""
deactivate Dst
end
== Device State Push ==
note over Src
Serialize Device State
end note
Src -> Dst: Send device state
== Architectural State Push ==
note over Src
Serialize Arch State
(e.g. registers, MSRs)
end note
Src -> Dst: Send arch state
== Destination Resume ==
Dst -> Src: ""Resume"" Acknowledge Dst will resume now
group Recoverable Error
note over Dst #FFAAAA
The destination can't resume for whatever
reason and it has yet to modify any guest
state nor attach to any backends. Give
source a change to resume instead.
end note
Dst -[#red]-> Src: Error, let source resume
Dst -[#red]-> Omicron: \nMigration failed.
end
Dst -> DstOpte: ""Resume"" OPTE
DstOpte -> Dst: ""ACK""
Dst -> DstCrucible: ""Resume"" Crucible
DstCrucible -> Dst: ""ACK""
note over Dst
Resume vCPUs
end note
group Unrecoverable Error
note over Dst #FFAAAA
The destination can't resume for whatever
reason but it has already possibly modified
guest state or attached to backends. Source
cannot easily resume from this so just fail.
end note
Dst -[#red]-> Omicron: \nMigration failed.
end
Dst -> Src: OK, resume succesful.
note over Src
The migration was successful.
Clean up source VM resources.
end note
Dst -> Omicron: Migration succeeded.
@enduml
@startuml migration-protocol-flow
skinparam shadowing false
skinparam ArrowColor #23684A
skinparam ActorBackgroundColor #F6FDFA
skinparam ActorBorderColor #23684A
skinparam NodeBackgroundColor #F6FDFA
skinparam NodeBorderColor #23684A
skinparam CardBackgroundColor #F6FDFA
skinparam CardBorderColor #23684A
skinparam DatabaseBackgroundColor #F6FDFA
skinparam DatabaseBorderColor #23684A
skinparam NoteBackgroundColor #D1F5E5
skinparam NoteBorderColor #23684A
skinparam sequence {
ParticipantBackgroundColor #23684A
ParticipantBackgroundColor #F6FDFA
ParticipantBorderColor #23684A
LifeLineBorderColor #23684A
}
title "Propolis Migration Protocol Flow"
skinparam sequenceMessageAlign center
participant Omicron order 10
participant Src as "propolis-server\n(Source)" order 30
create Dst as "propolis-server\n(Destination)" order 20
Omicron -> Dst: Create new propolis-server\ninstance for destination VM
Omicron -> Dst: ""PUT /instances/{dst}/migrate""\n""role = Destination""\n""src-addr = ...""\n""src-uuid = ...""
note left: Initiate\nmigration
note over Omicron
Omicron waits for
HTTP response
end note
== Protocol Negotiation ==
Dst -> Src: ""PUT /instances/{src}/migrate""\n""Connection = Upgrade""\n""Upgrade = propolis-migrate-ron/m, propolis-migrate-bson/n""\n""role = Source""
group Negotiation Failure
Src -[#red]-> Dst: ""Upgrade"" header didn't list a supported version or encoding.
Dst -[#red]-> Omicron: Migration failed.
note over Omicron #FFAAAA
Cleanup destination
propolis. Nothing to
do for source.
end note
end
Src -> Dst: ""101 Switching Protocols""\n""Upgrade: propolis-migrate-ron/m""
note right: Migration protocol\nsuccessfully negotiated.
note over Dst, Src
From this point on, the underlying TCP socket for the
HTTP request is now used as the migration transport.
end note
== Sync ==
create SrcOpte as "OPTE Port\n(Source)" order 31
Src -> SrcOpte: Get backend blob
create SrcCrucible as "Crucible Upstairs\n(Source)" order 32
Src -> SrcCrucible: Get backend blob
Src -> Dst: VM Description
note right: Num of vCPUs, RAM,\nbackend blobs, etc
group Negotiation Failure
note over Dst #FFAAAA
Destination couldn't
proceed with migration.
end note
Dst -[#red]-> Omicron: \nMigration failed.
note over Omicron #FFAAAA
Cleanup destination
propolis. Nothing to
do for source.
end note
end
note over Dst
Allocate new VM with given resources
and descriptions. State is paused.
end note
create DstOpte as "OPTE Port\n(Destination)" Order 18
Dst -> DstOpte: Pass backend blob
create DstCrucible as "Crucible Upstairs\n(Destination)" order 19
Dst -> DstCrucible: Pass backend blob
== RAM Push ==
group Initial RAM Sync [Repeat twice to whittle down working set]
Dst -> Src: ""Query"" for full address space
activate Dst
activate Src
Src -> Dst: ""Offer"" some dirty pages
Src -> Dst: ""End"" (query complete)
deactivate Src
Dst -> Src: ""Fetch"" full address space
Src -> Dst: ""Xfer"" full address space
Dst -> Src: ""Done""
deactivate Dst
end
== Source Pause ==
note over Src
Begin migrating
e.g. stop answering NVMe doorbells
end note
Src -> SrcOpte: Begin migrating
SrcOpte -> Src: ""ACK""
Src -> SrcCrucible: Begin migrating
SrcCrucible -> Src: ""ACK""
note over Src
Pause vCPUs
end note
== RAM Push (Dirty) ==
group Working Set RAM Sync
Dst -> Src: ""Query"" for dirty pages
activate Dst
activate Src
Src -> Dst: ""Offer"" some dirty pages
Src -> Dst: ""End"" (query complete)
deactivate Src
Dst -> Src: ""Fetch"" dirty pages
Src -> Dst: ""Xfer"" dirty pages
Dst -> Src: ""Query"" for dirty pages (should be none as VM is paused)
activate Src
Src -> Dst: ""End"" (query complete - no ""Offer""'s)
deactivate Src
Dst -> Src: ""Done""
deactivate Dst
end
== Device State Push ==
note over Src
Serialize Device State
end note
Src -> Dst: Send device state
== Architectural State Push ==
note over Src
Serialize Arch State
(e.g. registers, MSRs)
end note
Src -> Dst: Send arch state
== Destination Resume ==
Dst -> Src: ""Resume"" Acknowledge Dst will resume now
group Recoverable Error
note over Dst #FFAAAA
The destination can't resume for whatever
reason and it has yet to modify any guest
state nor attach to any backends. Give
source a change to resume instead.
end note
Dst -[#red]-> Src: Error, let source resume
Dst -[#red]-> Omicron: \nMigration failed.
end
Dst -> DstOpte: ""Resume"" OPTE
DstOpte -> Dst: ""ACK""
Dst -> DstCrucible: ""Resume"" Crucible
DstCrucible -> Dst: ""ACK""
note over Dst
Resume vCPUs
end note
group Unrecoverable Error
note over Dst #FFAAAA
The destination can't resume for whatever
reason but it has already possibly modified
guest state or attached to backends. Source
cannot easily resume from this so just fail.
end note
Dst -[#red]-> Omicron: \nMigration failed.
end
Dst -> Src: OK, resume succesful.
note over Src
The migration was successful.
Clean up source VM resources.
end note
Dst -> Omicron: Migration succeeded.
@enduml
PlantUML version 1.2021.12(Tue Oct 05 12:01:58 EDT 2021)
(GPL source distribution)
Java Runtime: OpenJDK Runtime Environment
JVM: OpenJDK 64-Bit Server VM
Default Encoding: UTF-8
Language: en
Country: US
--></g></svg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment