OCI Custom Image Import on ARM Always Free — hard-won notes

oci-byoi-arm-notes.md

OCI Custom Image Import on ARM Always Free — hard-won notes

Field notes from wiping a FreeBSD Always Free A1.Flex and standing up a fresh Fedora 43 aarch64 via BYOI. Captures the things that are either undocumented, contradicted between docs pages, or only discoverable by trying.

Environment: us-sanjose-1, VM.Standard.A1.Flex (4 OCPU / 24 GB), 200 GB BV / 10 VPUs, Fedora-Cloud-Base-Generic-43-1.6.aarch64.qcow2.

TL;DR — the working recipe

1. Upload the aarch64 qcow2 to an Object Storage bucket.
2. Create the custom image via raw API (CLI can't do it) with launchMode=CUSTOM and explicit launchOptions containing firmware=UEFI_64 plus PV everywhere.
3. Add A1.Flex shape compatibility — imported images don't inherit it.
4. Launch normally. Do not try to override firmware via --launch-options; it's rejected.

Everything else in this doc is the path not to take.

Gotcha 1: ephemeral public IPs are one-way

oci network public-ip update --lifetime RESERVED doesn't exist. From the docs:

You can't move an ephemeral public IP to a different private IP. If you want to unassign an ephemeral public IP from its private IP, use DeletePublicIp, which unassigns and deletes the ephemeral public IP.

Consequence: if you terminate the instance (or its VNIC), the exact address is gone. The only way to keep a specific IP across instance recreation is to allocate a reserved public IP up front and assign it. Converting after the fact is not supported.

Always Free includes 1 reserved public IP.

Gotcha 2: boot volume backups carry a "parent instance" lock

Restoring the "Before FreeBSD" backup into a new boot volume and attempting to attach it to the existing instance yielded:

Boot volume ... cannot be attached to instance ...
It can only be attached to its parent instance.

The backup was sourced from a prior (now terminated) instance's BV, and that lineage is permanently anchored. The workaround is to launch a new instance from the boot volume (not attach it to an existing one).

Not documented on the restore page; the error message is the only hint.

Gotcha 3: oci iam region list lies

oci iam region list returns the global list of OCI regions, sorted alphabetically. If you rely on data[0] you'll think your home region is Amsterdam. Use:

oci iam region-subscription list --tenancy-id <tenancy>

…for subscribed regions, or oci iam tenancy get --tenancy-id <tenancy> --query 'data."home-region-key"' for the home region.

Gotcha 4: aarch64 BYOI is hostile, three ways

oci compute image import from-object exposes --launch-mode but the modes have different silent failure modes on ARM:

launch-mode	firmware default	boot-volume-type	Result on Fedora aarch64 cloud qcow2
PARAVIRTUALIZED	BIOS	PARAVIRTUALIZED	Instance enters RUNNING, console stays empty, SSH times out. ARM cloud images are UEFI-only; OCI's ARM "BIOS" boot path is effectively a non-working placeholder inherited from the cross-arch API schema.
NATIVE	UEFI_64	ISCSI	Kernel boots fine, then dracut hangs indefinitely waiting for /dev/disk/by-uuid/<root>. Fedora/Debian generic cloud initramfs does not include iscsi-initiator-utils.
CUSTOM	unset — you must supply launchOptions	—	Works, but the CLI silently drops the launchOptions block even with --from-json. See Gotcha 5.

OCI docs say "imported images on ARM are supported in paravirtualized mode only" — true but misleading: PARAVIRTUALIZED mode hardcodes BIOS firmware which breaks aarch64 UEFI boot. You actually need CUSTOM with explicit firmware + PV.

Gotcha 5: oci compute image import from-object cannot set launchOptions

--generate-full-command-json-input confirms the field is absent from the CLI schema. Trying --launch-mode CUSTOM without launchOptions returns:

MissingParameter: launchOptions must be provided when using CUSTOM launchMode

Passing the launchOptions block via --from-json is silently ignored.

Workaround: call the raw API. The body supports everything the SDK model does:

oci raw-request \
  --http-method POST \
  --target-uri "https://iaas.us-sanjose-1.oraclecloud.com/20160918/images" \
  --request-body "$(cat image-create.json)"

With image-create.json:

{
  "compartmentId": "ocid1.tenancy...",
  "displayName": "Fedora-43-Cloud-ARM64",
  "launchMode": "CUSTOM",
  "launchOptions": {
    "bootVolumeType": "PARAVIRTUALIZED",
    "networkType": "PARAVIRTUALIZED",
    "remoteDataVolumeType": "PARAVIRTUALIZED",
    "firmware": "UEFI_64",
    "isConsistentVolumeNamingEnabled": false,
    "isPvEncryptionInTransitEnabled": false
  },
  "imageSourceDetails": {
    "sourceType": "objectStorageTuple",
    "namespaceName": "...",
    "bucketName": "os-images",
    "objectName": "Fedora-Cloud-Base-Generic-43-1.6.aarch64.qcow2",
    "sourceImageType": "QCOW2",
    "operatingSystem": "Fedora",
    "operatingSystemVersion": "43"
  }
}

Gotcha 6: firmware cannot be overridden at launch

The obvious workaround to Gotcha 5 — import as PARAVIRTUALIZED, then override firmware at launch — does not work:

InvalidParameter: Overriding Firmware in LaunchOptions is not supported

Only networkType, bootVolumeType, remoteDataVolumeType and the two PV encryption bools are overridable at launch. firmware is locked to the image's value.

This forces the raw-API dance in Gotcha 5.

Gotcha 7: shape compatibility doesn't auto-attach

After import, the image rejects every shape by default:

Shape VM.Standard.A1.Flex is not valid for image ...

Fix:

oci compute image-shape-compatibility-entry add \
  --image-id ... --shape-name VM.Standard.A1.Flex

Silent requirement — nothing in the import output suggests it.

Gotcha 8: "Always Free" is not a tag

defined-tags, freeform-tags, and system-tags on an Always-Free-eligible instance contain no "Always Free" marker. The console's label is computed from shape + OCPU/RAM + BV size + VPUs at render time. No API field tells you "this is in the free tier" — you check the config against the published limits:

• Shape: VM.Standard.A1.Flex (ARM) or VM.Standard.E2.1.Micro (AMD)
• ARM: ≤ 4 OCPU, ≤ 24 GB RAM (aggregate across all A1 instances)
• BV: ≤ 200 GB total, Balanced performance (10 VPU/GB)

Cleanup ordering

When nuking a VCN stack, order matters — each step has a dependency that isn't auto-resolved:

1. Terminate instances (with --preserve-boot-volume false unless you want orphan BVs).
2. Delete subnets.
3. Clear route-table rules referencing the Internet Gateway before deleting the IG. Otherwise:

   The Route Table ... references the Internet Gateway ...
   

4. Delete IG.
5. Delete VCN. The default RT / SL / DHCP options are cleaned up automatically.

The default security list for a DHCP-created VCN often has cruft (UDP any-port open, stale IP allowlists). Reset it explicitly if you care.

Timing (us-sanjose-1)

• qcow2 upload (520 MB, WSL → SJC): ~1 min
• Image import (qcow2 → AVAILABLE): ~8 min
• Instance provision → RUNNING: ~1 min
• Cloud-init + sshd ready after RUNNING: ~10-30 s

Each failed launch-mode attempt = one import cycle. Doing Gotcha 4's matrix empirically costs ~25 min. Skip to CUSTOM + raw API and save the detour.

What actually worked, end-to-end

# 1. Upload
oci os object put -ns <ns> -bn os-images \
  --file Fedora-Cloud-Base-Generic-43-1.6.aarch64.qcow2 \
  --name Fedora-Cloud-Base-Generic-43-1.6.aarch64.qcow2

# 2. Create image via raw API (see image-create.json above)
oci raw-request --http-method POST \
  --target-uri "https://iaas.<region>.oraclecloud.com/20160918/images" \
  --request-body "$(cat image-create.json)"

# 3. Wait for AVAILABLE, then:
oci compute image-shape-compatibility-entry add \
  --image-id <image-ocid> --shape-name VM.Standard.A1.Flex

# 4. Launch (no launch-options override needed — image already has them)
oci compute instance launch \
  --availability-domain "<AD>" --compartment-id <tenancy> \
  --shape VM.Standard.A1.Flex \
  --shape-config '{"ocpus":4,"memoryInGBs":24}' \
  --subnet-id <subnet> --display-name us-west \
  --source-details '{"sourceType":"image","imageId":"<image-ocid>","bootVolumeSizeInGBs":200,"bootVolumeVpusPerGB":10}' \
  --metadata file://meta.json \
  --assign-public-ip true

# 5. ssh fedora@<public-ip>

meta.json:

{"ssh_authorized_keys": "ssh-ed25519 AAAA... user@host"}

Fedora image choices for aarch64 on OCI

• Fedora-Cloud-Base-Generic-<ver>.aarch64.qcow2 — lean (~520 MB), cloud-init enabled, includes Oracle datasource. Default choice.
• Fedora-Server-Guest-Generic-<ver>.aarch64.qcow2 — larger (~820 MB), Fedora Server variant (Cockpit, firewalld).
• Fedora-Cloud-Base-UEFI-UKI-<ver>.aarch64.qcow2 — UEFI + Unified Kernel Image variant. Also works with the CUSTOM+PV+UEFI_64 recipe.

Default user is fedora, not opc / ec2-user / root.