Extract the iso image and the contained squashfs image
$ mkdir android-image
$ bsdtar -C android-image -xf xxx.iso
$ cd android-image
$ unsquashfs system.sfs
Modify system.img size.
$ sudo e2fsck -f ./squashfs-root/system.img
$ sudo resize2fs ./squashfs-root/system.img 6G
Mount the contained filesystem image as a loopback device.
$ sudo mkdir /mnt/iso
$ sudo mount ./squashfs-root/system.img /mnt/iso -o loop
In the android-x86 build the boot folder is in a separate location so we move it from the extracted image to repack it later.
$ mkdir -p ~/Temp/boot
$ cd ..
$ sudo cp -r android-image/isolinux ~/Temp/boot
Now you can enter cd /mnt/iso to modify system.img
Repackage iso.
$ cd android-image
$ sudo umount /mnt/iso/
$ mksquashfs squashfs-root/system.img system.sfs
$ cd ..
$ genisoimage -vJURT -o Android-x86.iso -V "Android-x86 LiveCD" android-image/