Skip to content

Instantly share code, notes, and snippets.

@Hengjie
Last active November 27, 2024 12:51
Show Gist options
  • Save Hengjie/1520114890bebe8f805d337af4b3a064 to your computer and use it in GitHub Desktop.
Save Hengjie/1520114890bebe8f805d337af4b3a064 to your computer and use it in GitHub Desktop.
How to passthrough SATA drives directly on VMWare ESXI 6.5 as RDMs

How to passthrough SATA drives directly on VMWare EXSI 6.5 as RDMs

There aren't many tutorials about this, the only tutorials I've found were about passing through entire PCIe cards to VMs, or refered to old ESXI versions (below 6.5) that used a more comprehensive desktop client instead of the web app. In v6.5, the web app was introduced and the desktop client was deprecated. You used to be able to setup RDMs in the desktop client, but with the introduction of the web console, this is no longer the case. This tutorial shows you how to pass SATA HDDs to the virtual machine on VMWare ESXI 6.5. This tutorial is partially based on VMWare's own KB and the now deprecated Forza IT blog post.

A word about VMWare ESXI 6.7

There is now an option while editing your VM's settings to add a New raw disk when you click Add hard disk. I have not personally tried it yet, but the comments below have confirmed it works.

Summary

We attach the SATA HDDs as a RDM (Raw Device Mapper) into an existing virtual disk in the command line, then on the web app, we attach a new SCSI controller to the VM, and attach the newly created RDM as an "existing HDD" to the newly created SCSI controller.

Prerequisite knowledge

If you're new to EXSI, you should know a few things:

  • A Datastore is basically this separate layer that sits between a physical device/disk and a virtual disk. A datastore can max the storage of a physical disk, or not. A datastore can have many virtual disks. Virtual disks are what a virtual machine will stores files on. In other words Physical Disk > Datastore > Virtual Disks > Your filesystem e.g. EXT4
  • RDM aka Raw Device Mapper is a pointer file that acts like a virtual disk but instead maps directly to a physical disk. RDMs on EXSI 6.5 have to be created in the command line.

Instructions

  1. Before you get started, make sure you've got a VM setup, and running. In my example, I have a virtual machine and virtual disk both conveniently named Ubuntu Storage

  2. In EXSI's web interface, log in, go to the home page. Click on Actions > Services > Enable Secure Shell (SSH).

  3. Open a SSH session to the ESXi/ESX host.

# ssh root@YOUR_EXSI_IP -p 22
  1. Run this command to list the disks that are attached to the ESXi host:
# ls -l /vmfs/devices/disks

It should look like this. We care about the physical disks with the prefix t10. As mentioned in the comments, it may not necessarily start with t10. You can determined the name/path by going to the web console then Storage, then Device, click on the device you're setting up as RDM, and then copying the path.

[root@localhost:~] ls -l /vmfs/devices/disks
total 12221184373
-rw-------    1 root     root     5000981078016 Feb  1 10:04 t10.ATA_____HGST_HDN726050ALE610____________________NAG4P4YX____________

Here I have a HGST 5TB disk attached to SATA that I'm trying to passthrough to my VM. There will be other listings here of datastores, and virtual disks that is not relevant.

  1. Attach a physical disk as a RDM
# vmkfstools -z /vmfs/devices/disks/t10.ATA_____HGST_HDN726050ALE610____________________NAG4P4YX____________ "/vmfs/volumes/Samsung 850 Pro/Ubuntu Storage/HGST_RDM_1.vmdk"
  1. Attach the newly created RDM to the VM
  • Go back to the web app, go to the Virtual Machine, and click Edit.
  • Under Add other device, click SCSI controller (If you already have an existing SCSI controller for RDMs, you don't need to create another one). In other words, you can reuse the controllers.
  • Under Add hard disk, click Existing hard disk and select the newly created vmdk file. You'll have to use the Datastore file browser to select the HGST_RDM_1.vmdk file
  • Once the HDD is created, under the new disk e.g. Hard disk 2, expand it and make sure it's using the newly created SCSI controller. You'll have to click on the drop down and select e.g. SCSI controller 1 and SCSI (1:0)
  • Change Disk Mode to Independent - persistent. If the Disk Mode is greyed out then change Disk compatibility from Physical to Virtual.

Note: We use independent persistent mode because RDMs won't work with VMWare disk snapshots. More thorough explanation here.

FAQ

  1. What if I want to change the RDM to another VM?

Simply rm the vmdk file and go through steps 5 and 6 again with the new virtual disk for the new VM.

  1. How can I passthrough multiple disks to the same VM?

Simply follow steps 4 to 6 but with the new disk. You can mount the RDM drives to the same SCSI controller. In other words, the first drive could be on 0:0, and the next drive could be on 0:1. There is a limit of 4 SCSI controllers per VM. nb: Thanks to commenters for pointing out that you don't actually need a separate SCSI controllers per RDM. Personally, I'd have a separate SCSI controller just for RDMs only for cleaniness.

@marshylucas
Copy link

First of all thank you for that tutorial, it helped with my "virtual NAS". Now for the bad news... It worked fine for me up to ESXi 7.0U2d, I don't know if there's a bug in 7.0U3 but my VM won't load the VMDK files made using that method. It looks like ESXi can't get a lock on the VMDK even if there's no lock in place, I checked with "vmfsfilelockinfo -p" and rebooted the ESXi server, nothing will do. Reverting back to ESXi 7.0U2d (SHIFT+R during the ESXi boot process) solved my problem.

@MihataBG
Copy link

First of all thank you for that tutorial, it helped with my "virtual NAS". Now for the bad news... It worked fine for me up to ESXi 7.0U2d, I don't know if there's a bug in 7.0U3 but my VM won't load the VMDK files made using that method. It looks like ESXi can't get a lock on the VMDK even if there's no lock in place, I checked with "vmfsfilelockinfo -p" and rebooted the ESXi server, nothing will do. Reverting back to ESXi 7.0U2d (SHIFT+R during the ESXi boot process) solved my problem.

Same problem with ESXi 7.0U3 here.

@Felixrising
Copy link

Same issue here, ESXi 7.0u3 throws an error about can't lock the rdm device.

In console after trying to power on the VM with RDM:
Failed to power on virtual machine Apollo.local. File system specific implementation of Ioctl[file] failed Click here for more details.

File system specific implementation of OpenFile[file] failed
Failed to lock the file
Cannot open the disk '/vmfs/volumes/601fcbe4-b615a2a2-4e1d-10e7c61984d6/Apollo.local/WDC_WD160EDFZ2D11AFWA0_RDM01.vmdk' or one of the snapshot disks it depends on.
Module 'Disk' power on failed.
Failed to start the virtual machine.

in vmkernel.log
2021-10-21T10:32:15.600Z cpu2:529079)WARNING: RDM3: 123: Error opening device vml.01000000003242474457413844202020202020202020202020574443205744: Busy
2021-10-21T10:32:15.600Z cpu2:529079)FSS: 7377: Failed to open file 'WDC_WD160EDFZ2D11AFWA0_RDM01-rdmp.vmdk'; Requested flags 0x8, world: 529079 [vmx], (Existing flags 0x0, world: 0 [unknown]): Busy

I'm reverting. Thanks for pointing out this is possible @marshylucas

@HyperCriSiS
Copy link

Hi,
Do I understand right that I can make a RAID5 array available with path through for multiple VMs at same time with this tutorial?

Thanks,
Hyper

@variantolog
Copy link

Hi, Do I understand right that I can make a RAID5 array available with path through for multiple VMs at same time with this tutorial?

No. This tutorial about forward ESXi host's local HDD in VM by creating RDM.

@Felixrising
Copy link

Update on broken 7.0U3 broken RDM. There is a VMWare KB open for it now. No fix but mention that maybe re-creating can fix it (it didn't work on mine). https://kb.vmware.com/s/article/86158

@MihataBG
Copy link

MihataBG commented Dec 1, 2021

I can confirm, rebuild does not work.

@maysam2007
Copy link

Thanks a lot.

I used this on ESXi 7 u2 and worked well.

@Felixrising
Copy link

This local RDM issue is fixed in 7.0u3c and following releases c,d e and f.

@sweets9
Copy link

sweets9 commented Sep 4, 2022

I ran some benchmark tests on a single SSD. I found that SCSI SAS, SCSI Paravirtual and SATA all produced the same speeds (all within 1% std deviation). I didnt monitor CPU usage but its all passthrough so should be similar.

Also, for those having issues with different adapters it might be due to the combination of adapters/boot adapters on your VM.

Reference: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-5872D173-A076-42FE-8D0B-9DB0EB0E7362.html

@L3G4CYOne
Copy link

L3G4CYOne commented Oct 27, 2022

Hi. Tried to "passthrough" my 14 TB drive, for some reasons it is shown in VM as 590 GB drive. I've also tested on another 1 TB SSD, it is shown correctly. Anybody experienced such an issue?

SOLUTION: If the disk is big you must use SATA Controller instead of SCSI controller, other than that follow tutorial.

@highruned
Copy link

Leave it to me to have the oddest issue of all, empty dropdowns:

image

The VMDK file is there, correct size, not sure why the above happens... will post back if I figure it out

@mzlumin
Copy link

mzlumin commented Feb 5, 2023

Leave it to me to have the oddest issue of all, empty dropdowns:

image

The VMDK file is there, correct size, not sure why the above happens... will post back if I figure it out

This is a bug in 8.0. check virtualghetto (google) for more Information.
there is a solution via powershell though :)

@mzlumin
Copy link

mzlumin commented Feb 5, 2023

Leave it to me to have the oddest issue of all, empty dropdowns:
image
The VMDK file is there, correct size, not sure why the above happens... will post back if I figure it out

This is a bug in 8.0. check virtualghetto (google) for more Information. there is a solution via powershell though :)

Here is the Link:
https://williamlam.com/2022/12/heads-up-esxi-8-0-host-client-unable-to-attach-existing-virtual-disk-to-vm.html

@k-w-1
Copy link

k-w-1 commented Mar 5, 2023

Here is the Link: https://williamlam.com/2022/12/heads-up-esxi-8-0-host-client-unable-to-attach-existing-virtual-disk-to-vm.html

Only because I was fighting with it for the past month, just wanted to say that the PowerCLI solution worked for me on ESXi 6.7u3.

Note that if like me you had used an old version of PowerCLI that's installed via add/remove, you'll need to remove that and install it via Install-Module -Name VMware.PowerCLI -AllowClobber -Force (and probably subsequently Set-PowerCLIConfiguration -InvalidCertificateAction Ignore since 'aint nobody got time to assign valid certs to their ESXi servers, right?).

@TheNighthawk99
Copy link

Hello, I tried this procedure but:

  • I have only “naa.#######” disks, nothing with t10 prefix;
  • I am unable to to create a raw disk and getting the following error: “… function not implemented”.
    After some investigation, it looks like this works only with 512b formatted HDD, mine is a 4Kn.
    Is there any workaround maybe?

I run ESXI 7.0U3 on the host and the disk is a HGST HC DC520, which a SAS drive, formatted as 4kn as stated before.

@k-w-1
Copy link

k-w-1 commented Mar 25, 2023

* I have only “naa.#######” disks, nothing with t10 prefix;

FWIW, this might be a red herring as I believe t10 is for SATA and naa is for iSCSI/SCSI/SAS.

That said, my drive is in fact 512 (or 512e), so that could be your problem. A quick google seems to indicate your disk might be capable of 512e, although idk how you'd enable/set that.

@TheNighthawk99
Copy link

TheNighthawk99 commented Mar 25, 2023

Edit (see below): the disk was till formatted as VMFS, after cleaning the partition table, I was able to attach to a VM.
The WD Red disks should be 512e (4096 bytes physical, 512 bytes logical) so they work as RDM. HGST/WD with white labels and 4Kn are not supported yet.

Ok, just tried with another HDD, which is a WD Red 3TB (512e). The vmdk is created but I cannot attach the disk to the virtual machine. Sometimes I get the error “Incompatible device backing specified for device X.”, sometimes I get no error but nothing is really attached.

@Zamana
Copy link

Zamana commented May 15, 2023

Hi!

Thank you so much for these instructions.

I have 2 HDDs e 2 SSDs, both sets pertaining to 2 different ZFS pools, and both was recognized and imported by a virtualized TrueNAS CORE 13.

Thanks!

@MSBBZBL
Copy link

MSBBZBL commented Jun 23, 2023

I had so many trouble with RDM after I switched from ESXi 6.7 to 8. I couldn't write any new data on it (2x 8TB WD Red). I had no clue what the problem is. I bought a new Seagate 20TB an copied all my file to this disk. Also with this new disk, my W10 VM could read it - the disk was RAW.

A few minutes ago, I found this post on reddit: https://www.reddit.com/r/esxi/comments/yfwszf/rdm_disk_not_showing_correct_size_on_guest_os/

I changed from

vmkfstools -z /vmfs/devices/disks/t10.ATA_____ST20000NM007D2D3DJ103________________________________ZVT6FYZS "/vmfs/volumes/648cf5df-f7c5604d-fb97-40a6b79c281e/LOST-Server/Exo_X20.vmdk"

to

vmkfstools -r /vmfs/devices/disks/t10.ATA_____ST20000NM007D2D3DJ103________________________________ZVT6FYZS "/vmfs/volumes/648cf5df-f7c5604d-fb97-40a6b79c281e/LOST-Server/Exo_X20.vmdk"

And i worked. The 20TB disk is there, and all files are accessible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment