Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to passthrough SATA drives directly on VMWare ESXI 6.5 as RDMs

How to passthrough SATA drives directly on VMWare EXSI 6.5 as RDMs

There aren't many tutorials about this, the only tutorials I've found were about passing through entire PCIe cards to VMs, or refered to old ESXI versions (below 6.5) that used a more comprehensive desktop client instead of the web app. In v6.5, the web app was introduced and the desktop client was deprecated. You used to be able to setup RDMs in the desktop client, but with the introduction of the web console, this is no longer the case. This tutorial shows you how to pass SATA HDDs to the virtual machine on VMWare ESXI 6.5. This tutorial is partially based on VMWare's own KB and the now deprecated Forza IT blog post.

A word about VMWare ESXI 6.7

There is now an option while editing your VM's settings to add a New raw disk when you click Add hard disk. I have not personally tried it yet, but the comments below have confirmed it works.

Summary

We attach the SATA HDDs as a RDM (Raw Device Mapper) into an existing virtual disk in the command line, then on the web app, we attach a new SCSI controller to the VM, and attach the newly created RDM as an "existing HDD" to the newly created SCSI controller.

Prerequisite knowledge

If you're new to EXSI, you should know a few things:

  • A Datastore is basically this separate layer that sits between a physical device/disk and a virtual disk. A datastore can max the storage of a physical disk, or not. A datastore can have many virtual disks. Virtual disks are what a virtual machine will stores files on. In other words Physical Disk > Datastore > Virtual Disks > Your filesystem e.g. EXT4
  • RDM aka Raw Device Mapper is a pointer file that acts like a virtual disk but instead maps directly to a physical disk. RDMs on EXSI 6.5 have to be created in the command line.

Instructions

  1. Before you get started, make sure you've got a VM setup, and running. In my example, I have a virtual machine and virtual disk both conveniently named Ubuntu Storage

  2. In EXSI's web interface, log in, go to the home page. Click on Actions > Services > Enable Secure Shell (SSH).

  3. Open a SSH session to the ESXi/ESX host.

# ssh root@YOUR_EXSI_IP -p 22
  1. Run this command to list the disks that are attached to the ESXi host:
# ls -l /vmfs/devices/disks

It should look like this. We care about the physical disks with the prefix t10. As mentioned in the comments, it may not necessarily start with t10. You can determined the name/path by going to the web console then Storage, then Device, click on the device you're setting up as RDM, and then copying the path.

[root@localhost:~] ls -l /vmfs/devices/disks
total 12221184373
-rw-------    1 root     root     5000981078016 Feb  1 10:04 t10.ATA_____HGST_HDN726050ALE610____________________NAG4P4YX____________

Here I have a HGST 5TB disk attached to SATA that I'm trying to passthrough to my VM. There will be other listings here of datastores, and virtual disks that is not relevant.

  1. Attach a physical disk as a RDM
# vmkfstools -z /vmfs/devices/disks/t10.ATA_____HGST_HDN726050ALE610____________________NAG4P4YX____________ "/vmfs/volumes/Samsung 850 Pro/Ubuntu Storage/HGST_RDM_1.vmdk"
  1. Attach the newly created RDM to the VM
  • Go back to the web app, go to the Virtual Machine, and click Edit.
  • Under Add other device, click SCSI controller (If you already have an existing SCSI controller for RDMs, you don't need to create another one). In other words, you can reuse the controllers.
  • Under Add hard disk, click Existing hard disk and select the newly created vmdk file. You'll have to use the Datastore file browser to select the HGST_RDM_1.vmdk file
  • Once the HDD is created, under the new disk e.g. Hard disk 2, expand it and make sure it's using the newly created SCSI controller. You'll have to click on the drop down and select e.g. SCSI controller 1 and SCSI (1:0)
  • Change Disk Mode to Independent - persistent. If the Disk Mode is greyed out then change Disk compatibility from Physical to Virtual.

Note: We use independent persistent mode because RDMs won't work with VMWare disk snapshots. More thorough explanation here.

FAQ

  1. What if I want to change the RDM to another VM?

Simply rm the vmdk file and go through steps 5 and 6 again with the new virtual disk for the new VM.

  1. How can I passthrough multiple disks to the same VM?

Simply follow steps 4 to 6 but with the new disk. You can mount the RDM drives to the same SCSI controller. In other words, the first drive could be on 0:0, and the next drive could be on 0:1. There is a limit of 4 SCSI controllers per VM. nb: Thanks to commenters for pointing out that you don't actually need a separate SCSI controllers per RDM. Personally, I'd have a separate SCSI controller just for RDMs only for cleaniness.

@sIRwa2

This comment has been minimized.

Copy link

@sIRwa2 sIRwa2 commented Sep 6, 2017

thanks for this very helpful guide. worked like a charm.

Q: last sentence, why would you need to create a new SCSI controller for each new disk? ESXI only allows for 4 max.

comment @ 6. you wrote "Add Another Device" at the 3th bullet. should that be: " Add hard disk" ?

@madmunki

This comment has been minimized.

Copy link

@madmunki madmunki commented Sep 30, 2017

Q: last sentence, why would you need to create a new SCSI controller for each new disk?

I have no idea why he did that. I setup a FreeNAS server, and its OS is a VM disk on a SCSI Controller, so I added ONE new SCSI Controller to attach 3 RDM drives to be used as NAS. Works fine. A disk controller is made to support multiple drives, I could have even attached the RDMs to the same one the OS is one without issues, but wanted to separate these for possible performance issues.

@NextBlaubeere

This comment has been minimized.

Copy link

@NextBlaubeere NextBlaubeere commented Nov 4, 2017

How can i remove the maping again?

@epardee

This comment has been minimized.

Copy link

@epardee epardee commented Nov 28, 2017

In my case, the physical disks attached to my ESXi host didn't start with t10. I determined the name by selecting the Device within the Storage section and copying the Path.

@TimeToFixStuff

This comment has been minimized.

Copy link

@TimeToFixStuff TimeToFixStuff commented Nov 29, 2017

Very nice tutorial. I have one question - Can I use that one RDM drive in multiple VM's, which could be running at the same time? Would I need to create additional VMDK files or use the same one for all the VM's that use this shared space?

@coeusite

This comment has been minimized.

Copy link

@coeusite coeusite commented Feb 19, 2018

Could anyone please confirm whether drives are physical RDMs or virtual RDMs? Or is disk’s SMART info sent to the VM?

@Pac23

This comment has been minimized.

Copy link

@Pac23 Pac23 commented Feb 23, 2018

@TimeToFixStuff additional vmdk file would be required to pass to another vm.I tried using the same rdm but it didn't work.

@Hengjie

This comment has been minimized.

Copy link
Owner Author

@Hengjie Hengjie commented Apr 29, 2018

Thanks everyone, I've incorporated your comments into the guide.

@kininwong

This comment has been minimized.

Copy link

@kininwong kininwong commented Jun 13, 2018

Does this method work on esxi 6.7?

i got the error below.
Failed to create virtual disk: The system cannot find the file specified (25).

@AGSPhoenix

This comment has been minimized.

Copy link

@AGSPhoenix AGSPhoenix commented Jul 10, 2018

Under Step 6, the Disk mode dropdown was greyed out until I set the Disk compatibility from Physical to Virtual.

@timanttikuutio

This comment has been minimized.

Copy link

@timanttikuutio timanttikuutio commented Jul 31, 2018

@kininwong Same thing happened to me. And i'm using ESXi 6.5

@danzirulez

This comment has been minimized.

Copy link

@danzirulez danzirulez commented Sep 23, 2018

Hi All, does this work with devices already having data on them? I am struggling with a PCI-E card that is in pass through mode, but it seems to be faulty. And if disks can be assigned directly to the VM, I could just skip the card altogether.

@AnotherRandomTech

This comment has been minimized.

Copy link

@AnotherRandomTech AnotherRandomTech commented Oct 2, 2018

Notes if people have disks more than 2TBs or cannot access the full disk. and are using it as a standalone install and not with vCentre

  1. Set RdmFilter.HbalsShared to TRUE in Advanced Configuration
  2. on the virtual machine add a SATA controller and attach the vmdk to this. you will loose the SMART data though i believe
@danzirulez

This comment has been minimized.

Copy link

@danzirulez danzirulez commented Oct 23, 2018

Hi All, does this work with devices already having data on them? I am struggling with a PCI-E card that is in pass through mode, but it seems to be faulty. And if disks can be assigned directly to the VM, I could just skip the card altogether.

Just for the record, I have passed through 3 disks, 2x 2 Tb red nas WD drives with ZFS mirror config to XigmaNAS Freebsd based VM, and a UFS storage of 3 Tb, also red nas WD disk, with no issues to the same VM.

Smart is lost in the process as someone else noted. Otherwise it seems stable.

@hatefuel

This comment has been minimized.

Copy link

@hatefuel hatefuel commented Jan 13, 2019

I am trying to use these steps to add a secind drive to Xpenology and no matter what configuration I try i get the error Incompatible device backing specified for device (device number)

I've tried using vmware 5.5, 6.5 and 6.7 and no matter what I do I get the error above. Any advice on this error message?

@bryanfritz

This comment has been minimized.

Copy link

@bryanfritz bryanfritz commented Feb 11, 2019

@kininwong Same thing happened to me. And i'm using ESXi 6.5

I found the issue with this was that you have to have a directory in an EXISTING Datastore. The file created is just the pointer to the RDM and needs to be located on an existing datastore. Open up ESXi and open up a esixting datastore, click on the file browser and create a folder in the datastore. Point your vmkfstools command to the directory you just created and the Pointer Files will be created there even though the RDM are not actually residing there.

@k-vladyslav

This comment has been minimized.

Copy link

@k-vladyslav k-vladyslav commented Feb 23, 2019

Small trick (which I found accidently) - if you would name your mapping something like /vmfs/volumes/DATASTORE/VIRTUAL_DISK/disk-rdm.vmdk that wont work and will end you with error on starting VM like "The value was rejected by rule "Virtual SCSI or NVMe Devices"
Fix is - avoid having word -rdm in the end of filename, and name it like /vmfs/volumes/DATASTORE/VIRTUAL_DISK/XXX_RDM_1.vmdk
more info here:
https://communities.vmware.com/thread/491979

@joeferg

This comment has been minimized.

Copy link

@joeferg joeferg commented Feb 24, 2019

I am getting the same error as @hatefuel, anyone have a suggestion ?

I have 2 10tb drives
created an RDM for each one after making a folder on the datastores to hold the vmdk file
I tried making a scsi controller, all 3 options, this part worked
But when attaching the existing drive and pointing to the vmdk file, I get the error "Incompatible device backing specified for device (device number)"

Went back in a few times, tried each of teh scsi controller types
Changed my flag for RdmFilter.HbalsShared to TRUE and tried sata controller, no luck

Anyone have any other suggestions ?
ESXi 6.7

@zaped212

This comment has been minimized.

Copy link

@zaped212 zaped212 commented Mar 12, 2019

Just a heads up. I installed esxi a few days ago and just got my HDDs added to one of my VMs this morning.
I was going to follow this tutorial until I noticed that under the VMs add hard drive drop down that there was a "add RDM" option.

I clicked that and it showed me the 3 drives I was wanting to add.

Note: it does not show any drives that are configured as datastores.

@kennyparsons

This comment has been minimized.

Copy link

@kennyparsons kennyparsons commented Mar 22, 2019

I have the same problem as @joeferg and am at a loss of what to do.

@swarajthakur

This comment has been minimized.

Copy link

@swarajthakur swarajthakur commented Apr 6, 2019

@kennyparsons @joeferg @hatefuel If you are using the referenced disk as datastore as well, then simply removing the datastore resolves this problem. ESXi won't allow to use the same disk as datastore.

@joeferg

This comment has been minimized.

Copy link

@joeferg joeferg commented Apr 10, 2019

@swarajthakur and others thanks. I was getting the error "Incompatible device backing specified for device (device number)" I had my vdmk files on the datastores I created for the drives, meaning the vdmk was pointed to a datastore that was supposed to be passed through. Instead I needed to host the vdmk files on a path on a datastore that was not on the physical disk I was trying to passthrough to.

@kennyparsons @hatefuel
Cleaning that up worked. I setup my vdmk files in separate folders on the datastore.

Now I am having a new issue, and I am guessing, because I had so many problems with this first part, I am missing something simple.

When I added the drives I added them once on individual sata controllers, and then on the same, but each time, when I get into the VM I set these 2 drives up for, I am only seeing one drive.

Anyone have any idea what I am messing up on this part?

@MrAmos123

This comment has been minimized.

Copy link

@MrAmos123 MrAmos123 commented Apr 13, 2019

Anyone able to help? I'm getting this:

Failed to create virtual disk: The system cannot find the file specified (25).

I'm running:

vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD4003FFBX2D6XXXXXX0__________________VXXXXXXR____________ "/vmfs/volumes/HDD\ Data\ Store\ 2\ -\ Toshiba\ HP/FreeNAS/WDRED4TB_RDM_1.vmdk"
vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD4003FFBX2D6XXXXXX0__________________VXXXXXXS____________ "/vmfs/volumes/HDD\ Data\ Store\ 2\ -\ Toshiba\ HP/FreeNAS/WDRED4TB_RDM_2.vmdk"

I think I'm doing something wrong... But I'm not sure...

https://i.imgur.com/hM8azaX.png - VM path
https://i.imgur.com/cNPYidC.png - Disk list

EDIT: Figured it out...

It didn't like the symlink for some reason. I cd'd to the VM folder and then ran it without the path.

[root@ESXi:~] cd /vmfs/volumes/HDD\ Data\ Store\ 2\ -\ Toshiba\ HP/FreeNAS/
[root@ESXi:/vmfs/volumes/58c458be-beb05bbe-XXXX-a0369fa25e38/FreeNAS] vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD4003FFBX2D6XXXXX0__________________VXXXXXR____________ "WDRED4TB_RDM_1.vmdk"

[root@ESXi:/vmfs/volumes/58c458be-beb05bbe-XXXX-a0369fa25e38/FreeNAS] vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD4003FFBX2D6XXXXX0__________________VXXXXXS____________ "WDRED4TB_RDM_2.vmdk"

[root@ESXi:/vmfs/volumes/58c458be-beb05bbe-XXXX-a0369fa25e38/FreeNAS] ls
FreeNAS-flat.vmdk         FreeNAS.nvram             FreeNAS.vmdk              FreeNAS.vmsd              FreeNAS.vmx               WDRED4TB_RDM_1-rdmp.vmdk  WDRED4TB_RDM_1.vmdk       WDRED4TB_RDM_2-rdmp.vmdk  WDRED4TB_RDM_2.vmdk       vmware.log
@Jaharmi

This comment has been minimized.

Copy link

@Jaharmi Jaharmi commented May 3, 2019

I just got this working on a PowerEdge R530 with an installation from the Dell ESXi 6.7 iso. I have two SAS drives in RAID-1 on a PERC H730 and another six that are Non-RAID disks. The RAID-1 serves as the ESXi datastore. I installed FreeNAS 11.2 in a VM there.

I followed the directions as described. I used a second SCSI controller. I think I set “RdmFilter.HbalsShared” somewhere along the line, but I’d have to confirm.

All six non-RAID disks are seen with SMART enabled and serial numbers that match what I see in iDRAC. The have “da#” disk identifiers in FreeNAS. They do not show capacity, description, or SMART extra options. Adv Power Management and Acoustic Level are listed as DISABLED.

@ravibowman

This comment has been minimized.

Copy link

@ravibowman ravibowman commented May 18, 2019

I was able to follow this to load a couple of 10TB WD RED drives to my PLEX VM however I wanted to access the data already present on the HDDs. What I got instead were HDDs that were mounted with partitions that didnt match the physical drives. I took out one of the HDDs and mounted it via a usb dock to my PC and it was fine will all the data that was supposed to be there. How do I get this HDD with data already on it mounted to browse the data on it. I mounted the drives on a SATA controller.

Thanks,

@Hengjie

This comment has been minimized.

Copy link
Owner Author

@Hengjie Hengjie commented Jun 20, 2019

Thanks @bryanfritz and @k-vladyslav I've incorporated your info into the tutorial. I've also updated it mentioning ESXI 6.7 support for raw disks (though untested by me atm)

@juan9999

This comment has been minimized.

Copy link

@juan9999 juan9999 commented Oct 12, 2019

Above steps worked great for me in 6.7 NOT as a RAW drive. That option is greyed out.

I am running XPEnology under ESXI but if I try to access S.M.A.R.T. information in the VM, I get an error.

If I I SSH'd into my ESXi server and ran the following command to read the S.M.A.R.T. data from the device below I get what look like weird results.

My questions:
1- Would adding as a RAW drive solve my issue within the VM?

2- Are my results from within ESXi normal?

Thanks!

esxcli storage core device smart get -d device

I got the following info pretty much for all my 4 brand new drives:
Parameter Value Threshold Worst


Health Status OK N/A N/A
Media Wearout Indicator N/A N/A N/A
Write Error Count 100 1 100
Read Error Count 83 44 78
Power-on Hours 100 0 100
Power Cycle Count 100 20 100
Reallocated Sector Count 100 10 100
Raw Read Error Rate 83 44 78
Drive Temperature 42 0 42
Driver Rated Max Temperature 58 40 58
Write Sectors TOT Count 100 0 253
Read Sectors TOT Count 100 0 253
Initial Bad Block Count N/A N/A N/A

@Marfjeh

This comment has been minimized.

Copy link

@Marfjeh Marfjeh commented Oct 30, 2019

Worked great, as adding raw disks was grayed out for some reason, this method worked for me fine on 6.7

@JonasAlfredsson

This comment has been minimized.

Copy link

@JonasAlfredsson JonasAlfredsson commented Nov 19, 2019

@Marfjeh According to this article from VMware it seem like local disks are not eligible for RDM by default, and this "workaround" method is necessary.

@Marfjeh

This comment has been minimized.

Copy link

@Marfjeh Marfjeh commented Nov 22, 2019

@Marfjeh According to this article from VMware it seem like local disks are not eligible for RDM by default, and this "workaround" method is necessary.

Ah. that explains. thanks!

@keitetran

This comment has been minimized.

Copy link

@keitetran keitetran commented Feb 19, 2020

Other user use Synology install
Skip step 6 then switch to thread https://xpenology.com/forum/topic/13061-tutorial-install-dsm-62-on-esxi-67/

@rjafarkhani

This comment has been minimized.

Copy link

@rjafarkhani rjafarkhani commented Mar 15, 2020

I'm getting the below error performing this on 6.7u3.
Failed to create virtual disk: Operation not permitted (65545).

When I try to do the raw disk method described, the option is grayed-out in the console. Any thoughts?

UPDATE:
I figured it out. I was adding the vmdk to the root /vmfs/volumes/ instead of to my datastore. Changed this to /vmfs/volumes/datastore1/DISKS and all is well.

@ArmandH

This comment has been minimized.

Copy link

@ArmandH ArmandH commented Apr 4, 2020

Anybody knows what are the best settings to use for the SCSI controller , i have seen Disk mode - Independent Persistent, but what about LSI Logic SAS, LSI Logic Parallel or Vmware Paravirtual which one should i used for best speed? and SCSI Bus Sharing, none, virtual or physical? Thanks for the help

@nem8

This comment has been minimized.

Copy link

@nem8 nem8 commented Apr 10, 2020

Step 6 is somewhat confusing for me. There is no "Disk Mode" or "Disk compatibility" under SCSI Controller or Hard disk "n".
Im thinking this changed between 6.5 and 6.7?

Under Hard disk "n" i can set Compatibility Mode. This is greyed out and it does not matter which type of SCSI Bus Sharing i choose. It is set to Physical.
Under SCSI Controller, i can set the type of controller and SCSI Bus Sharing, the latter have choices "None", "Physical", "Virtual"

Not sure if you strictly want to keep the guide to 6.5, if not then maybe you can clarify that step somewhat for people using 6.7?
Anyways, i appreciate this guide, it was hard to find any walk through on how to accomplish this. Thanks to your effort it was made a bit easier. :)

@Strider3000

This comment has been minimized.

Copy link

@Strider3000 Strider3000 commented Apr 12, 2020

A heads-up for ESXI 6.7u3 1204 trying this method with large (8TB) drives for window VMs. Disks either need to be pre-formatted on a physical computer using GPT for all the space to show on the VM, or you need to use a SATA controller instead of an iSCSI controller as shown in the tutorial. If you don’t do either of these steps, then the drive will not show all the drive’s space to the VM (my drives were showing as 1308 GB).

@jakubsuchybio

This comment has been minimized.

Copy link

@jakubsuchybio jakubsuchybio commented May 8, 2020

My problem:
I have 2 disks. One 6TB NTFS with files on it. Second 5TB vmfs6 empty.
I need to mount NTFS 6TB as RDM to copy files to vmfs6 so that I can migrate the 6TB to vmfs6 too.

However after following the guide it loaded up the disk in VM, but I couldn't browse it. It showed as 100% used in Disk Manager.
Solution?
I was using new iSCSI controller for that RDM disk. That is wrong if you need to read data from it. You need to use SATA Controller instead.

@nonyhaha

This comment has been minimized.

Copy link

@nonyhaha nonyhaha commented May 19, 2020

Hello!
I am trying to pass through some drives. I have 4x wd red 4tb on another server running windows storage spaces. I have a storage pool on these 4 disks.
I am able to map the drive in esxi. I start the windows machine. In computer management, I can see the ATA disk in disk drives, but nowhere else. Not in disk management, not in storage spaces. Does anybody know it if it possible to push this storage pool to the virtual machine?
many thanks!

@Pac23

This comment has been minimized.

Copy link

@Pac23 Pac23 commented Jul 8, 2020

After using this solution for almost 2+ years i have come to the conclusion that using a pcie sata card/controller is just easier and less troubleshooting is involved over the long run.Paying 20$ for a raid card is worth it.

The cons of directly passing the disks to the vm are -

  1. if you disk has read/write or some other issues vmware wont often let you edit the vm settings even throw a error sometimes.
  2. SMART data is non existent.
  3. Hotswapping disks wont be possible.
  4. Assigning disks from one vm to another is just way easier.
@cucac

This comment has been minimized.

Copy link

@cucac cucac commented Jul 26, 2020

I made a video https://www.youtube.com/watch?v=EGGA84N4b0c claiming that with one SATA controller on MB with it won't work with FreeNAS in VM setting. And someone asked in comment section if I have heard of RDM in ESXi and said I am wrong. FreeNAS can work with just one controller. OK Maybe I am wrong. but let me ask how reliable is RDM? what happen if you have to reinstall ESXi? would the RDM data be intact after the re installation? can you recover the data after the re-installation of ESXi?

@JonasAlfredsson

This comment has been minimized.

Copy link

@JonasAlfredsson JonasAlfredsson commented Jul 26, 2020

@cucac RDM stands for Raw Device Mapping, which basically means that you forward the device directly to the OS running inside the VM.

This has the advantage that you get direct access to the drive from withing the guest OS, and if you don't do anything fancy with the disk in there it should be as easy as just pulling the drive and mounting it in another computer and all the data is there. If you have FreeNAS running inside a VM in ESXi, and just give it RDMs, it should behave (almost) the same as if you installed it as the only OS on the machine and mounted all the disks normally.

The drawback with this method is that you can not share the disk between VMs inside ESXi, since the "forwarding" of the drive means that it is "locked/managed" by the OS inside the single VM.

@kidhasmoxy

This comment has been minimized.

Copy link

@kidhasmoxy kidhasmoxy commented Oct 15, 2020

Has anyone tried to use the option in 6.7 to add new raw device? I see that it's possible but not clear on whether I follow these instructions at all or if I can go right to updating the VM? The option is greyed out for me initially.

@PenguinPaws

This comment has been minimized.

Copy link

@PenguinPaws PenguinPaws commented Dec 6, 2020

Has anyone tried to use the option in 6.7 to add new raw device? I see that it's possible but not clear on whether I follow these instructions at all or if I can go right to updating the VM? The option is greyed out for me initially.

Apparently it's greyed out because you don't have a controller that supports RDM commands, so instead we use vmkfs tools to create a vmdk to accomplish this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment