Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Fixing khugepaged CPU usage VMware Workstation

If you run VMware Workstation 11 or above you may encounter high CPU usage from process khugepaged on Ubuntu 15.04+

The fix is to disable transparent hugepages. It seems Ubuntu has it enabled by default.

You can check the current status on your system by running:

cat /sys/kernel/mm/transparent_hugepage/enabled

cat /sys/kernel/mm/transparent_hugepage/defrag

Fedora outputs: always [madvise] never but Ubuntu outputs: [always] madvise never

Fedora seems to not be effected but I havn't tested it myself.

So I suggest not using madvise and just disable it totally.

To disable it run the following commands as root:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

echo never > /sys/kernel/mm/transparent_hugepage/defrag

That will only disable it for the current session.

To have it persistant across reboots I suggest adding this to your rc.local:

# Fix for VMware Workstation 11+ khugepaged.
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Ensure this goes above the line:

exit 0

More info and references:

@gene-olson
Copy link

gene-olson commented Nov 23, 2021

In early 2021, turning off transparent hugepages used to avoid the frozen-vmware, hi-cpu problem on ubuntu 20.04.
It no longer does. Not on my system anyway.

AMD 3900X
64 GB Memory
Nvidia GTX 1660 Ti
Two Samsung EVO 970 drives
LSA SAS2308 SATA drive controller

image

% cat /sys/kernel/mm/transparent_hugepage/enabled 
always madvise [never]
% cat /sys/kernel/mm/transparent_hugepage/enabled 
always madvise [never]
% uname -a
Linux hauser 5.11.0-40-generic #44~20.04.2-Ubuntu SMP Tue Oct 26 18:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

@neves-0
Copy link

neves-0 commented Dec 4, 2021

Same problem here. New workaround, as root:

echo 0 > /proc/sys/vm/compaction_proactiveness

I do not know the other consequences but I have no more kcompactd0 which freeze my VMs.

@gene-olson
Copy link

gene-olson commented Dec 5, 2021

neves-0, your fix is working for me. After 12 hours active CPU on vmware, kcompact has absorbed no CPU at all. It is working GREAT!

THANK YOU THANK YOU!

I have added the following lines to /etc/sysctl.conf so that your fix is applied on boot:

# Fix problem where vmware battles with kcompactd0.
vm.compaction_proactiveness=0

@ckuhtz
Copy link

ckuhtz commented Dec 30, 2021

You guys rock. THANK YOU

@ajgringo619
Copy link

ajgringo619 commented Jan 5, 2022

I ran into this as well on EndeavourOS with the 5.15 zen kernel. After switching to the 5.10 LTS kernel the problem went away. I will definitely try this latest fix as my overall system performance was better with zen; thank you!

@Slapbox
Copy link

Slapbox commented Jan 10, 2022

Ubuntu doesn't use rc.local anymore. What's the correct way to implement these modifications without an rc.local file?

@ckuhtz
Copy link

ckuhtz commented Jan 10, 2022

Script launched via systemd. Pretty straightforward to setup. Seh Google can help.

@Slapbox
Copy link

Slapbox commented Jan 10, 2022

All I've found is suggestions to create an rc.local file and service. Is that really the best way? If so, why was the rc.local file removed past Ubuntu 16.x?

@neves-0
Copy link

neves-0 commented Jan 10, 2022

Best way is to add vm.compaction_proactiveness=0 to the /etc/sysctl.conf file, as said by @gene-olson, and reboot (or use "sysctl -p" to apply the modification without reboot)

@ckuhtz
Copy link

ckuhtz commented Jan 10, 2022

@neves-0 is right, that’s the most efficient way, although I personally have sometimes additional logic around these things and keep them as sysctl calls in a script for that reason.

@Slapbox you need to ask Linux folks why systemd :-)..

@ajgringo619
Copy link

ajgringo619 commented Jan 11, 2022

Testing went well with the Arch linux-zen kernel; thanks again.

For Arch users, create a .conf file in /etc/sysctl.d/.

@shuhaowu
Copy link

shuhaowu commented Mar 2, 2022

Setting vm.compaction_proactiveness to be 0 doesn't solve the problem for me...

@xeyownt
Copy link

xeyownt commented May 25, 2022

Same for me.

None of the above works for me. Did the hugepage_transparent things, the compaction_proactiveness, both directly, in etc/, and in vmware *.vmx file. Checking the various variable from /proc and /sys returns the expected value. Still, kcompactd0 kicks in. It is slightly less severe than it uses to be, but still it freezes the guest to death.

I'm thinking about recompiling the kernel and disabling that kcompactd0 thing altogether.

Host Debian 5.16.18-1 (2022-03-29) x86_64 GNU/Linux, 16GB RAM.
VMWare 16.2.3 build-19376536
Guest Windows 10 (64-bit), 8GB RAM allocated.

@gene-olson
Copy link

gene-olson commented May 25, 2022

When I first set only:
vm.compaction_proactiveness=0
back in December it appeared a complete fix, just like a number of previous (but different) fixes did.

However I now see some regular pauses, but much less serious ones. I might lock up occasionally for 10-30 seconds, but generally the problem clears after a few iterations and the system is fine again. It's an annoyance, for sure, but it doesn't block my work.

In case it lends any light on the problem, I am using a 12-core AMD with 64 GB of memory on Ubuntu 20.04. I have seen this problem in Windows VMs with 16 GB of virtual RAM, and 4 or 6 virtual CPUs.

I fear that this fix, like all the others, will gradually become less effective, finally making VMWare Workstation unusable. Previously this problem became so bad I temporarily switched to VirtualBox, and never saw the problem there.

@exeq89
Copy link

exeq89 commented Jun 3, 2022

i can confirm what gene-olson says.
i started get freezes even with compaction_proactiveness is 0.

@lijx10
Copy link

lijx10 commented Jun 8, 2022

Comfirm that win10 guest freezes and kcompactd0 kicks in with compaction_proactiveness=0.

Tested with:
Ubuntu 22.04 kernel 5.15.35 / 5.16.20
VMWare 16.2.3 / 16.2.0 / 16.1.2
Win10 Guest, 4GB / 6GB memory allocated

@exeq89
Copy link

exeq89 commented Jun 12, 2022

i have lowered the memory for 3d graphics and its better.
my config is:
ryzen 5 5600g / 32 gb ram
guest is win10 with 6 cpu and 16 gb ram.
lowered graphics to 256mb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment