Fixing "successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero" problem

NUMA node problem.md

What is NUMA (Non-Uniformed Memory Access)

Non-Uniform Memory Access (NUMA) is one of the computer memory design methods used in multiprocessor systems, and the time to access the memory varies depending on the relative position between the memory and the processor. In the NUMA architecture, when a processor accesses its local memory, it is faster than when it accesses the remote memory. Remote memory refers to memory that is connected to another processor, and local memory refers to memory that is connected to its own processor. In other words, it is a technology to increase memory access efficiency while using multiple processors on one motherboard. When a specific processor runs out of memory, it monopolizes the bus by itself, so other processors have to play. , and designate 'access only here', and call it a NUMA node.

1. Check Nodes

lspci | grep -i nvidia
  
01:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060 12GB] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)

The first line shows the address of the VGA compatible device, NVIDIA Geforce, as 01:00 . Each one will be different, so let's change this part carefully.

2. Check and change NUMA setting values

If you go to /sys/bus/pci/devicecs/, you can see the following list:

ls /sys/bus/pci/devices/
  
0000:00:00.0  0000:00:06.0  0000:00:15.0  0000:00:1c.0  0000:00:1f.3  0000:00:1f.6  0000:02:00.0
0000:00:01.0  0000:00:14.0  0000:00:16.0  0000:00:1d.0  0000:00:1f.4  0000:01:00.0
0000:00:02.0  0000:00:14.2  0000:00:17.0  0000:00:1f.0  0000:00:1f.5  0000:01:00.1

01:00.0 checked above is visible. However, 0000: is attached in front.

3. Check if it is connected.

cat /sys/bus/pci/devices/0000\:01\:00.0/numa_node
  
-1

-1 means no connection, 0 means connected.

4. Fix it with the command below.

sudo echo 0 | sudo tee -a /sys/bus/pci/devices/0000\:01\:00.0/numa_node
  
0

It shows 0 which means connected!

5. Check again:

cat /sys/bus/pci/devices/0000\:01\:00.0/numa_node
  
0

That's it!

Thank you!

Thanks for the script -- here's a version for a multi-GPU system:

#!/bin/bash
if [[ "$EUID" -ne 0 ]]; then
  echo "Please run as root."
  exit 1
fi
PCI_ID=$(lspci | grep "VGA compatible controller: NVIDIA Corporation" | cut -d' ' -f1)
#PCI_ID="0000:$PCI_ID"
for item in $PCI_ID
do
  item="0000:$item"
  FILE=/sys/bus/pci/devices/$item/numa_node
  echo Checking $FILE for NUMA connection status...
  if [[ -f "$FILE" ]]; then
    CURRENT_VAL=$(cat $FILE)
    if [[ "$CURRENT_VAL" -eq -1 ]]; then
      echo Setting connection value from -1 to 0.	  
      echo 0 > $FILE
    else
      echo Current connection value of $CURRENT_VAL is not -1.
    fi  
  else
    echo $FILE does not exist to update.
  fi
done

Hi Li-Xin, I suspect the multi-GPU version may not be what you want on a laptop, but both scripts need to run as root. On my machine (Ubuntu 22.04 LTS) using "sudo ./gpu_numa.sh" worked fine. You might need to add execute permissions to the .sh file if you haven't already. * Alex

…

________________________________ From: Li-Xin Li ***@***.***> Sent: Sunday, September 24, 2023 4:46 AM To: lxlofpku ***@***.***> Cc: Mention ***@***.***> Subject: Re: zrruziev/NUMA node problem.md @lxlofpku commented on this gist.

________________________________ Thanks for the script -- here's a version for a multi-GPU system: #!/bin/bash if [[ "$EUID" -ne 0 ]]; then echo "Please run as root." exit 1 fi PCI_ID=$(lspci | grep "VGA compatible controller: NVIDIA Corporation" | cut -d' ' -f1) #PCI_ID="0000:$PCI_ID" for item in $PCI_ID do item="0000:$item" FILE=/sys/bus/pci/devices/$item/numa_node echo Checking $FILE for NUMA connection status... if [[ -f "$FILE" ]]; then CURRENT_VAL=$(cat $FILE) if [[ "$CURRENT_VAL" -eq -1 ]]; then echo Setting connection value from -1 to 0. echo 0 > $FILE else echo Current connection value of $CURRENT_VAL is not -1. fi else echo $FILE does not exist to update. fi done @AlexLandauer<https://github.com/AlexLandauer> Hi Alex: This is very helpful indeed. However, as I try it on my laptop with Ubuntu 22.04, I met a problem. I want to put the script in a file called (e.g.) a.sh then each time when I want to make the change I just need to run "source a.sh". However, it does not work. The reason is that as a regular user my EUID and UID have the same values (none zero) so that at the stage if [[ "$EUID" -ne 0 ]]; then echo "Please run as root." exit 1 fi the code just exists and the terminal is closed. I guess I need to give the code the sudo right (i.e., change the value of EUID to zero) but I do not know how to do it in a.sh file. Can you help? Or is there a better place to put the script than an a.sh file? Best wishes, Li-Xin — Reply to this email directly, view it on GitHub<https://gist.github.com/zrruziev/b93e1292bf2ee39284f834ec7397ee9f#gistcomment-4702386> or unsubscribe<https://github.com/notifications/unsubscribe-auth/A6WJTQB7IZ4WYQVT3GRRIN3X37XM5BFKMF2HI4TJMJ2XIZLTSGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTCNJWGI3TMNJSU52HE2LHM5SXFJTVOBSGC5DF>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

For some reason for me (Gentoo) I had a slightly different path so I had to comment out the string
#item="0000:$item"
Hope that helps someone.

hi, i have a problem, my value after rebooting is restoring to -1
i made:

lspci | grep -i nvidia
06:10.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
06:10.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

after i made changes:
sudo echo 0 | sudo tee -a /sys/bus/pci/devices/0000\:06\:10.0/numa_node

after i check:
cat /sys/bus/pci/devices/0000\:06\:10.0/numa_node i get 0

after reboot and execute again cat /sys/bus/pci/devices/0000\:06\:10.0/numa_node i get -1, how to made this value to be persistent after reboot?

OS ubuntu 22.04, thanks, best regards

Thank you so muuch
This should be first in google,
for SEO
[solved] tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node

It helps just one time, i.e., show o once upon making the change, however, it is not a persistent solution. After rebooting, it still shows "successful NUMA node read from SysFS had negative value (-1),..."

python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2024-05-03 17:14:58.945528: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-03 17:15:00.791206: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-03 17:15:02.618108: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-03 17:15:03.056970: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-03 17:15:03.057654: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

After checking it again, it shows the same error as follows.

cat /sys/bus/pci/devices/0000\:01\:00.0/numa_node

-1

Please help to solve the problem.

Thanks

Thanks a lot for the manual fix!

If this value is -1 in the file, will it cause any problem?