wmealing/nmi-interrupts.txt

## nmi-interrupts.txt
A non-maskable interrupt (NMI) is an interrupt type which differs from standard interrupt mechanism by enforcing attention from the interrupt processor (usually the CPU).  This solution discusses an NMI is in more depth and how they are handled.

### What is an Interrupt ? ###

Modern systems architecture has created tightly coupled connect between system components.  Work for components can be handed off to a component for completion.  Rather than wait for the component the main CPU can be tasked to do other pending work.

When the component has completed its work it will raise a signal to the main processor.  The main processor considers this signal an "interrupt", as the current work on the CPU will be interrupted immediately   Each component has a number assigned to it.

### Why "mask" an interrupt ? ###

On a single CPU system, each interrupt is (virtually) routed to a single CPU, and it is primarily responsible for dealing with interrupts generated from every hardware component.  Multiple CPU systems allow for different CPU's to handle interrupts.  This can distribute the workload across CPU's.  For this to take place all CPU's in the system must be aware of the interrupts that they will be handling.  The operating system will create a blacklist of interrupts so that there is no overlap in different CPU's dealing with the same hardware interrupt.

### What is an NMI? ###

A Non maskable interrupt (NMI) is an interrupt that is unable to be ignored/masked out by standard operating system mechanisms.  A non-maskable interrupt (NMI) cannot be ignored, and is generally used only for critical hardware errors however recent changes in behavior has added additional functionality of:

1. Physical NMI button.

This can be used to signal the operating system when other standard input mechanisms (keyboard, ssh, network) have ceased to function.  It can be used to create an intentional panic for additional debugging.

FIXME: what does it look like, how do I know ? how do i set it up ?

2. A Watchdog-like software on the system that monitors for perceived system hangs

The NMI watchdog monitors system interrupts and sends an NMI if the system appears to have hung. On a normal system hundreds of device and timer interrupts are received per second. If there are no interrupts in a 30 second interval*, the NMI watchdog assumes that the system has hung and sends an NMI to the system to trigger a kernel panic or restart.


#### How an NMI watchdog works ####

A standard system level watchdog waits for regular events to fire and reboots the machine if no event is received within a designated timeframe.  The NMI watchdog is no different. When using the NMI watchdog the system generates periodic NMI interrupts, and the kernel can monitor whether any CPU has locked up and print out debugging messages if so.

### Enabling NMI Watchdog ###

The Red Hat Enterprise Linux 6 kernel is built with NMI watchdog support on currently supported x86 and x86-64 platforms.

To ensure NMI is being used:

* For SMP machines and Single processor systems with an IO-APIC use nmi_watchdog=1.

* For Single processor systems without an IO-APIC use nmi_watchdog=2.


#### Confirmation of NMI watchdog working ####

Boot the system with the the parameter as stated above and check the /proc/interrupts file for the "NMI count" line.  This value should be non zero and increase over time.  If the value is zero and does not increase over time the wrong NMI watchdog parameter has been used, change

  If it is still zero then log a problem, you probably have a processor that needs to be added to the nmi code.

Here is an example from /etc/grub.conf for systems which utilize the GRUB boot loader:

~~~
title Red Hat Enterprise Linux Server (2.6.32-358.6.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-358.6.1.el6.x86_64 ro root=/dev/mapper/vg_worklaptop-lv_root crashkernel=auto  rd_LVM_LV=vg_worklaptop/lv_root rhgb quiet nmi_watchdog=1
	initrd /initramfs-2.6.32-358.6.1.el6.x86_64.img
~~~

To determine if the NMI watchdog was properly activated, check the /proc/interrupts file. The NMI interrupt should display a non-zero value. If the NMI interrupt displays a zero, alter the nmi_watchdog value, restart the system, and examine this file again. If a zero is still displayed, then the processor in the test system is not supported by the NMI watchdog code.

The output, when functioning correctly, should look similar to the following:

~~~
[root@work-laptop wmealing]# cat /proc/interrupts | grep ^NMI
NMI:        861        636        377        357   Non-maskable interrupts

~~~

Each processor core has an NMI count.  These should all be increasing over time.  The above example is a quad core system.

####  System wide NMI settings ####

The NMI settings can be configured at runtime by using the sysctl interface.

In the /etc/sysctl.conf, to enable, set:
~~~
kernel.nmi_watchdog = 1
~~~

To disable, set:
~~~
kernel.nmi_watchdog = 0
~~~

Note that this does not enable the functionality, the kernel parameter is required to correctly enable the NMI watchdog.

#### unknown_nmi_panic ####

A feature was introduced in kernel 2.6.9 which helps to make easier the process of diagnosing system hangs on specific hardware.
The feature utilizes the kernels behavior when dealing with unknown NMI sources.  The behavior is to allow it to panic, rather than handle the unknown nmi source.  This feature cannot be utilized on systems that also use the NMI Watchdog or some oprofile (and other tools that use performance metric  features as both of these also make use of the undefined NMI interrupt. If unknown_nmi_panic is activated with one of these features present, it will not work.

Note that this is a user-initiated interrupt which is really most useful for helping to diagnose a system that is experiencing system hangs for unknown reasons.

To enable this feature, set the following system control parameter in the /etc/sysctl.conf file as follows:

~~~
kernel.unknown_nmi_panic = 1
~~~

To disable, set:
~~~
kernel.unknown_nmi_panic = 0
~~~

Once this change has taken effect, a panic can be forced by pushing the system's NMI switch. Systems that do not have an NMI switch can still use the NMI Watchdog feature which will automatically generate an NMI if a system hang is detected.

#### panic_on_unrecovered_nmi ####

Some systems may generate an NMI based on vendor configuration, such as power management, low battery etc.  It may be important to set this if your system is generating NMI's in a known-working environment.

To enable this feature, set the following system control parameter in the /etc/sysctl.conf file as follows:

~~~
kernel.panic_on_unrecovered_nmi = 1
~~~

To disable, set:
~~~
kernel.panic_on_unrecovered_nmi = 0
~~~


#### panic_on_io_nmi ####

This setting was only available in Red Hat Enterprise Linux 6.   When set, this will cause a kernel panic when the kernel receives an NMI caused by an Input/Output error.

### Common NMI error messages ###

Certain messages are printed to the system logs when an NMI is received. The most common of these are:

* Red Hat Enterprise Linux 5

>    "NMI received for unknown reason 00"

* Red Hat Enterprise Linux 6

>    "Uhhuh. NMI received for unknown reason b0"

### References ###

Kernel documentation:

Slackware NMI Documentation: http://slacksite.com/slackware/nmi.html
https://access.redhat.com/knowledge/sources/source_rpms/kernel-2.6.32-358.6.1.el6/tree/Documentation/nmi_watchdog.txt
	A non-maskable interrupt (NMI) is an interrupt type which differs from standard interrupt mechanism by enforcing attention from the interrupt processor (usually the CPU). This solution discusses an NMI is in more depth and how they are handled.

	### What is an Interrupt ? ###

	Modern systems architecture has created tightly coupled connect between system components. Work for components can be handed off to a component for completion. Rather than wait for the component the main CPU can be tasked to do other pending work.

	When the component has completed its work it will raise a signal to the main processor. The main processor considers this signal an "interrupt", as the current work on the CPU will be interrupted immediately Each component has a number assigned to it.

	### Why "mask" an interrupt ? ###

	On a single CPU system, each interrupt is (virtually) routed to a single CPU, and it is primarily responsible for dealing with interrupts generated from every hardware component. Multiple CPU systems allow for different CPU's to handle interrupts. This can distribute the workload across CPU's. For this to take place all CPU's in the system must be aware of the interrupts that they will be handling. The operating system will create a blacklist of interrupts so that there is no overlap in different CPU's dealing with the same hardware interrupt.

	### What is an NMI? ###

	A Non maskable interrupt (NMI) is an interrupt that is unable to be ignored/masked out by standard operating system mechanisms. A non-maskable interrupt (NMI) cannot be ignored, and is generally used only for critical hardware errors however recent changes in behavior has added additional functionality of:

	1. Physical NMI button.

	This can be used to signal the operating system when other standard input mechanisms (keyboard, ssh, network) have ceased to function. It can be used to create an intentional panic for additional debugging.

	FIXME: what does it look like, how do I know ? how do i set it up ?

	2. A Watchdog-like software on the system that monitors for perceived system hangs

	The NMI watchdog monitors system interrupts and sends an NMI if the system appears to have hung. On a normal system hundreds of device and timer interrupts are received per second. If there are no interrupts in a 30 second interval*, the NMI watchdog assumes that the system has hung and sends an NMI to the system to trigger a kernel panic or restart.


	#### How an NMI watchdog works ####

	A standard system level watchdog waits for regular events to fire and reboots the machine if no event is received within a designated timeframe. The NMI watchdog is no different. When using the NMI watchdog the system generates periodic NMI interrupts, and the kernel can monitor whether any CPU has locked up and print out debugging messages if so.

	### Enabling NMI Watchdog ###

	The Red Hat Enterprise Linux 6 kernel is built with NMI watchdog support on currently supported x86 and x86-64 platforms.

	To ensure NMI is being used:

	* For SMP machines and Single processor systems with an IO-APIC use nmi_watchdog=1.

	* For Single processor systems without an IO-APIC use nmi_watchdog=2.


	#### Confirmation of NMI watchdog working ####

	Boot the system with the the parameter as stated above and check the /proc/interrupts file for the "NMI count" line. This value should be non zero and increase over time. If the value is zero and does not increase over time the wrong NMI watchdog parameter has been used, change

	If it is still zero then log a problem, you probably have a processor that needs to be added to the nmi code.

	Here is an example from /etc/grub.conf for systems which utilize the GRUB boot loader:

	~~~
	title Red Hat Enterprise Linux Server (2.6.32-358.6.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-358.6.1.el6.x86_64 ro root=/dev/mapper/vg_worklaptop-lv_root crashkernel=auto rd_LVM_LV=vg_worklaptop/lv_root rhgb quiet nmi_watchdog=1
	initrd /initramfs-2.6.32-358.6.1.el6.x86_64.img
	~~~

	To determine if the NMI watchdog was properly activated, check the /proc/interrupts file. The NMI interrupt should display a non-zero value. If the NMI interrupt displays a zero, alter the nmi_watchdog value, restart the system, and examine this file again. If a zero is still displayed, then the processor in the test system is not supported by the NMI watchdog code.

	The output, when functioning correctly, should look similar to the following:

	~~~
	[root@work-laptop wmealing]# cat /proc/interrupts \| grep ^NMI
	NMI: 861 636 377 357 Non-maskable interrupts

	~~~

	Each processor core has an NMI count. These should all be increasing over time. The above example is a quad core system.

	#### System wide NMI settings ####

	The NMI settings can be configured at runtime by using the sysctl interface.

	In the /etc/sysctl.conf, to enable, set:
	~~~
	kernel.nmi_watchdog = 1
	~~~

	To disable, set:
	~~~
	kernel.nmi_watchdog = 0
	~~~

	Note that this does not enable the functionality, the kernel parameter is required to correctly enable the NMI watchdog.

	#### unknown_nmi_panic ####

	A feature was introduced in kernel 2.6.9 which helps to make easier the process of diagnosing system hangs on specific hardware.
	The feature utilizes the kernels behavior when dealing with unknown NMI sources. The behavior is to allow it to panic, rather than handle the unknown nmi source. This feature cannot be utilized on systems that also use the NMI Watchdog or some oprofile (and other tools that use performance metric features as both of these also make use of the undefined NMI interrupt. If unknown_nmi_panic is activated with one of these features present, it will not work.

	Note that this is a user-initiated interrupt which is really most useful for helping to diagnose a system that is experiencing system hangs for unknown reasons.

	To enable this feature, set the following system control parameter in the /etc/sysctl.conf file as follows:

	~~~
	kernel.unknown_nmi_panic = 1
	~~~

	To disable, set:
	~~~
	kernel.unknown_nmi_panic = 0
	~~~

	Once this change has taken effect, a panic can be forced by pushing the system's NMI switch. Systems that do not have an NMI switch can still use the NMI Watchdog feature which will automatically generate an NMI if a system hang is detected.

	#### panic_on_unrecovered_nmi ####

	Some systems may generate an NMI based on vendor configuration, such as power management, low battery etc. It may be important to set this if your system is generating NMI's in a known-working environment.

	To enable this feature, set the following system control parameter in the /etc/sysctl.conf file as follows:

	~~~
	kernel.panic_on_unrecovered_nmi = 1
	~~~

	To disable, set:
	~~~
	kernel.panic_on_unrecovered_nmi = 0
	~~~


	#### panic_on_io_nmi ####

	This setting was only available in Red Hat Enterprise Linux 6. When set, this will cause a kernel panic when the kernel receives an NMI caused by an Input/Output error.

	### Common NMI error messages ###

	Certain messages are printed to the system logs when an NMI is received. The most common of these are:

	* Red Hat Enterprise Linux 5

	> "NMI received for unknown reason 00"

	* Red Hat Enterprise Linux 6

	> "Uhhuh. NMI received for unknown reason b0"

	### References ###

	Kernel documentation:

	Slackware NMI Documentation: http://slacksite.com/slackware/nmi.html
	https://access.redhat.com/knowledge/sources/source_rpms/kernel-2.6.32-358.6.1.el6/tree/Documentation/nmi_watchdog.txt