gpu_crash_report.txt

# Description / logs for GUI crashes on MacPro

I got this Mac Pro from the my IT department on October 9th.  It has AMD
FirePro D300 GPUs.  The group that had initially ordered the machine did not
want it, for reasons unclear.  I suspect they were suffering from the same
symptoms I'm describing here.

The IT department gave me the machine with a fresh install of El Capitan,
along with some other standard University installs, such as antivirus software
and MS Office.

At that point, the GUI would crash every few hours.  The symptoms were (and
remain):

* the window I am using stops responding to input;
* I usually get the spinning beach ball in the active window;
* I can still move the mouse pointer, and it changes to a pointer when I move
  it to the menu bar, but clicking has no effect anywhere on the interface;
* I can log in from another computer via SSH; the machine appears to respond
  normally to commands in the Terminal;
* I cannot kill Finder, or the Dock, or the WindowServer with "kill" commands;
* to recover use of the machine, I have to reboot it.
* the freezes have only happened while using the GUI at the machine; the
  machine has never frozen overnight, for example.
* there doesn't appear to be any particular application generating the
  freezes; it has happened using Safari to read web-pages, using VirtualBox,
  and mounting a disk image for an install.

There is a picture of the screen after a freeze at "freeze_screen.jpg" (see
below for more background).

After the first few freezes, I looked in the Console output, and found one of
these lines, after a freeze:

Kernel_2015-xx-xx-xxxxxx_Mac-Pro.gpuRestart

or...

PM WindowServer: GPU Driver appears to be hung (over 5 continuous seconds of unreadiness)

I don't remember which of these two lines I found, but the search led me to
this page:

https://www.macobserver.com/tmo/article/2013-mac-pro-freezes-continue-owners-have-little-recourse-apple-is-helpless

The page implies that there have been a reasonable number of reports of MacPro freezes of this type.

I also noticed that there had been a repair program for MacPro graphics cards:

https://9to5mac.com/2016/02/06/apple-mac-pro-repair-program-graphics-video/

From the advice found via the first of those two pages, I decided to do a
clean install of High Sierra.  To do this I:

* made a new USB install disk for High Sierra, from the App Store download;
* reset the SMC by unplugging for 15 seconds;
* reset the PVRAM with the usual keyboard chord on boot;
* booted into Recovery Mode, and erased the hard drive;
* booted into the USB install disk, and installed a fresh copy of High Sierra
  (not using any time machine backups);

I also installed the xcode command line utilities via "xcode-select --install"
at the Terminal.  This was so I could use basic developer tools like "git" at
the command line.

After a couple of days, the machine froze again, with the same symptoms above.
I logged into the frozen machine and dumped the full log to a file with "log
show > clean_hs_log1.txt".

At the time that the machine froze, lines like this started to appear in the
logs (see "clean_hs_log1.txt"):
 
2017-10-18 16:57:35.875628+0100 0x4ad      Error       0x0                  127    14   WindowServer: (CoreDisplay) [ERROR] - Fallback GPU Display Pipe transaction submission failed

They continue until I reboot the machine.  You can see another example of a
freeze in the photo "freeze_screen.jpg".  Notice the spinning beachball in the
centre of the screen, and the time at the top left of the screen: 11:38.  The
logs show the GPU Display Pipe ERROR appearing at 11:38:

2017-10-23 11:38:32.142995+0100 0x6e36c    Error       0x0                  21338  14   WindowServer: (CoreDisplay) [ERROR] - Fallback GPU Display Pipe transaction submission failed

See `clean_hs_log2.txt`.

At this stage, I started the chat for case number 100326037367.  You booked me
an appointment for Apple Store on October 25th, where I dropped off the
machine.

You contacted me on November 1st to tell me that you had replicated the
freezes with my own install, but when you ran a fresh High Sierra, you had not
had any problems.  You also mentioned that a more central Apple source had
said that some Mac Pros have problems with High Sierra, and that it may be
worth downgrading to Sierra.  I suspected that you had just not waited long
enough with the fresh install, and asked you to install Sierra for me, which
you did.

I used the machine fairly regularly from November 1st without freezes.

On November 6th (today), I concluded that Sierra had resolved the crashes, and
began to set up the machine for more regular use. I:

* wiped and formatted the hard disk as encrypted;
* installed various open-source software packages via Homebrew
  (https://brew.sh);

While I was opening a disk image, the machine froze again, with the symptoms
above.  I logged in and did "log show > cleanish_sierra_log1.txt". The 
time of the freeze was 16:03, as recorded on the desktop clock (which freezes
with the rest of the GUI). At the time of the freeze, I see these lines in the log:

2017-11-06 16:03:22.914095+0000 0x460      Fault       0x0                  0      kernel: (IOAcceleratorFamily2) virtual IOReturn IOAccelEventMachine2::waitForStamp(int32_t, stamp_t, stamp_t *): initial wait for 1 second expired. Continue wait for 4 seconds. stamp 44765 (gpu_stamp=44764)
2017-11-06 16:03:24.034437+0000 0x70d18    Default     0x0                  265    CommCenter: [com.apple.CommCenter.default] #I CSIAppInfo.ApplicationActivationObserver: handleLSNotitifcation_sync: Application exited: <private>
2017-11-06 16:03:24.198290+0000 0x73607    Default     0x0                  2079   diskimages-helper: (Foundation) *** -[NSMachPort handlePortMessage:]: dropping incoming DO message because the connection is invalid
2017-11-06 16:03:24.369411+0000 0xdc5      Default     0x0                  276    accountsd: (AccountsDaemon) [com.apple.accounts.core] "Idle-exit-preventing transaction ended!"
2017-11-06 16:03:26.915089+0000 0x460      Fault       0x0                  0      kernel: (IOAcceleratorFamily2) virtual IOReturn IOAccelEventMachine2::waitForStamp(int32_t, stamp_t, stamp_t *): timeout waiting for stamp 44765 (gpu_stamp=44764)
2017-11-06 16:03:27.034800+0000 0x5c5      Fault       0x0                  0      kernel: (IOAcceleratorFamily2) virtual void IOAccelEventMachineFast2::checkGPUProgress() - Signaling hardware error on channel 0..
2017-11-06 16:03:27.034822+0000 0x5c5      Fault       0x0                  0      kernel: (IOAcceleratorFamily2) void IOAccelEventMachine2::signalHardwareError(eRestartRequest, int32_t): GPURestartSignaled stampIdx=0 type=2 prevType=0 numStamps=9

which then continue until I reboot the machine.  I got another freeze later on
today, around 18:48. I against saw large numbers of lines such as:

2017-11-06 18:48:25.885246+0000 0x7b8b     Fault       0x0                  0      kernel: (IOAcceleratorFamily2) virtual IOReturn IOAccelEventMachine2::waitForStamp(int32_t, stamp_t, stamp_t *): initial wait for 1 second expired. Continue wait for 4 seconds. stamp 27991 (gpu_stamp=27990)

See "cleanish_sierra_log2.txt".

Based on these log messages, and the web pages linked above, I believe that I
am getting intermittent graphics driver or graphics hardware problems, causing
the macOS WindowServer to crash.  Given that I have now tried the three latest
versions of macOS, without fixing the problem, I wonder whether it will be
necessary to replace the GPUs, to stop the freezes.

I would like to emphasise, that it took 6 days of fairly regular use before
the first freeze after the fresh Sierra install. I also want to emphasise
that, unless you are exercising the GUI, I don't think you will see a crash.
So it may be that you will have to trust my reports on what happens, including
the photo and the logs I've recorded.