nr-role.yml

---

- name: Add APT repo key
  apt_key:
    url: "https://download.newrelic.com/548C16BF.gpg"
    id: "548C16BF"
    state: "present"

- name: Add APT repo
  apt_repository:
    repo: "deb http://apt.newrelic.com/debian/ newrelic non-free"
    update_cache: True

- name: Install NR sysmon and PHP agent
  apt:
    pkg: "{{ item }}"
  with_items:
    - "newrelic-php5"

- name: Disable newrelic daemon (PHP starts it)
  service:
    name: "newrelic-daemon"
    enabled: False

- name: Template newrelic php config file
  template:
    src: "newrelic.ini.j2"
    dest: "{{ php_etc }}/mods-available/newrelic.ini"
    group: "root"
    owner: "root"
    mode: "0644"

After a number of experiments installing NR and starting services I believe I have discovered the problem.

Starting from a base VM snapshot with PHP installed but not NR, I used three variations of an Ansible role to install NR.

1. The first role is basically the same that I used to install the hosts with the pathology with which I started this thread. It installs and starts newrelic-sysmond before the PHP agent is used. The sysmond creates log files thus:

   $ ls -la /var/log/newrelic/
   total 20
   drwxr-xr-x  2 root     root     4096 Jul 13 17:58 .
   drwxrwxr-x 14 root     syslog   4096 Jul 13 17:58 ..
   -rw-------  1 root     root      270 Jul 13 17:58 newrelic-daemon.log
   -rw-r--r--  1 newrelic newrelic 1088 Jul 13 17:58 nrsysmond.log
   -rw-------  1 root     root      270 Jul 13 17:58 php_agent.log
   

   in which newrelic-daemon.log and php_agent.log both contain the following text

   #
   # This file was installed by newrelic-sysmond to ensure that older versions of
   # the New Relic PHP agent can correctly write to their log file.
   #
   # If you have no intention of installing the new Relic PHP agent, you can
   # safely remove this file from your system.
   #
   

   I verified the above with this simplified role .

2. With the sysmond and PHP agent installed using first role I run as an unprivileged user this PHP pseudo-daemon

   $ php -r 'sleep(3600); echo "done";'
   

   which writes this error

   open /var/log/newrelic/newrelic-daemon.log: permission denied
   

   and creates a zombie like so

   F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
   0  1000 19413 19289  20   0 239484 13868 hrtime S+   pts/2      0:00 php -r sleep(3600); echo "done";
   0  1000 19414 19413  20   0      0     0 exit   Z+   pts/2      0:00 [newrelic-daemon] <defunct>
   

   and writes to the php_agent.log like so

   2016-07-13 20:09:01.693 -0400 (19629 19629) info: New Relic 6.4.0.163 ("jitomirskaya" - "adf3967cac6f") [daemon='/tmp/.newrelic.sock'  php='5.5.9-1ubuntu4.17' zts=no sapi='cli'  pid=19629 ppid=19627 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=agent os='Linux' rel='3.13.0-49-generic' mach='x86_64' ver='#83-Ubuntu SMP Fri Apr 10 20' node='trusty11']
   2016-07-13 20:09:01.693 -0400 (19629 19629) info: spawned daemon child pid=19630
   2016-07-13 20:09:01.694 -0400 (19629 19629) warning: daemon connect(fd=4 uds=/tmp/.newrelic.sock) returned -1 errno=ENOENT. Failed to connect to the newrelic-daemon. Please make sure that there is a properly configured newrelic-daemon running. For additional assistance, please see: https://newrelic.com/docs/php/newrelic-daemon-startup-modes
   

3. With the sysmond and PHP agent installed using this role then permisisons on the log files are different:

   $ ls -la /var/log/newrelic/
   total 16
   drwxr-xr-x  2 root root   4096 Jul 13 17:59 .
   drwxrwxr-x 14 root syslog 4096 Jul 13 17:56 ..
   -rw-rw-rw-  1 root root   2326 Jul 13 17:59 newrelic-daemon.log
   -rw-r--r--  1 root root    876 Jul 13 17:59 php_agent.log
   

   and the permission error running the PHP pseudo-daemon goes away. The zombie, however, is still created.

4. Proceeding from 3., I kill the PHP pseudo-daemon, restart php5-fpm and the /tmp/.newrelic.sock is created. Now I can run the
   PHP pseudo-daemon and no zombie is created.

5. I installed using the first role and then deleted newrelic-daemon.log and php_agent.log and then restarted php5-fpm. Now everything runs properly and there are no zombies.

I draw two conclusions from this:

• nrsysmond's creation of newrelic-daemon.log and php_agent.log owned by root with mode 0600 is unhelpful.
• If an unprivileged PHP CLI script's agent cannot find /tmp/.newrelic.sock then it tries to start a
  newrelic-daemon which becomes a zombie.