tummychow/readme.md

## readme.md

      
    Raw
  

              readme.md
            
          
    Just a brief proof of concept for running bugzilla on nginx using fcgiwrap

Software development needs issue tracking cause there's too much stuff to remember. Raise issues and manage tasks. Get stuff done. Keep track of backlogs without losing yourself in the ocean of stuff that needs doing. Real life is quite similar, so lately I have been experimenting with small self-hosted instances of various issue tracking solutions. Bugzilla is one of the most mature and feature-complete foss options in this area so obviously I had to try it.
The server that I run all my personal toys on is using nginx exclusively (with passenger for my numerous ruby and python applications, more on that in another gist? perhaps!). Unfortunately bugzilla does not support nginx out of the box. It mainly targets apache's mod_perl (or mod_cgi). However since bugzilla is ultimately a cgi application, we can find ways to serve it on nginx.
Basically the solution is to wrap bugzilla's cgi scripts in a fastcgi server so that nginx can talk to it. There are tutorials all over the internet for doing this with perl fastcgi wrappers from various places. Mysterious perl scripts are great and all, but there's a better solution (fcgiwrap) which we can use instead. fcgiwrap is the easiest way to run cgi applications over nginx without having to run a separate httpd (eg thttpd or - heaven forbid - apache!) that is cgi-aware. Also, mysterious perl scripts are not exactly well maintained, and I have no idea if they will work in the future or what to do if they break. I think fcgiwrap is a better choice here.
This guide will target archlinux because that's what I use. However, many distributions won't have official packages for fcgiwrap. I guess it's a bit... obscure. Arch has an official package, at least. Find the appropriate instructions for installing it on your system, and then you can probably follow along with the same instructions as yours truly. Note that I am also targeting systemd because arch uses systemd. Fwiw, debian is moving to systemd anyway (it's in jessie) and that also means ubuntu will be leaving upstart for systemd, so all the big players will have abandoned sysvint within the foreseeable future. As far as the linux kernel is concerned, systemd has clearly won the init war.
Step 1 - install stuff

I am starting with a fairly minimal (but not absolutely minimal) arch linux installation, with the base and base-devel groups. (I also have python2 and ssh installed so that the VM is an ansible node, but I haven't tried doing this with an ansible playbook yet. Maybe in the future?)
Pretty sure perl is a dependency of the base group so you probably won't need to install that. You can use bugzilla from the official repo (haven't tried this yet, plan on doing it in the future) or try to just hack it out with the install scripts yourself. I have no experience with the Perl ecosystem whatsoever and I have never written a word of the language. Nevertheless I decided to try the latter solution first. Not the best idea, but it's a learning experience, right?
$ pacman -S nginx fcgiwrap postgresql
Pretty self explanatory. If you want to use official bugzilla package, obviously add bugzilla. Again I have no experience with the official package... yet. I plan to roll back my VM and test with that as well to see if the experience is smoother - I expect it will be. arch package maintainers are good at what they do and I have confidence that the installation will be easier if I use official packages.
Oh and yes I am using postgres. You can use mariadb (arch does not have the original mysql in its official repos - I'm fine with that personally) as well, but I host other applications with pgsql on my personal server so I wanted to share the DB between them. If you've never installed postgres on an arch system before, please refer to the article on archwiki for more deets. In particular you can't create any databases or such until you run an initialization command which is specified there. I don't remember the exact command but hey, archwiki is great so you should probably read it anyway.
I distinctly recall bugzilla needs you to configure the pgsql port to a default that bugzilla expects. Don't forget to do that.
step 2 - acquire bugzilla

A lot of this is basically coming out of the bugzilla setup instructions. You can probably follow most of those without difficulty. The interesting part will come later.
Unpack a tarball for bugzilla into a suitable directory. I am actually doing all of this as root (terrible for security seriously don't do this on a real server) but I assume you are comfortable with chowning a tree of files to the http server's user. This will be important because files need to be accessible by fcgiwrap's user. After all, it's fcgiwrap that will be doing the actual cgi script execution.
$ curl -L http://link.to.bugzilla.com/bugzilla-4.4.4.tar.gz | tar -xz
$ cd bugzilla-4.4.4
I ran ./checkmodules.pl as recommended by the bugzilla install instructions. I am by no means a perl hacker so the only perl module I had installed was... uh, perl itself. Not sure if that counts as a module. Obviously this means there are a ton of unsatisfied deps. I ran install-modules several times to acquire the core deps plus some of the ones I personally cared about (patch viewing, graphical report generation). This installation wasn't heavily tested or anything and I wasn't actually going to use it in this state, I just wanted to see if the hosting was feasible - so I was a bit laissez faire about mysterious CPAN build errors. Hell if I know what they mean, man. (again forgive me, I don't speak perl at all)
Now I cracked open localconfig as recommended. We're using nginx so no .htaccess files required. I configured the postgres database as well. (Follow the instructions in bugzilla docs to set up the initial postgres db)
After doing the config stuff (again, refer to the bugzilla docs), checkmodules.pl tells me that the system is ready to roll. Time to get to the interesting part.
step 3 - fcgiwrap on bugzilla (ie the part you actually care about)

The default arch linux systemd unit files for fcgiwrap (as of writing) are twofold. There's a service unit file which actually invokes fcgiwrap with the appropriate variables to launch a script, and then there's a socket unit file which handles the dirty work of launching the fcgiwrap service when requests hit the socket. I've seen systemd unit files on archwiki that use spawn-fcgi for this work instead, but it seems spawnfcgi is not part of fcgiwrap (looks like it's part of lighttpd). So for now I'm just using the default systemd unit files.
The essence of this is that, to launch fcgiwrap, you need to systemctl start fcgiwrap.socket. To terminate it, you need to stop both the socket and service units. Someone with more systemd experience could probably explain why this is the case. It seems to me that the service processes don't get torn down when the socket unit gets stopped, so systemctl restart fcgiwrap.socket is not sufficient. I'll learn more about this sometime in the future, maybe.
Anyway. Next step is to make sure fcgiwrap can execute the many perl CGI scripts in the bugzilla directory. The user under which fcgiwrap is running is set in the fcgiwrap.service unit file. If you've never used systemd before, basically this is under /usr/lib/systemd/system/fcgiwrap.service, but you should copy that file to the directory for custom systemd units before editing it. I think the custom systemd unit directory was /etc/systemd... can't remember clearly. Oops. Oh well. If it's a throwaway installation for you, like it was for me, then just edit the unit file in place and move on. Just remember that you aren't supposed to do that in practice. After that, make sure the fcgiwrap user owns the CGI scripts so it can execute them.
One other thing I recommend is to add the -f flag to the invocation of fcgiwrap, in the service unit file. This redirects fcgiwrap's errors to the nginx error log. Really helpful for when you have to bang your head on dumb errors and you wish you had a log of what the actual fastcgi server was doing.
Finally you can set up your nginx configuration block for this fastcgi server. It's the same as pretty much any other fastcgi server, so if you've configured nginx before there's nothing freaky going on here. Something like this will be satisfactory:
server {
    server_name bugzilla.lan;
    port 80;
    root /srv/bugzilla-4.4.4;
    index index.cgi index.html;
    
    location ~ \.cgi$ {
        fastcgi_pass unix:/run/fcgiwrap.sock;
        fastcgi_index index.cgi;

        # note: include fastcgi_params is not sufficient, unless your distro has patched it
        # to cover other params like SCRIPT_FILENAME. fcgiwrap depends on those params to
        # determine what cgi script to actually run.
        # I believe some distros have made those patches, but arch includes the newer
        # fastcgi.conf from nginx upstream, so I recommend just using that. It specifies
        # pretty much all the parameters.
        include fastcgi.conf;
    }
}
Btw sorry if there are dumb syntax errors in there. I'm not sitting in front of my nginx instance right now, and I haven't configured nginx often enough to just remember the syntax straight off the top of my head. Just look at a config file for php-fpm and it'll be pretty much the same as this.
Now systemctl start nginx and navigate to the homepage of your bugzilla hostname. You should get the ever-so-familiar bugzilla homepage. If you get a 403 forbidden error, you probably made a silly permissions error. For example I was running nginx workers as root (too lazy to chown the bugzilla CGI scripts, don't be lazy!) and I was getting 403 errors from the fcgi gateway (ie fcgiwrap was giving me the 403, not nginx). I was staring at this for half an hour before I realized that fcgiwrap needed permissions as well, to execute those cgi scripts. Silly mistake. So I ran fcgiwrap as root (say it with me: don't do that) and that fixed the issue. Yay for abusing root on throwaway systems.
Anyway if you did all this, you should have a semifunctional bugzilla instance hosted by nginx. Some caveats:

I have no idea if actual bugzilla functionality works. For me, the smoke test is "does it serve a webpage?" Until then, all the issues are usually server-config-related. For all I know, the rest of the site was functionally broken and maybe it would have blown up if I tried to log in or create an issue. Whatevs. I'll put more effort into verifying that it works after I clean up my installation procedure.
Everything's running as root. Don't do that on an actual server. (For what it's worth, I also have passwordless root ssh on this VM. The sheer amount of bad security is beyond enumeration. If not for my router's NAT I'd have lost the host system as well as all my VMs by now.) If you're familiar with unix permissions and users and such, it should be trivial to move these instructions around and serve as a different user. Hit up the archwiki article on users and groups. Archwiki is good stuff.
Perl seems to get out of memory errors that lead to it being killed by the kernel if the fcgiwrap process runs for a while. I'm guessing fcgiwrap and perl are interacting in an unpleasant way that leads to a memory leak, killing an fcgiwrap process. Not sure if this is a bugzilla issue, but I haven't seen any evidence that it is. I haven't inspected this in much detail, and the site actually kept serving even after two OOM kills (presumably, the systemd socket unit was launching more fcgiwrap processes in response to the new requests proxied by nginx, so the death of old processes for the service unit didn't mean much). Still, this out of memory stuff is kind of disconcerting. I do not like the idea of serving memory-leaky perl applications on a server that will have several other applications asking for memory. Merits more investigation.