Edit file /home/gitlab/gitlab/config/unicorn.rb
Find line listen "#{app_dir}/tmp/sockets/gitlab.socket"
and comment it.
Uncomment line listen "127.0.0.1:8080"
sudo a2enmod proxy
sudo a2enmod proxy_balancer
sudo a2enmod proxy_http
sudo a2enmod rewrite
Note: For apache 2.4 the Order / Allow / Deny syntax has changed to Require syntax. I also agree with comments above about not needing a load balancer, or passenger. In fact the config is very simple, this is my file which works on 2.4.7: