Get it from the App Store.
In XCode's Preferences > Downloads you can install command line tools.
sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
Visit http://xquartz.macosforge.org/trac/wiki and download and install 2.7.2+.
You will need to fix the symlink it makes:
ln -s /opt/X11 /usr/X11
Will tell you what you need to check. Try out everything one by one and when one doesn't work
brew remove it and then reinstall it. If the install doesn't work, try
brew install it --use-gcc to prevent llvm from getting in the way.
Things that gave me issues that I had to remove and install:
You might need to install libxml2 and link it manually to make sure you don't get the wrong xml2-config if your path:
brew install libxml2 brew link libxml2
brew doctor and fix anything it tells you.
You may also need to symlink gcc for certain ruby gems and other stuff like that:
sudo ln -s /usr/bin/gcc /usr/bin/gcc-4.2
Maybe even g++ if you use that:
sudo ln -s /usr/bin/g++ /usr/bin/g++-4.2
You are good to go.