Open Apple menu -> System Preferences -> Bluetooth and disable Bluetooth on Mac as well as any other nearby Macs or devices which will try to pair with and confuse the controller.
Reset PS3 controller by inserting paperclip into pinhole near L2 button.
Connect PS3 controller to Mac with USB cable.
Press "PS" button for 1-2 seconds until the red lights flash.
Unplug USB cable.
Click the "+" icon in the preferences to open the Bluetooth Setup Assistant.
A window will appear asking you to enter an access code. Enter "0000" (without the quotes) and click "Accept."
Close the Setup Assistant.
Click the "PLAYSTATION3 Controller" item in the Bluetooth list in the preferences. Then select "Add To Favorites" and then "Update Services" from the little gear icon menu.
Disable Bluetooth. Wait a second.
Enable Bluetooth. Wait another second.
The controller will now appear connected in the list and can be used in applications.