It seems even the Internet has trouble finding a solution for this. Here's some from retropie and sixad that actually bypass the default bluetooth toolset and it actually works:
Reference 1(Part: For older versions of RetroPie) Reference 2
Ignore all the tutorials about using bluetoothctl, they won't work as the PS3 controller will keep connecting and disconnecting. You might managed to connect it using bluetoothctl in a Pi but definitely Nano is different.
- Start with sixpair, you'll need to do this usb-pairing first. Connect the PS3 controller via USB, you don't have to press any buttons yet.
wget http://www.pabr.org/sixlinux/sixpair.c
gcc -o sixpair sixpair.c -lusb
sudo ./sixpair
This will print you messages like:
Current Bluetooth master: 00:15:83:0c:bf:eb
Setting master bd_addr to 00:15:83:0c:bf:eb
That means the usb-pairing is done. On to the next step.
- Compile and install sixad service
git clone https://github.com/RetroPie/sixad.git
cd sixad
make
sudo make install
This will compile and install sixad, which is daemon to support PS3 controller's bluetooth pairing/connection.
- Test connection
# Still in the sixad source directory
sudo ./sixad --start
Unplug the PS3 controller and hit the PS button. The LED will blink, the blink left-to-right indicating connecting, then solid one LED on later.
Test it with jstest:
sudo apt-get install -y joystick
jstest /dev/input/js0
You should see the numbers on screen change while rubbing the thumbsticks or hitting buttons.
- Make it auto start on boot
sudo systemctl enable sixad
sudo systemctl start sixad