The evdev joystick code in RPCS3 is primarily designed to use XBox controllers by default. For other controllers, you'll probably need to make a few changes.
Configuration is stored in $HOME/.config/rpcs3/config_linuxjoystick.yml. The easiest way to test your joystick is with the PS3 Game Pad Test homebrew.
If you're not using an XBox controller, and your joysticks flat-out aren't working, try adding this to the config file:
Axis scaling: false
In addition, if when using one of the joysticks, the trigger buttons (e.g. R1/R2/L1/L2) are pressed down, add this:
Z axis triggers: false
TODO: Remapping axes.
Use the evtest
utility on your joystick, e.g. evtest /dev/input/by-id/my-joystick, and note which buttons are sending which button codes. You can then remap the buttons using this syntax:
Select: 8
Start: 9
Triangle: 0
Circle: 1
Cross: 2
Square: 3
R1: 7
R2: 5
R3: 11
L1: 6
L2: 4
L3: 10
You don't need to remap all the buttons; just do the ones you need to change.
Some controllers set the D-pad to be configured as an extra hat. In this case, RPCS3 will work out-of-the-box. However, if your controllers sends D-pad events as buttons instead, then note the button codes (see the Buttons section above), and add them to the config file like so:
Up: 1
Down: 2
Left: 3
Right: 4
Here's an example config file for a PS2 controller:
Axis scaling: false
Z axis triggers: false
Select: 8
Start: 9
Triangle: 0
Circle: 1
Cross: 2
Square: 3
R1: 7
R2: 5
R3: 11
L1: 6
L2: 4
L3: 10