The evdev joystick code in RPCS3 is primarily designed to use XBox controllers by default. For other controllers, you'll probably need to make a few changes.
Configuration is stored in $HOME/.config/rpcs3/config_linuxjoystick.yml. The easiest way to test your joystick is with the PS3 Game Pad Test homebrew.