This took me some time to figure out so writing it down for posterity.
Setting up Rundeck in AWS running on Amazon Linux 2 using RPM install: https://docs.rundeck.com/docs/administration/install/linux-rpm.html
JRE runtime using AWS Corretto 8: https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/amazon-linux-install.html
DB Backend using Aurora MySQL Serverless v2. Create rundeck user/pass.
MySQL JDBC Driver using AWS MySQL JDBC Driver: https://github.com/awslabs/aws-mysql-jdbc
Download latest driver & put in /var/lib/rundeck/lib (lib needs to be created). Do not put it in /var/lib/rundeck/libext, it will not work there.
Edit /etc/rundeck/rundeck-config.properties:
...
dataSource.url = jdbc:mysql:aws://<rds-serverless-url>:3306/rundeck
dataSource.username = <db_username>
dataSource.password = <db_password>
dataSource.driverClassName = software.aws.rds.jdbc.mysql.Driver
...
(above snippet assumes your DB is named rundeck)
Start rundeck up: systemctl start rundeckd.service. Everything should work.