My issue was that the passwords were being sent in the wrong order. I needed to send my regular password first and my token 2nd.
I had to apply this diff to auth.c which made rsa token generation similar to whatever "LIBOAUTH" does.
- if ((strcmp(opt->name, "password") && strcmp(opt->name, "answer")) ||
+ if ((strcmp(opt->name, "secondary_password") != 0) ||