Two ways to do it, but only worked for me so I'll put it first and the second for reference:
$ openssl pkcs12 -export -in hostname.crt -inkey hsotname.key -out hostname.p12
$ openssl pkcs12 -in hostname.p12 -nodes -out hostname.pem
Other options for this method in comments below:
# Note, the -certfile root.crt appends all CA certs to the export, I've never needed these so it's optional for my personal steps
$ openssl pkcs12 -export -in hostname.crt -inkey hsotname.key -certfile root.crt -out hostname.p12