Starts with:
-----BEGIN RSA PRIVATE KEY-----
OpenSSL RSA key. Decode w/ openssl rsa -in $FILENAME -noout -text
Internal format is ASN.1, and can be viewed with openssl asn1parse.
Relevant standard: PKCS#1 / RFC 8017
The pubkey format,
-----BEGIN RSA PUBLIC KEY-----
can be decoded with,
openssl rsa -noout -RSAPublicKey_in -text
Starts with:
-----BEGIN PRIVATE KEY-----
Decode w/ openssl pkcs8 -in $FILENAME -nocrypt, but this subcommand lacks a -text/-noout flags, so you can't get a human representation, sadly.
Unless the private key inside of it is an RSA key, in which can openssl rsa can read this format. (But it is different than the -----BEGIN PRIVATE KEY---- format.)
Internal format is ASN.1, and can be viewed with openssl asn1parse.
Starts with:
-----BEGIN OPENSSH PRIVATE KEY-----
Not ASN.1.
These are what most people would consider "normal" keys, and look like this:
ssh-rsa <base64> <comment>
This format isn't specified anywhere (to my knowledge!) but it's just <key-type> <key-material, base64> <comment>. That's it.
The key material (the second/middle component) is specified in RFC 4253 §6.6. At least, for the keys I've encountered. Note that that also has the type, and if you de-base64 the encoded content, you'll see the ssh-rsa (or similar) in the output.
There's also these:
----BEGIN SSH2 PUBLIC KEY----
These are RFC 4716 keys.
They can be converted to the usual ssh format with,
<keymat> | ssh-keygen -i -f /dev/stdin
Note that RFC 4716 is, to oversimplify, just the base64'd contents of RFC 4253 (see above), inside some PEM guards. That means you can take the base64 data from between the PEM guards, slap ssh-rsa (assuming it's an RSA key; peek inside the base64 if you're unsure) on the front, join the lines, and tada, it's OpenSSH. Don't actually do this, of course, use the ssh-keygen command above, but the format is pretty simplistic.
To be honest, I don't know what generates these. But IME, they're usually RSA, and usually short!, so double-check the work of whatever party sent you that key, if you get one.
To get a printable version of either, first normalize on the normal ssh format, then,
<keymat> | ssh-keygen -e -m PEM -f /dev/stdin | openssl rsa -noout -RSAPublicKey_in -text
(-i is "import", -e is "export")