If you created your ssh keys with default rsa settings, their format is not compatible in openssl.
To encrypt and decrpt messages in openssl, you need your public key and private key in pem format, and start like this:
-----BEGIN PUBLIC KEY-----
or
-----BEGIN RSA PRIVATE KEY-----
Examples:
# convert your ssh public key to openssl pem format
% ssh-keygen -f ~/.ssh/id_rsa.pub -e -m pem | openssl rsa -RSAPublicKey_in -pubout > id_rsa.pub.pem
# convert your ssh private key to openssl pem format
# this will replace your private key in-place, MAKE A COPY first.
% ssh-keygen -p -N "" -m pem -f tmp.private
# encrypt and decrypt your messages
% echo "test" | openssl rsautl -encrypt -pubin -inkey id_rsa.pub.pem > tmp.txt
% openssl rsautl -decrypt -inkey tmp.private -in tmp.txt
this method can only be applied on short text, such as a symmetric key, or password. i use it to store my web site user and password on github...