Here are my notes on how to create S/MIME Certificates signed by OS X Server's Open Directory Certificate Authority. It's pretty easy to set up your own small-scale PKI on OS X Server using the OD CA, which will already be a trusted CA.
Make sure that you have OS X Server's Certificate Authorities installed on your local admin account:
test -d ~/Library/Application\ Support/Certificate\ Authority || sudo cp -R /var/root/Library/Application\ Support/Certificate\ Authority ~/Library/Application\ Support
The rest is done in Keychain Access.app:
Keychain Access>Certificate Assistant>Create a Certificate…
Name: <Your name>
Identity Type: Self Signed Root
Certificate Type: S/MIME (Email)
Check Let me override defaults
Choose number of days. One year (365 days) is the default, but pretty frequent to go through this.
Fill out the certificate information
2048 bits (Is there a way to tell Keychain Access to generate 4096 bit certs?)
Key Usage Extension: Check Include and Critical, plus Signature, Non-Repudiation, Key Encipherment
Extended Key Usage: Check Critical, Email Protection
Basic Constraints Extension: Do not include
Include Subject Name Alternate Extension: make sure that the RFC 822 name is the certificate user's email
Create into your login keychain.
In Keychain Access, login keychain, My Certificates, highlight the newly create self-signed root certificate, which will have the name you used above
Keychain Access>Certificate Assistant>Request a Certificate from a Certificate Authority…
User Email Address: Use the RFC 822 name from above
Common Name: The name on the certificate (from above)
CA Email: the email used for your Server's OD CA
Saved to disk
Show in Finder…
Keychain Access>Certificate Assistant>Create a Certificate for Someone Else as a Certificate Authority… Drag the CSR from Step Two into the box.
Issuing CA: This must be the OD CA that looks like "Domainname Open Directory Certification Authority". If you choose the one that looks like "IntermediateCA_HOST.DOMAIN_1" it will not be verified or trusted on iOS devices when you install the certificate into your profiles.
Check Make this CA the default Check Let me override defaults for this request
Proceed exactly as in Step One above.
You'll have to authenticate as an admin a few times to access the OD CA Certificate and key in your System Keychain. Finally, you'll be prompted to email a file certificate.pem to the user in the RFC 822 name above, and the OD CA-signed certificate will be in your admin keychain. Because the certificate and key are in your keychain, you can delete this email. Step Four will be to create an encrypted .p12 file of the cert and key, then send this to the user by a email.
Observe the newly created valid S/MIME certificate in your login Keychain, next to the original unverified self-signed root certificate with the same name. Toggle the little LHS arrow next to the original unverified certificate, use shift to highlight both the cert and its private key immediately below, and delete this certificate-key pair.
Highlight the valid certificate and its private key, then:
Keychain Access>File>Export Items…
Save the certificate-key par into an encrypted Personal Information Exchange File (.p12) using a strong passphrase for this file.
Email this .p12 file to the user, who will be able to install the valid S/MIME certificate on all OS X and iOS devices. They must, of course, know the passphrase for the .p12 file. Alternatively, push the certificate to devices using Profile Manager.