I like the idea of digital signatures, so I recently obtained an S/MIME certificate. My early attempts with GPG had poor results, because the adoption rate is extremely low even between the nerds that I am usually in contact with. I decided to give a chance to S/MIME, because, despite experts call it technically worse, it has wider support. There is native support on iOS/macOS native mail clients, mutt, Thunderbird and also some web clients.
I noticed two things when using S/MIME in my daily routines.
In this article, I will:
Explore some S/MIME signed emails and figure out what is the signature and what data are actually signed. Just the body? Or some headers?
What is the signature format? What parameters does it need to be recognized as valid by my email clients?
Discover how Apple Mail uses the certificates from system keychain to sign emails, and finally
combine all the above knowledge to sign emails in mutt with certificates from macOS system Keychain
From vit@kabele.me Thu Sep 8 22:11:12 2022
Date: Thu, 8 Sep 2022 22:11:12 +0200
From: Vit Kabele <vit@kabele.me>
To: vit@kabele.me
Subject: Test email subject
Message-ID: <YxpMVsJNj/dFuJvF@Vit-MacBook-Pro.localdomain>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Status: RO
Content-Length: 29
Lines: 1
This is the test email body.
I will use mutt to learn more about the message formats and signing in general.
It follows the nice UNIX philosophy of single tool doing one thing great and thus it offloads the cryptographic responsibility to an external program.
The actual commands used to sign/verify/encrypt/decrypt are configurable and we can find defaults in contrib/smime.rc file.
These commands are actually format strings with special escape sequences expanding to the path of file with message content, file with signature etc (see docs).
By utilizing this nice architecture I modified the format strings so that I get the expanded commands and also copied the files with content/signature etc. Deducing the proper format of inputs to the signing command was then more or less straightforward.
I care mainly about signing and verifying emails, so I will skip the commands related to en/decryption.
Below find the exercept from default mutt smime.rc file with signing and verification commands.
(I replaced the hash of my private key by default-key-hash value)
set smime_default_key="default-key-hash"
...
# Sign.
set smime_sign_command="openssl smime -sign -md %d -signer %c -inkey %k -passin stdin -in %f -certfile %i -outform DER"
# Verify a signature of type multipart/signed
set smime_verify_command="openssl smime -verify -inform DER -in %s %C -content %f"
# Verify a signature of type application/x-pkcs7-mime
set smime_verify_opaque_command="\
openssl smime -verify -inform DER -in %s %C || \
openssl smime -verify -inform DER -in %s -noverify 2>/dev/null"
For these commands to work, you must import your S/MIME keys with special tool smime_keys that is shipped with mutt.
This tool manages own database of keys in ~/.smime and the expanded escape sequences from the above commands often point there.
Consulting the official mutt documentation provided me with a limited information. I was able to learn what is which sequence, but this doesn’t gave me an answer to what format is which file etc.
I modified the config parameter to get the following sign_command:
# Sign.
set smime_sign_command="echo openssl smime -sign -md %d -signer %c -inkey %k -passin stdin -in %f -certfile %i -outform DER > ~/signcommand; cp %f ~/signcontent;"
Luckily the command is not just execved, but really passed to a shell, so it is possible to use redirections and semicolons to actually provide more commands.
After trying to sign an email with this configuration, I obtained two files
The ~/signcommand file
openssl smime -sign -md sha256 -signer /Users/vitkabele/.smime/certificates/default-key-hash -inkey /Users/vitkabele/.smime/keys/default-key-hash -passin stdin -in /var/folders/gb/s3gn4l215tlbbw6hdfyrtdpm0000gn/T/mutt-Vit-MacBook-Pro-501-62732-14097872011691264914 -certfile /Users/vitkabele/.smime/certificates/default-key-hash -outform DER
and ~/signcontent file:
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
This is the test email body.
As we can see, mutt created the ~/signcontent file by taking the original message body and adding some new headers to it.
With this information, I can now invoke the command with proper paths and I will obtain the actual signature.
The signature itself is binary blob in the DER format.
To attach it to the message, mail client encodes it in base64 and attaches to the email as an attachment.
If you send such message to someone using a mail client without S/MIME support, the signature is actually presented as smime.p7s attached file.
From vit@kabele.me Sat Sep 10 15:10:53 2022
Return-Path: <vit@kabele.me>
Delivered-To: vitkabele@home.kabele.me
Date: Sat, 10 Sep 2022 15:10:39 +0200
From: Vit Kabele <vit@kabele.me>
To: vit@kabele.me
Subject: Test email subject
Message-ID: <YxyMz3q07dzAU2tJ@Vit-MacBook-Pro.localdomain>
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
micalg=sha-256; boundary="wCMJ7z35VtCvZW18"
Content-Disposition: inline
Status: RO
Content-Length: 7049
--wCMJ7z35VtCvZW18
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
This is the test email body.
--wCMJ7z35VtCvZW18
Content-Type: application/x-pkcs7-signature
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIQrTCC
B7AwggWYoAMCAQICAhARMA0GCSqGSIb3DQEBDQUAMGUxCzAJBgNVBAYTAkNaMRcwFQYDVQRh
Ew5OVFJDWi00NzExNDk4MzEdMBsGA1UECgwUxIxlc2vDoSBwb8WhdGEsIHMucC4xHjAcBgNV
BAMTFVBvc3RTaWdudW0gUm9vdCBRQ0EgNDAeFw0xODA5MjcwNzM5MjNaFw0zMzA5MjcwNzM5
MjNaMGkxCzAJBgNVBAYTAkNaMRcwFQYDVQRhEw5OVFJDWi00NzExNDk4MzEdMBsGA1UECgwU
xIxlc2vDoSBwb8WhdGEsIHMucC4xIjAgBgNVBAMTGVBvc3RTaWdudW0gUXVhbGlmaWVkIENB
IDQwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC5+HJtGjLPdjydYsAJ5U1SLtlg
c0XD0h4FNAuzt/rsSpSlV6UVTpb6N44kFD4e9rBdeA2dBGnkg4oXT39gUjoDz9jv1TSidcKm
0VtEY+igYZ0Rs7NIuQEoMNzkBmgYLLZyO5RceNa94yoLVdpGu1cIbh5WJ6CEw0PUrNupYKAD
Uv8tjwAl2Qf6I3Yuru79hhIF0y4ML4Tti+18uNu0A/1bL17Mh/7MM/7aagv1nUXn/0XXp9MR
m5MuqjWC+9HgX6oL6xlRRKHRRmueN6A7pxBYOklb+C/pHZcanqEmeo/YkGpoTqoGHtYDPb5I
RpQP9UFy0PMYaQ0D/DjFaH8YcZ1pScDC88WnbFu8TpmiSZ2z6JNG+UZikPjMNvjjMBe7Jus2
UxMxhm+unR4eWd+2thfvPwrjzSSwOi6Y7J/vZi5mWjovF1WJrIGwJBqiA6N35xAWIC1KcnFF
R0gvqUStb+SDgV8CFxlUy48N0j+gn0MPthyWcibHFMAMUC+2nAOpiZ2uZpJNJ4deSJ0qLuRf
vRysqZu6d4mOkvu69VyMgWQEkWaiXp+Q5ykF3vSOaegV1XmwLMPiWvBwFzFUd63Q4+og1JJ8
zM7X7xPJMpmegwWdlxagybh7nUQ7ciGjKbJw93Xcjuoa/flgP9IjvOh1m4sFP8hSZBW43Afe
Pq3g1kS0lQIDAQABo4ICZDCCAmAwgdUGA1UdIASBzTCByjCBxwYEVR0gADCBvjCBuwYIKwYB
BQUHAgIwga4agatUZW50byBjZXJ0aWZpa2F0IHBybyBlbGVrdHJvbmlja291IHBlY2V0IGJ5
bCB2eWRhbiB2IHNvdWxhZHUgcyBuYXJpemVuaW0gRVUgYy4gOTEwLzIwMTQuVGhpcyBpcyBh
IGNlcnRpZmljYXRlIGZvciBlbGVjdHJvbmljIHNlYWwgYWNjb3JkaW5nIHRvIFJlZ3VsYXRp
b24gKEVVKSBObyA5MTAvMjAxNC4wEgYDVR0TAQH/BAgwBgEB/wIBADB6BggrBgEFBQcBAQRu
MGwwNwYIKwYBBQUHMAKGK2h0dHA6Ly9jcnQucG9zdHNpZ251bS5jei9jcnQvcHNyb290cWNh
NC5jcnQwMQYIKwYBBQUHMAGGJWh0dHA6Ly9vY3NwLnBvc3RzaWdudW0uY3ovT0NTUC9SUUNB
NC8wDgYDVR0PAQH/BAQDAgEGMB8GA1UdIwQYMBaAFJMYNh+paXBRNapPP6yNUH4mBSkKMIGl
BgNVHR8EgZ0wgZowMaAvoC2GK2h0dHA6Ly9jcmwucG9zdHNpZ251bS5jei9jcmwvcHNyb290
cWNhNC5jcmwwMqAwoC6GLGh0dHA6Ly9jcmwyLnBvc3RzaWdudW0uY3ovY3JsL3Bzcm9vdHFj
YTQuY3JsMDGgL6AthitodHRwOi8vY3JsLnBvc3RzaWdudW0uZXUvY3JsL3Bzcm9vdHFjYTQu
Y3JsMB0GA1UdDgQWBBQPKHw+NgA4EFCuPbghl4v3YFxheDANBgkqhkiG9w0BAQ0FAAOCAgEA
G4YWLGKRdBc494orA7yWtnFq8JfOOjCJYEObIHOm+J0rSnKJg2eppgonaT338tfSXyfxMXS3
tczRMPNk3a43ulg8MOgu17eg6AhPrTLK8CUoNFvYqxa0AyJsklEEIZRSbKAHwPDybKx8TT5M
yklrRYbfkipUqCw/tnhlLevBTSbVNNwT5vmtnk1+fmjypmjPuLurYfeWnNLIZcFtk9fe8blZ
r+lneqVQNL1s0u1l0xm7vsxaLX2KBnserhen/w4T/MUkQ9jtzFb5aP4N20JlGyG3R7rTfDYX
EUV59696GJwVgF/qDjxPNTG6tG8H7Xkgyc1nM0Ve0TnTajJvY8atw7KR6FbV8A49UeB9A0XG
LHIPZLrwu+L8h8q3rVfvqUgE63dGeCUwe94jFcaMwimp3B/r26Vs4Ay562Y9DyC2/LzhTpMw
ZMlGjJ58kG/aupCb3VkOCgswCzWCtSM46Ohl15WO+610kP/7NHs/TN0Y1n82f2xGU6a7B2VT
Vn+52u9yj0+0pPyo3l9B+lhj3Bid65tE9TgsLCLHwWeqHEpxHua1tVV0SOsMKMP60RRvbtEd
KSVPJBdRSrUQ1rQWmUw2ziWd6DnAytoirlEcbeKoCoT1OHkT6z4nKgBkHnin7LynN2CUR1dj
/z6yvxU9SPC2rOZLcSCsaqduI3WJh4B3WjQwggj1MIIG3aADAgECAgQBWgysMA0GCSqGSIb3
DQEBCwUAMGkxCzAJBgNVBAYTAkNaMRcwFQYDVQRhEw5OVFJDWi00NzExNDk4MzEdMBsGA1UE
CgwUxIxlc2vDoSBwb8WhdGEsIHMucC4xIjAgBgNVBAMTGVBvc3RTaWdudW0gUXVhbGlmaWVk
IENBIDQwHhcNMjIwNzI3MTcxMTIyWhcNMjMwODE2MTcxMTIyWjBaMQswCQYDVQQGEwJDWjEZ
MBcGA1UEAwwQTWdyLiBWw610IEthYmVsZTEPMA0GA1UEBBMGS2FiZWxlMQ0wCwYDVQQqDARW
w610MRAwDgYDVQQFEwdQODczOTgwMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
ny0YVYZaeR8mwuKQL2dZSHaIL35ZrT/az8aOc52S13llcT+Z5LnQAhXkgmDq2WDUuyTddTgs
bcDTvi7bHrDEgkYdF4IuEBYrN+oggl1uScUwOGyWvpcwhfsvrSrJ13e5coAt/WxSp69/U65p
vo+0muyOgJDP2kNBD4Iu3IhJRPvE2o7GxW0yEWfwhVBL3IBl/LZzF2RoceoSUhSl3x7TTs7F
yxZztGKXxFZHLFaCJXEkXpvzC8bD2mxycS94U3+4JWsLQ5gF5tmcj/dTBW8osCtg0h37h3n7
SItDPSCiEUBJ2q8jc13CEdRoE7aBH3hQvxoNmuP6Wr5Khrfb5ZcDTlGoPDAUPuv5DA1mTJke
NIPNObNY3OqAnrwPrEqChCLnyOarclGKTpKd0rB8oDtEdJaaGnR0+B+xiRVgQ7AvQIPSCfIx
gjlqsu+aGcEFqYV/4Nldw/8YYjdWVNqZ6ALuYfoBJHy/MInmcQKZBRWXYDZXEmL+IGQRKX3+
afWsPOoPNSxJ/HrtC6kwt4QyNjOR2rnhU5Hvxw3dPyyL84hevak9snM2AYbbglra13pnWT9V
yOExv7mgBrLTUBIZ2+uIP5MzDtjAi5e+w55XfX4un2v2FPwdt8mq/uf/D3EP8k+9FCizaZr/
tArWla2WWktDhLuhGzQKvbAVyyZUsPsWgo8CAwEAAaOCA7IwggOuMC8GA1UdEQQoMCaBDXZp
dEBrYWJlbGUubWWBFXZpdC5rYWJlbGVAaWNsb3VkLmNvbTAJBgNVHRMEAjAAMIIBLAYDVR0g
BIIBIzCCAR8wggEQBglngQYBBAERgVIwggEBMIHYBggrBgEFBQcCAjCByxqByFRlbnRvIGt2
YWxpZmlrb3ZhbnkgY2VydGlmaWthdCBwcm8gZWxla3Ryb25pY2t5IHBvZHBpcyBieWwgdnlk
YW4gdiBzb3VsYWR1IHMgbmFyaXplbmltIEVVIGMuIDkxMC8yMDE0LlRoaXMgaXMgYSBxdWFs
aWZpZWQgY2VydGlmaWNhdGUgZm9yIGVsZWN0cm9uaWMgc2lnbmF0dXJlIGFjY29yZGluZyB0
byBSZWd1bGF0aW9uIChFVSkgTm8gOTEwLzIwMTQuMCQGCCsGAQUFBwIBFhhodHRwOi8vd3d3
LnBvc3RzaWdudW0uY3owCQYHBACL7EABADCBmwYIKwYBBQUHAQMEgY4wgYswCAYGBACORgEB
MGoGBgQAjkYBBTBgMC4WKGh0dHBzOi8vd3d3LnBvc3RzaWdudW0uY3ovcGRzL3Bkc19lbi5w
ZGYTAmVuMC4WKGh0dHBzOi8vd3d3LnBvc3RzaWdudW0uY3ovcGRzL3Bkc19jcy5wZGYTAmNz
MBMGBgQAjkYBBjAJBgcEAI5GAQYBMH0GCCsGAQUFBwEBBHEwbzA7BggrBgEFBQcwAoYvaHR0
cDovL2NydC5wb3N0c2lnbnVtLmN6L2NydC9wc3F1YWxpZmllZGNhNC5jcnQwMAYIKwYBBQUH
MAGGJGh0dHA6Ly9vY3NwLnBvc3RzaWdudW0uY3ovT0NTUC9RQ0E0LzAOBgNVHQ8BAf8EBAMC
BeAwHwYDVR0lBBgwFgYIKwYBBQUHAwQGCisGAQQBgjcKAwwwHwYDVR0jBBgwFoAUDyh8PjYA
OBBQrj24IZeL92BcYXgwgbEGA1UdHwSBqTCBpjA1oDOgMYYvaHR0cDovL2NybC5wb3N0c2ln
bnVtLmN6L2NybC9wc3F1YWxpZmllZGNhNC5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwyLnBvc3Rz
aWdudW0uY3ovY3JsL3BzcXVhbGlmaWVkY2E0LmNybDA1oDOgMYYvaHR0cDovL2NybC5wb3N0
c2lnbnVtLmV1L2NybC9wc3F1YWxpZmllZGNhNC5jcmwwHQYDVR0OBBYEFLNpQmdCEaML0Xfi
UrBJoEzW+xS2MA0GCSqGSIb3DQEBCwUAA4ICAQBOoO0mT1osxoMxUsC0GHqpEPCFZsHNjsck
zxxhvYLBU+DioStOLshoC90CRP7OXMSijfUaa11k3hq9qGMIjR1mTgYuQ4H2bvKrxy8/ps3l
NtSUwUJNxCqCZ9U04RVbyChR/quUknmi0MyJfxF/y1dAt7XTv20mpaCZyw5jMEJGTqLAn9oH
JuN/Z3t073MMBNNbFynW+97jQvtptQxPgHlvRaF++7xeJZytc6e6iWwhMZNGdp34gz4M/17/
HtHkbuDBKsPPr3MpkPiQX8Xm28tSXjLTEiDY/kzUIrE0X9Iq+hTM6nE9Z2UyiEKm6zKtQQI9
mtwebXI7Ps9+4tru9DDegWNtr2Wjmx1P5oWsHZaebEfAzQztp1PAR31pbB6WG65jhft/pfUX
wF3CkZuNJMFRfBsLA9e+i9MyWLcV5jAn7maJ8/qXEOohXc5PhEn5Pugw7vpjgiRrczmrMqfK
ijFB3s7yyPRIwbu87CO1YJ8pj2HEchnfXDTD0TZzN5QJNyLXoaxaVSIWzBUynB8KU2GcmPWP
3w181CxKcQ0OotIMvdOKO6f7gDZBMv9eHNhUbTfbsAltzUmAOvhVOgmt2OnBT8GmZzn/OODy
6PGUuXIho1GbLKeIC9ksJXo+byVbXZMmqOr0i7TF34eBKtOaB7Ucrpffoni5Aa1UFfHLFMSb
jjGCApgwggKUAgEBMHEwaTELMAkGA1UEBhMCQ1oxFzAVBgNVBGETDk5UUkNaLTQ3MTE0OTgz
MR0wGwYDVQQKDBTEjGVza8OhIHBvxaF0YSwgcy5wLjEiMCAGA1UEAxMZUG9zdFNpZ251bSBR
dWFsaWZpZWQgQ0EgNAIEAVoMrDAJBgUrDgMCGgUAMA0GCSqGSIb3DQEBBQUABIICAIp3Q6Ir
hMi5e7TO6pha4G/QNCn8sIuOoNZxfOLegYcZszkEQnnQ42A8RkCfi2lP5m6tcCHdimjuFqWa
8BzOdtVgeW8lTFp8rqBj/xW90btxDfBFLmi3kFo8jzRIgU2cvW/8VNfykxVMVszqM1QRd748
8dc+aDaWIAWzQD2AcyBe5a+wWu7RHjXV/sE68s3rUt9GPSNo/J1C7gWvFgA3H0vkiIrRu5h+
/NxSDU6vfxAeTnPbTey6l9ifMhfE9R/zv94Ia6uc0A+glwmcXPWSgzUwjONNZA/Qy+e9DRrr
CVMVmwdfWN02ylYIQbseOnhRjcY+duWbKH2dI8HlLs7lRSogI90LbwPBHwbfex7/mnXGu7qA
9PeOW91LvgFO9CPvXAhsvfsif2OhFKGaZWH3Yfwhox7K2OdMywS2x/xE9ZBEd5D2wa4t3uey
rJG5hg0xM6BEtiaOCTzvNqUh9/UU8WsUqGh+trCZXrh2uYLOFue3UT6jlH/8+nLfw+C4Ic2y
EqCQH6nsR5uQVi5fDf7HGoIIN2imzPFWICtfgul6/H3ZsJgS9cQO2RBkMG/cNUcAzQtoG94k
NxEBilNmU3xaKVlK7EpOlyDlhEWQbtwaQIo/cgKkbCaaHdwz8bS7xx1qgwdxRDRSqLK1Lx6z
5IzFMGx4crOp3KTy+E/pFOairpl4AAAAAAAA
--wCMJ7z35VtCvZW18--
Although it is very long, I attached the whole message so everyone can try to verify the signature yourself.
By consulting the mutt configuration file, I got the signature verification command:
openssl smime -verify -inform DER -content ~/signcontent -in message.sig
The ~/signcontent file is the same file as above.
The content of a message.sig file is base64 decoded signature from the email.
To obtain the message.sig, do something like echo MIAGCSqGSIb3DQEHAqCAMI...irpl4AAAAAAAA | base64 -decode > message.sig
Running the above openssl command should verify the signature and say “Verification Successful”.
(Note that you need to have the ~/signcontent with CRLF line endings, otherwise the verification fails.)
The macOS already contains a built in way of managing certificates and private keys.
This framework can be accessed in GUI by the means of Keychain Access.app or via security(1) cli tool.
Both apps are provided directly by Apple and are part of clean macOS installation.
I want to avoid having the certificates stored twice on my computer because I’d have to import all of them to each email client separately.
There is an official API for the Security Framework with bindings to Objective-C and Swift. After some digging in the documentation I found the CMSEncodeContent function that does exactly what I was looking for.
I wrote a simple terminal wrapper for the function, before I accidentally bumped to the macOS security(1) tool which has a cms subcommand that does the exact same thing.
Equivalent of the openssl smime -sign ... command is therefore security cms -S -T -i signcontent -N "<certificate id>".
The -T option must be used to generate a detached signature.
If you ommit it, the email signature will be correctly recognized and verified by Apple Mail on macOS, but the verification on iOS and in mutt will fail.
Because of the nice architecture of mutt, all we need to do is change the configuration.
By replacing the sign command in mutt, it is possible to sign emails without importing the key to special S/MIME keychain (smime_keys for mutt)
When you use the security(1) tool to generate the signature, you keep getting a verification error.
I spent half of my saturday by comparing different versions of email contents and its signatures, before I eventually noticed an inconsistency in newline handling.
The content that mutt handed over to the command for signature used LF only, while the file handed for verification used CRLF. Because of this, the verification command actually tries to verify different data and it couldn’t be possibly considered as a valid signature.
In my setup I solved this issue by prepending the unix2dos -q -o %f; command to my smime_sign_command, so that the newlines are properly converted to CRLF before the signing utility is launched.
The CRLF line breaks are anyway requred by the RFC2045 and mutt should probably pass the content to be signed in this format.
Following is the snippet with working options:
set smime_default_key="<my_email>"
# %c is replaced by smime_default_key
set smime_sign_command="unix2dos -q -o %f ;security cms -S -T -i %f -N %c"
It might require some trial and error to determine the proper value of smime_default_key, as it depends on your key.
The utility attempts to access the system keychain and as such it presents the user with GUI password prompt. As mutt is not aware of this, it keeps asking for the certificate password every time a message is to be signed. This can be safely ignored and confirmed with no input as the CLI utility doesn’t require any.