How eMail encryption works and why it is an utter public failure

You have a message for someone that is both confidential and sensitive, you want to ensure that onl the receipent can open it and that (s)he also gets reassurance that it actually was you who sent this and that the message has not been tampered with. Enter the world of digital encryption and digital signatures.
Notes users are used to just checking the respective check box and their messages get signed and encrypted without any further action required. When sending messages to external parties this is usually is accompanied by error/warning messages that this won't work. So it is a fault in Lotus Notes (usual business practise: if anything doesn't work, first blame Notes)? Far from that. To be able to send an encrypted message one needs to have access to some information about the recipients which today is neither publicly available nor easy to obtain. Lets have a look how exactly signature and encryption are working.
They both depend on the availability of a key pair to operate. So the first step is the creation of a public/private key pair by a certificate authority (CA). For other platforms setting up a CA is quite a task, you can check some public sources and explanations for more details on that. In Lotus Notes that key pair is created by the Domino Administrator when registering new users (using Domino's CA process and kept save in Domino's ID Vault). Public/Private key pairs are very interesting constructs. Using advanced mathematics two strings are created with some interesting properties:

• You can't (at least not with timely effort) compute one of the keys out of the other
• When you encrypt something with your private key anybody holding the public key can decrypt and read it. This is useful to check if something really came from you without alterations
• When someone encrypts something with your public key only you (the owner of the private key) can decrypt and read it. This is useful to transmit confidential information

So in an ideal world anybody's public key would be accessible for anybody. In Notes the Public Key is saved into the Domino Directory on creation of the ID file. So it is convenient to use. In public the big certificate authorities like Thatwe or Verisign offer query possibilities for their database. Unfortunately there is no prevalent standard nor any of the big email programs or services offers easy access to all of this (I happily stand corrected). LDAP could be used for the task, but I yet see a public service there. I would expect something like a security service that knows where to look for keys and imports them as needed. Fiddling with PKS files is out of question. The complexity and inconvenience of managing your key is IMHO one of the big failures of the IT industry. So millions of sensitive messages go through the internet every single day (I would sniff out eMails from/to private banks for starters). How does it exactly work?

1. Encrypting:
   An encrypted message can only be read by the intended recipient
   
   When the sender wants to encrypt a message the email client requests the public key. That key can either be stored in the user's address book or be provided by the directory service. In a corporate environment the availability of a public key is the default for Lotus Notes. Other email systems need extra configuration and key-generation work. Once the key is retrieved the message is encrypted with this key and can only be decrypted with the private key of the recipient. Once the recipient receives the message her private key, stored in the Notes.id for Lotus Notes or the keystore for other applications, decrypts the messages and displays it. EMail encryption only encrypts the body of the message (including attachments), but not the subject line or the from/to fields.
2. Electronic Signature
   An electronic signature verifies the authenticity of the message content. The message itself is not encrypted when signed. Signatures can be used together with encryption. Used together signature comes first.
   
   The signing process computes a check sum (e.g. a MD5 hash value) from the body of the message. The resulting string gets encrypted with the public key of the sender. A recipient executes the same check sum computation and then uses the public key of the sender to decrypt the original result. When the two values match the recipient has the confirmation that the specific message really comes from that sender without any alterations

What would need to happen, so encryption would be available more easily? (Why that doesn't happen makes a field day for interested parties):

• Addition of a new record type to the DNS. It could be an ENC (for encryption) record type or a convention to use a TXT record like the SPF framework. It would point to one or more LDAP servers that can provide Public keys for the Domain of a recipient
• The naming standard for LDAP's X509 attributes gets implemented in an interoperable way
• eMail clients and eMail services would use the new DNS and LDAP entries to retrieve public keys when encryption is requested by a user or a signature needs to be verified. Of course some caching and deferred operation capabilities need to be build into the clients
• GMail, Yahoo mail and the big others offer certificates to users
• The other corporate eMail servers implement less painful certificate management

I doubt we see any of this anytime soon. After all the spooks don't want to spend all their CPU horsepower on regular citizens' eMail.