I previously discussed SPF and DKIM setup for the Postfix mail server. Here's some notes on TLS transport encryption. (Although, maybe those articles should have come in opposite order).

Using a self-signed certificate (which should be fine for small scale usage), setup is rather easy and straight forward. Creating the keys and certificats boils down to these instructions, copied from here. (Similar instructions here).

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -out rootCA.pem
openssl genrsa -out device.key 2048
openssl req -new -key device.key -out device.csr
openssl x509 -req -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out device.crt -days 500

Modifying /etc/postfix/main.cf, you might end up with something like this, assuming you've copied the keys as indicated by the linked article.
smtp_use_tls = yes
smtpd_use_tls = yes
 
smtp_tls_note_starttls_offer = yes
 
smtpd_tls_security_level = may
smtpd_tls_ask_ccert = yes
smtpd_tls_security_level = may
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom
 
smtpd_tls_key_file = /usr/share/ssl/certs/postfix/device.key
smtpd_tls_cert_file = /usr/share/ssl/certs/postfix/device.crt
smtpd_tls_CAfile = /usr/share/ssl/certs/postfix/rootCA.pem

Once all the changes are made, restart postfix:
service postfix restart

Now you can verify the setup with telnet:
telnet mail.example.com 25
 
EHLO example.com
STARTTLS

This should yield:
220 Ready to start TLS

Another way to confirm the setup is to send an email to a gmail.com account, and observe the lock status icon on the header field drop-down, explained in detail here.

Finally, the official Postfix documentation and notes on authentication (older doc) might come in handy.