conversation. The question is: why are all of these remote servers trying to make connections back to me on high-numbered ports? Should I be allowing these connections somehow?
The remote server probably thinks that it's still supposed to be making connections back to you -- a couple of the lines you posted showed FIN flags indicating that the TCP connection was being shut down. At that point, the mail message has already been sent.
If you get REJECT messages for all SMTP connections, look at your iptables rules and see if you have a specific rule for smtp that only permits NEW conns.
One possibility is that iptables no longer thinks that the connection is active -- possibly the connection tracking database has already pushed that connection out. You can check your conntrack max value with the command
cat /proc/sys/net/ipv4/ip_conntrack_max
Yet another possibility is that these are duplicated packets (for whatever reason) and the connection has already been closed out.
M