So my question is: if my system has granted RELAY permission to a system which is in a dnsbl used by the sendmail configuration, does the sendmail RELAY, or does it deny the connection attempt?
Thanks for wading through this completely hypothetical situation.
I think you would be served by doing some googling on backscatter. Any time you have a "backup mx" server that does not do recipient validation for the domains it serves not only is it going to receive a lot of spam, it is going to be producing a lot. This is exactly the type of thing that lands IP addresses in blacklists in my experience. That being said you should be able to whitelist the IP of the blacklisted host before you do the rbl-checking. I know how to do this with postfix but not sendmail. I am not a sendmail user, but there are some sendmail users on the list who may be willing to help there.
My guess is that if you post to the mailing list of the MTA in question you may raise their ire a bit as you seem to be trying to solve a problem further downstream than you should be (idiots on your network).
I would fix your local problem (if you can).
alex