diff -ruN Spam-Filtering-for-MX-v0.12/chapter-background.xml Spam-Filtering-for-MX/chapter-background.xml --- Spam-Filtering-for-MX-v0.12/chapter-background.xml 2004-07-29 23:38:44.000000000 -0700 +++ Spam-Filtering-for-MX/chapter-background.xml 2004-08-02 11:32:13.000000000 -0700 @@ -59,7 +59,7 @@ standing is probably, well, ignorance. If you have been doing any type of spam filtering, even by manually moving mails to the trash can in your mail reader, let alone by experimenting with - primitive filtering techniques such as DNS blocklists (SpamHaus, + primitive filtering techniques such as DNS blacklists (SpamHaus, SPEWS, SORBS...), chances are that you have lost some valid mail. @@ -358,7 +358,7 @@ My view stands in sharp contrast to that of a large number of spam hacktivists, such as the maintainers of the SPEWS - blocklist. One of the stated + blacklist. One of the stated aims of this list is precisely to inflict as a means to put pressure on ISPs to react on abuse complaints. Listing complaints are typically diff -ruN Spam-Filtering-for-MX-v0.12/chapter-considerations.xml Spam-Filtering-for-MX/chapter-considerations.xml --- Spam-Filtering-for-MX-v0.12/chapter-considerations.xml 2004-07-28 13:40:54.000000000 -0700 +++ Spam-Filtering-for-MX/chapter-considerations.xml 2004-08-01 23:27:34.000000000 -0700 @@ -17,11 +17,11 @@ Most domains list more than one incoming s (a.k.a. MX hosts). If you do so, then bear in - mind that in order to have any effect, any SMTP time filtering you - incorporate on the primary MX has to be duplicated on all others - too. Otherwise, the sending host would simply sidestep - filtering by retrying the mail delivery through your backup - server(s). + mind that in order to have any effect, any SMTP time filtering + you incorporate on the primary MX has to be incorporated on all + the others as well. Otherwise, the sending host would simply + sidestep filtering by retrying the mail delivery through your + backup server(s). diff -ruN Spam-Filtering-for-MX-v0.12/chapter-techniques.xml Spam-Filtering-for-MX/chapter-techniques.xml --- Spam-Filtering-for-MX-v0.12/chapter-techniques.xml 2004-07-30 00:00:47.000000000 -0700 +++ Spam-Filtering-for-MX/chapter-techniques.xml 2004-08-04 11:41:56.000000000 -0700 @@ -105,7 +105,9 @@ linkend="callback"/>s is 30 seconds - if you impose delays this long, the peer's sender callout verification would fail, and in turn the original mail delivery from you/your user might be - rejected. + rejected (usually with a temporary failure, which means the + message delivery will be retried for 5 days or so before finally + returned to the sender). In other words, 20 seconds is about as long as you can stall @@ -152,7 +154,7 @@ incorporate some less conclusive checks that we will discuss in the following sections. You probably do not wish to reject the mail outright based the results from e.g. the SPEWS blocklists, but on the other hand, it may + linkend="dnsbl">blacklist, but on the other hand, it may provide a strong enough indication of trouble that you can at least impose transaction delays. After all, legitimate mail deliveries are not affected, other than being subjected to a @@ -163,14 +165,27 @@ Conversely, if you find conclusive evidence of spamming (e.g. by way of certain ), and your server can afford it, you may choose to impose an extended delay, - e.g. 15 minutes or so, before finally rejecting the delivery. + e.g. 15 minutes or so, before finally rejecting the delivery + + + Beware that while you are holding up an incoming SMTP + delivery, you are also holding up a TCP socket on your + server, as well as memory and other server resources. If + your server is generally busy, imposing SMTP transaction + delays will make you more vulnerable to Denial-of-Service + attacks. A more scalable option may be to + drop the connection once you have conclusive evidence that + the sender is a ratware client. + + . This is for little or no benefit other than slowing down the spammer a little bit in their quest to reach as many people as - possible before DNS blocklists and other collaborative network + possible before DNS blacklists and other collaborative network checks catch up. In other words, pure altruism on your side. :-) + In my own case, selective transaction delays and the resulting SMTP synchronization errors account for nearly 50% of rejected @@ -192,7 +207,7 @@ Some indication of the integrity of a particular peer can be gleaned directly from the Domain Name Service, DNS, even before - SMTP commands are issued. In particular, various DNS blocklists + SMTP commands are issued. In particular, various DNS blacklists can be consulted to find out if a particular IP address is known to violate or fulfill certain criteria, and a simple pair of forward/reverse (DNS/rDNS) lookups can be used as a vague @@ -218,23 +233,30 @@ -
- DNS Blocklists +
+ DNS Blacklists - DNS blocklists (DNSbl's, formerly called "Real-time Black-hole - Lists" after the original blocklist, "mail-abuse.org") make up + DNS blacklists (DNSbl's, formerly called "Real-time Black-hole + Lists" after the original blacklist, "mail-abuse.org") make up perhaps the most common tool to perform transaction-time spam blocking. The receiving server performs one or more rDNS lookups of the peer's IP address within various DNSbl zones, - such as "bl.spamcop.net", "sbl-xbl.spamhaus.org", - "dnsbl.sorbs.net", and so forth. If a matching DNS record is - found, a typical action is to reject the mail delivery. (A - notable exception is "bondedsender.org", which lists "trusted" - IP addresses, and whose owners have posted a financial bond - that will be debited in the event that someone feel they have - received spam from the subscriber. The idea is that these - addresses would be white-listed by the recipient). + such as "dnsbl.sorbs.net", "opm.blitzed.org", + "lists.dsbl.org", and so forth. If a matching DNS record is + found, a typical action is to reject the mail delivery. + + + Similar lists exist for different purposes. For instance, + bondedsender.org is a DNS + whitelist (DNSwl), containing + trusted IP addresses, whose owners have + posted a financial bond that will be debited in the event + that spam originates from that address. Other lists + contain IP addresses in use by specific countries, + specific ISPs, etc. + + @@ -301,12 +323,12 @@ For that reason, rather than rejecting mail deliveries outright based on a single positive response from DNS - blocklists, many administrators prefer to use these lists in a + blacklists, many administrators prefer to use these lists in a more nuanced fashion. They may consult several lists, and assign a "score" to each positive response. If the total score for a given IP address reaches a given threshold, deliveries from that address are rejected. This is how DNS - blocklists are used by filtering software such as + blacklists are used by filtering software such as SpamAssassin (). @@ -463,12 +485,20 @@ - First, there is no reason to accept a plain IP address in - the Hello greeting. Even if you wish to generously allow - everything RFC 2821 mandates, recommends, and suggests, you - will note that IP addresses should always be enclosed in - square brackets when presented in lieu of a fully qualified - domain name. + First, feel free to reject plain IP addresses in the Hello + greeting. Even if you wish to generously allow everything + RFC 2821 mandates, recommends, and suggests, you will note + that IP addresses should always be enclosed in square + brackets when presented in lieu of a name. + + + Although this check is normally quite effective at + weeding out junk, there are reports of buggy L-Soft + listserv + installations that greet with the plain IP address of + the list server. + + @@ -502,7 +532,6 @@ negations) legitimate. - Similarly, you can reject host names that contain invalid characters. For Internet domains, only alphanumeric letters @@ -528,7 +557,6 @@ delay after each SMTP command (HELO/EHLO, MAIL FROM:, RCPT TO:). -
@@ -558,6 +586,11 @@ + If either of these two checks succeeds, the name has been + verified. + + + Your MTA may have a built-in option to perform this check. For instance, in Exim (see ), you want to set "helo_try_verify_hosts = *", and create ACLs @@ -593,8 +626,16 @@ After the client has presented the MAIL FROM: <address> - command, you can validate the supplied address as follows. + command, you can validate the supplied + address as follows. + + + A special case is the NULL envelope sender address + (i.e. MAIL FROM: <>) used in + s and other automatically generated + responses. This address should always be accepted. + +
@@ -606,7 +647,7 @@ Is the domain part a syntactically valid ? - + Often, your MTA performs these checks by default. @@ -650,25 +691,40 @@ Sender Callout Verification - This is a mechanism that is offered by the Exim MTA to - validate the local part of a remote sender - address. + This is a mechanism that is offered by some MTAs, such as + Exim and Postfix, to validate the local part + of a remote sender address. In Postfix terminology, it is + called Sender Address Verification. Your server contacts the MX for the domain provided in the sender - address, attempting to initiate a SMTP transaction as if - returning mail to this address (i.e. with an empty address). It does not actually send any - mail; rather, once the RCPT TO: command - has been either accepted or rejected by the remote host, - your server sends QUIT. + address, attempting to initiate a secondary SMTP transaction + as if delivering mail to this address. It does not actually + send any mail; rather, once the RCPT TO: + command has been either accepted or rejected by the remote + host, your server sends QUIT. - The goal is to determine if a would be - accepted if returned to the sender. + By default, Exim uses an empty envelope sender address for + such callout verifications. The goal is to determine if a + would be accepted if returned to the + sender. + + + + Postfix, on the other hand, defaults to the sender address + <domain> + for address verification purposes + (domain is taken from the + variable). For this reason, you + may wish to treat this sender address the same way that you + treat the NULL envelope sender (for instance, avoid or , but + require s in recipient + addresses). More on this in the implementation appendices. @@ -752,8 +808,8 @@ Preventing your servers from acting as open relays is extremely important. If your server is an open relay, and spammers find you, you will be listed in numerous DNS - blocklists instantly. If the maintainers of certain other - DNS blocklists find you (by probing, and/or by acting on + blacklists instantly. If the maintainers of certain other + DNS blacklists find you (by probing, and/or by acting on complaints), you will be listed in those for an extended period of time. @@ -761,8 +817,8 @@ -
- Recipient Address Verification +
+ Recipient Address Lookups This, too may seem banal to most of us. It is not always so. @@ -816,11 +872,12 @@ Recipient Callout Verification - This is a mechanism that is offered by the Exim MTA to - verify the local part of a remote recipient - address (see local part + of a remote recipient address (see for a description of how - this works). + this works). In Postfix terminology, this is called + Recipient Address Verification. @@ -893,14 +950,14 @@ If the machine(s) that host(s) your mailboxes is/are - running on some flavor of UNIX or Linux, you can perform - this task by way of a so-called cron job - (type man cron for details). This - could be a script that would generate such a list, perhaps - from the local /etc/passwd file, and then - copied it to your MX host(s) using the scp - command from the OpenSSH suite. + running on some flavor of UNIX or Linux, you could write a + script to first generate such a list, perhaps from the + local /etc/passwd file, and then copy it to + your MX host(s) using the scp command from + the OpenSSH + suite. You could then set up a cron job + (type man cron for details) to + periodically run this script.
@@ -943,7 +1000,7 @@ Legitimate s should be sent to only one recipient address - the originator of the original message that triggered the notification. You can drop the - connection if if the address is + connection if the address is empty, but there are more recipients than one.
@@ -1042,6 +1099,16 @@ expire, because certain ISPs (such as earthlink.net) retry deliveries only every 6 hours or similar. + + + Large sites often use multiple servers to handle outgoing + mail. For instance, one server or pool of servers may be + used for immediate delivery. If the first delivery + attempt fails, the mail is handed off to a fallback server + which has been tuned large queues. Hence, from such + sites, the first two delivery attempts will fail. + +
@@ -1095,7 +1162,9 @@ incoming mail exchangers. However, since the machine that hosts this database would become a single point of failure, you would have to take a sensible action if that machine is - down (e.g. accept all deliveries). + down (e.g. accept all deliveries). Or you could use database + replication techniques and have the SMTP server fall back to + one of the replicating servers for lookups. @@ -1571,16 +1640,33 @@ - One particular case is where the message contains a NULL - character (ordinal zero). Even if you decide that figuring + One particular case is where the message contains NUL + characters (ordinal zero). Even if you decide that figuring out what a non-printable character means is more complex than beneficial, you might consider checking - for NULL characters. That is because some s, such as the - Cyrus Mail - Suite, will ultimately reject mails that contain this - character. If you are delivering to such software, you should - definitely get rid such messages. + for this character. That is because some s, such as the Cyrus Mail Suite, + will ultimately reject mails that contain it. + + + The IMAP protocol does not allow for NUL characters to be + transmitted to the mail user agent, so the Cyrus + developers decided that the easiest way to deal with mails + containing it was to reject them. + + . + + If you use such software, you should definitely consider + getting rid of NUL characters. + + + + On the other hand, the (now obsolete) RFC 822 specification + did not explicitly prohibit NUL characters in the message. + For this reason, as an alternative to rejecting mails + containing it, you may choose to strip these characters off + the message before delivering it to Cyrus. diff -ruN Spam-Filtering-for-MX-v0.12/impl-exim.xml Spam-Filtering-for-MX/impl-exim.xml --- Spam-Filtering-for-MX-v0.12/impl-exim.xml 2004-07-29 23:51:46.000000000 -0700 +++ Spam-Filtering-for-MX/impl-exim.xml 2004-08-02 15:03:54.000000000 -0700 @@ -533,7 +533,7 @@ drop message = Legitimate bounces are never sent to more than one \ recipient. - senders = : + senders = : postmaster@* condition = $recipients_count @@ -622,7 +622,7 @@ message = Your message does not conform to RFC2822 standard log_message = missing header lines !hosts = +relay_from_hosts - !senders = : + !senders = : postmaster@* condition = ${if or {{!def:h_Message-ID:}\ {!def:h_Date:}\ {!def:h_Subject:}} {true}{false}} @@ -699,7 +699,7 @@ If you are like me, you want to be a little bit more selective about which hosts you subject to SMTP transaction delays. For instance, as described earlier in this - document, you may decide that a match from a DNS blocklist + document, you may decide that a match from a DNS blacklist or a non-verifiable Hello greeting are not conditions that by themselves warrant a rejection - but they may well be sufficient triggers for transaction delays. @@ -1039,7 +1039,7 @@ Please try later. log_message = greylisted. domains = +local_domains : +relay_to_domains - !senders = : + !senders = : postmaster@* set acl_m9 = $sender_host_address $sender_address $local_part@$domain set acl_m9 = ${readsocket{/var/run/greylistd/socket}{$acl_m9}{5s}{}{}} condition = ${if eq {$acl_m9}{grey}{true}{false}} @@ -1074,7 +1074,7 @@ delivery status reports to <$recipients>. \ Please try later. log_message = greylisted. - senders = : + senders = : postmaster@* set acl_m9 = $sender_host_address $recipients set acl_m9 = ${readsocket{/var/run/greylistd/socket}{$acl_m9}{5s}{}{}} condition = ${if eq {$acl_m9}{grey}{true}{false}} @@ -1305,9 +1305,9 @@ # use a warn verb to set a new expire time on automatic records, # but only if the mail was not a bounce, otherwise set to now(). - warn !senders = : + warn !senders = : postmaster@* condition = ${lookup mysql{GREYLIST_OK_NEWTIME}} - warn senders = : + warn senders = : postmaster@* condition = ${lookup mysql{GREYLIST_OK_BOUNCE}} deny @@ -1322,7 +1322,7 @@ .ifdef GREYLIST_ENABLED - defer !senders = : + defer !senders = : postmaster@* acl = greylist_acl message = greylisted - try again later .endif @@ -1337,7 +1337,7 @@ .ifdef GREYLIST_ENABLED - defer senders = : + defer senders = : postmaster@* acl = greylist_acl message = greylisted - try again later .endif @@ -1771,7 +1771,7 @@ - Also, a naive Bayesian scoring + Also, a Bayesian scoring feature is built in, and is turned on by default. We normally want to turn this off, because it requires training that will be specific to each user, and thus is not suitable for @@ -1802,7 +1802,7 @@ scoring (though I don't see why Although it is true that Bayesian training is specific to - each user, it should be noted that SpamAssassin's naive + each user, it should be noted that SpamAssassin's Bayesian classifier is, IMHO, not that stellar in any case. Especially I find this to be the case since spammers have learned to defeat such systems by seeding random dictionary @@ -1815,7 +1815,7 @@ As discussed in the section earlier in the document, there is a way for this to happen. - We need to limit the the number of recipients we accept per + We need to limit the number of recipients we accept per incoming mail delivery to one. We accept the first RCPT TO: command issued by the caller, then defer subsequent ones using a 451 SMTP @@ -1989,7 +1989,7 @@ - Nedless to say, after making these changes, you need to + Needless to say, after making these changes, you need to restart spamd. @@ -2304,7 +2304,7 @@ return path from here.\n\ You are responding to a forged sender address. log_message = bogus bounce. - senders = : + senders = : postmaster@* domains = +local_domains set acl_m9 = /home/${extract{1}{=}{${lc:$local_part}}}/.return-path-sign condition = ${if exists {$acl_m9}{true}} @@ -2338,7 +2338,7 @@ # indicate that we should keep stalling the sender. # warn - senders = : + senders = : postmaster@* domains = +local_domains set acl_m9 = /home/${extract{1}{=}{${lc:$local_part}}}/.return-path-sign condition = ${if exists {$acl_m9}{true}} @@ -2395,7 +2395,7 @@ message = This address never sends outgoing mail. \ You are responding to a forged sender address. log_message = bogus bounce for system user <$local_part@$domain> - senders = : + senders = : postmaster@* domains = +local_domains !mailbox check @@ -2485,7 +2485,7 @@ can ensure that incoming s are not routed through it by adding the following condition to the router: - !senders = : + !senders = : postmaster@* @@ -2494,7 +2494,7 @@ system_aliases: driver = redirect domains = +local_domains - !senders = : + !senders = : postmaster@* allow_fail allow_defer data = ${lookup{$local_part}lsearch{/etc/aliases}} @@ -2666,7 +2666,7 @@ message = Your message does not conform to RFC2822 standard log_message = missing header lines !hosts = +relay_from_hosts - !senders = : + !senders = : postmaster@* condition = ${if !eq {$acl_m0}{accept}{true}} condition = ${if or {{!def:h_Message-ID:}\ {!def:h_Date:}\ @@ -3055,7 +3055,7 @@ drop message = Legitimate bounces are never sent to more than one \ recipient. - senders = : + senders = : postmaster@* condition = $recipients_count delay = 5m @@ -3125,7 +3125,7 @@ # return path from here.\n\ # You are responding to a forged sender address. # log_message = bogus bounce. - # senders = : + # senders = : postmaster@* # domains = +local_domains # set acl_m9 = /home/${extract{1}{=}{${lc:$local_part}}}/.return-path-sign # condition = ${if exists {$acl_m9}{true}} @@ -3146,7 +3146,7 @@ # message = This address never sends outgoing mail. \ # You are responding to a forged sender address. # log_message = bogus bounce for system user <$local_part@$domain> - # senders = : + # senders = : postmaster@* # domains = +local_domains # set acl_m9 = ${extract{1}{=}{${lc:$local_part}}} # @@ -3191,7 +3191,7 @@ # Please try later. # log_message = greylisted. # domains = +local_domains : +relay_to_domains - # !senders = : + # !senders = : postmaster@* # set acl_m9 = $sender_host_address $sender_address $local_part@$domain # set acl_m9 = ${readsocket{/var/run/greylistd/socket}{$acl_m9}{5s}{}{}} # condition = ${if eq {$acl_m9}{grey}{true}{false}} @@ -3277,7 +3277,7 @@ # message = Your message does not conform to RFC2822 standard # log_message = missing header lines # !hosts = +relay_from_hosts - # !senders = : + # !senders = : postmaster@* # condition = ${if !eq {$acl_m0}{accept}{true}} # condition = ${if or {{!def:h_Message-ID:}\ # {!def:h_Date:}\ @@ -3308,7 +3308,7 @@ # delivery status reports to <$recipients>. \ # Please try later. # log_message = greylisted. - # senders = : + # senders = : postmaster@* # condition = ${if !eq {$acl_m0}{accept}{true}} # set acl_m9 = $sender_host_address $recipients # set acl_m9 = ${readsocket{/var/run/greylistd/socket}{$acl_m9}{5s}{}{}} diff -ruN Spam-Filtering-for-MX-v0.12/Spam-Filtering-for-MX.xml Spam-Filtering-for-MX/Spam-Filtering-for-MX.xml --- Spam-Filtering-for-MX-v0.12/Spam-Filtering-for-MX.xml 2004-07-29 23:40:42.000000000 -0700 +++ Spam-Filtering-for-MX/Spam-Filtering-for-MX.xml 2004-08-04 11:44:46.000000000 -0700 @@ -31,17 +31,26 @@ - + + + + Devdas + Bhagat + +
devdas (at) dvb.homelinux.org
+ + Technical Review + - Version 0.12 -- Released for technical review + Version 0.15 -- Final technical review anti-spam @@ -83,7 +92,7 @@ The discussions are conceptual in nature, but a sample - implementation is provided using the Exim MTAs and other + implementation is provided using the Exim MTA and other specific software tools. Miscellaneous other bigotry is expressed throughout. @@ -112,14 +121,8 @@ The newest version of this document can be found at . + Please check back periodically for corrections and additions. - - - This document is currently under review, and is undergoing - rapid changes. Please change back often for corrections and - additions. - -
@@ -129,6 +132,36 @@ + 0.15 + 2004-08-04 + TS + + Incorporated second round of changes from technical + review by Devdas Bhagat. + + + + + 0.14 + 2004-08-01 + TS + + Incorporated technical review comments/corrections from + Devdas Bhagat. + + + + + 0.13 + 2004-08-01 + TS + + Incorporated technical review comments/corrections from + Joost De Cock. + + + + 0.12 2004-07-27 TS @@ -409,15 +442,9 @@ - Timothy M. Lyons lyons (at) digitalvoodoo.org - is currently working on a Sendmail implementation of the - techniques described in this document. If you would like to - help out, please send him an e-mail. - - - If you are able to provide implementations for other s, such as Postfix, please let me know. + linkend="mta"/>s, such as Sendmail or Postfix, please let me + know.