This may work better with applications that decode the bits into individual list results, but it is a change from before and it may break other applications' use of the testpoints.
The multi.surbl.org BIND zone file contains:
test.surbl.org 604800 IN A 127.0.0.126 604800 IN TXT "multi.surbl.org permanent test point" test.multi.surbl.org 604800 IN A 127.0.0.126 604800 IN TXT "multi.surbl.org permanent test point" surbl-org-permanent-test-point.com 604800 IN A 127.0.0.126 604800 IN TXT "multi.surbl.org permanent test point" 2.0.0.127 604800 IN A 127.0.0.126 604800 IN TXT "multi.surbl.org permanent test point"The multi.surbl.org rbldnsd zone file contains:
test.surbl.org :126:multi.surbl.org permanent test point test.multi.surbl.org :126:multi.surbl.org permanent test point surbl-org-permanent-test-point.com :126:multi.surbl.org permanent test point 2.0.0.127 :126:multi.surbl.org permanent test pointThose resolve into the following Address records:
Name: test.surbl.org.multi.surbl.org Address: 127.0.0.126 Name: test.multi.surbl.org.multi.surbl.org Address: 127.0.0.126 Name: surbl-org-permanent-test-point.com.multi.surbl.org Address: 127.0.0.126 Name: 2.0.0.127.multi.surbl.org Address: 127.0.0.126But note that only the last, two-level domain surbl-org-permanent-test-point.com will work as the base domain for a URI in a test message for SpamAssassin. This is because URIs with test.multi.surbl.org.multi.surbl.org, etc., won't be detected by most SURBL-using programs because they're supposed to be reduced down to a two-level domain which would be surbl.org for those.
http://surbl-org-permanent-test-point-MUNGED.com/or:
http://127.0.0.2-MUNGED/without the "-MUNGED"s. So if you send yourself a message with any of those unmunged testpoints as URIs, the messages should match any SURBLs you have installed.
http://somedomain.com/can be rewritten as:
http://somedomain-MUNGED.com/That would require some awareness on the part of the person forwarding or discussing a listed site, but it's just as doable and humanly readable as munged email addresses, which people do all the time.
It's a good practice to use little or no filtering on your security mailing list messages and abuse contact addresses, or to bypass them around filtering.
http://drs.yahoo.com/covey/parr/*http://other.address/SpamCop itself seems to disambiguate (most of) the redirection. If someone is using a redirector to send traffic to somedomain.com, SpamCop seems to detect and resolve it correctly to somedomain.com most of the time. So the data that's used as input to sc.surbl.org already has redirectors correctly handled to some extent. In other words, we're protected on the data input side by the processing that happens at SpamCop to take out the redirection in reported URIs.
SpamAssassin programs such as SpamCopURI and urirhdbl that use SURBLs are capable of handling redirections to differing degrees. SpamCopURI 0.14 uses LWP to get Location information to untangle up to four levels of redirection sites without actually visiting the sites. URIDNSBL's urirhsbl includes patterns to extract the final domains from some redirection URIs. Further development will probably improve the handling of redirection sites.
The big picture solution is for the redirection sites to block abusive sites on their own. In other words, they should not let abusers redirect through their sites. Some redirection sites, such as tinyurl.com, reportedly actively block and report abusers of their site. Others such as Metamark and SnipURL are using SURBLs to deny abusers access to their redirection services. Here is an Open Letter to Redirection Sites that may be used or modified to contact them.
We've seen quite a few randomized or customized (to a username for example) host names in some of the top pill sites. There are different possible reasons for the randomization: to add chaos to the names to throw off message body checkers, or perhaps to "key" pill site web visits to specific mailings in order to build a confirmed mailing list. (Such confirmed mailing lists themselves are probably a valuable commodity to sell to other senders.) Randomization doesn't throw us off though; we catch them from the base domain part, which can't change.
Certainly the existing SURBL whitelist could be used to prevent Joe Jobs (false reporting or detection of legitimate domains). We've already added some of the common domains like yahoo, hotmail, ebay and amazon, etc. These seldom appear above the threshold yet, however, so the law of averages and careful reporting seem to be on our side so far.
(Note that the above comments apply to the handling of SpamCop URI data that goes into sc.surbl.org. However the gloabl whitelist applies to all SURBLs, including sc. Once a domain or IP address is whitelisted, it's excluded from all SURBLs.)
Update: Our whitelist, which we use only to exclude domains from SURBLs, not to "allow" messages, is growing but doesn't hit data from the various SURBLs too often. The goal is to keep whitehats off the lists in the first place by being careful with the input data. The whitelists are intended to be a safety backstop to make sure domains with legitimate uses don't get added.
URIDNBL rules: (Please see the actual rules for the standard list of domains to exclude from checking.)
# Top 125 domains whitelisted by SURBL uridnsbl_skip_domain yahoo.com w3.org msn.com com.com yimg.com uridnsbl_skip_domain hotmail.com doubleclick.net flowgo.com ebaystatic.com aol.com [...]SpamCopURI also has a built-in whitelist function:
whitelist_spamcop_uri *.yahoo.comOther SURBL applications may have similar exclusion features. If not, their authors may want to consider adding local whitelisting.
Notes:
There are also some security or privacy concerns about resolving a keyed domain name, since that could give out information about the success of a unsolicited messages, for example if the recipient is keyed in the full domain name as in:
http://resolving-this-confirms-specific-recipient.somedomain.com/The concern here is that the act of performing the resolution itself could be used as a confirmation of a delivery attempt given a URI customized (keyed) to a specific recipient. Such a confirmation could be used to build additional recipient lists, even if it just helps narrow down messages (and therefore recipients) which made it through the gauntlet of other filtering methods. In other words name resolution can potentially provide useful information to the senders.
Name resolution also adds a significant performance penalty, especially on high volume mail servers. For domains that don't resolve, a timeout of tens of seconds can result. These kinds of delays can make resolution of URI message body domains impractical for busy mail servers.
That said, if an IP address does appear in an unsolicited message's web site, then it can appear in SURBLs as an IP address. The principle is to accurately record what's in message body web sites. If there's a domain name in the message then that name can get onto a SURBL. If there's a number it too can get onto the SURBL.
Creating a list of the resolved addresses is something we considered. Doing so would be too similar to existing number-based approaches such as using the sbl.spamhaus.org list with the SpamAssassin command uridnsbl, of which SURBL-using urirhsbl is its domain-based twin. Another way to use number-based lists to check message body domains resolved into numbers is the sendmail milter which the SpamHaus site mentions: http://www.five-ten-sg.com/dnsbl.html .
However the current version of the sc.surbl.org data engine is a hybrid name and number approach, where if a domain resolves into an IP address commonly used with advertised sites, then that domain will get added to sc.surbl.org probably with the first report. (Note that this still requires at least one report, but the threshold for inclusion will be radically lower for major operators who repeatedly use the same IP address for their hosting.) This hybrid approach moves sc.surbl.org much closer towards the behavior of a number-based approach, though domains will still need that initial report, whereas a numbered list would catch the entire server by its IP address.
Of course a downside of using numbers is that they can false positive any legitimate domains that happen to be hosted on the same IP address as a blacklisted site. That could be disasterous for a large web hosting company that had one bad apple. That's another major reason why we went with names and not numbers. Numbers can be overly broad, whereas names are highly specific to the advertised site. To us names are a finer tool: if 30% of the domains on a given IP address are used by senders of unsolicited messages, then we could list all of them and not affect the 70% other domains that unfortunately happen to share the same IP address. That specificity is a strong benefit of using domain names.
The SpamAssassin 3 plugin URIDNSBL now has support for SURBL using its urirhsbl and urirhssub commands (see Quick Start and Usage sections), so that release of SA is covered.
Devin Carraway has written a plugin for the Perl-based MTA qpsmtpd to compare domains from message body URIs to SURBL domain lists. Here's his announcement on perl.qpsmtpd, and a link to his uribl plugin. Devin's was the first MTA implementation.
Exim, sendmail, postfix, qmail, qmail-ldap and Exchange programs and plugins have been written to support SURBLs in those MTA. (Please see the Links and News pages for more information.) Support of SURBL directly in other MTAs would also be useful. MTA support can reject messages directly back to the sender. MTA support has the added advantage of pre-empting other, more expensive processing of messages, for example in SpamAssassin. There is still code to write for any mail filter or MTA that doesn't support SURBLs yet.
A very popular and fast name server specifically meant for serving up lists is rbldnsd. SURBL zone files are available in rbldnsd format. (They are also available in BIND format, though rbldnsd is recommended as significantly leaner and faster.) There are links and instructions for using rbldnsd with rsync in the Links section.
Then arrange with the lists to get rsync access to their zone files. Since rsync only transmits differences, the zone files are kept updated in a very efficient manner. To get rsync access to SURBL zone files, please fill out our rsync access form. Other lists may have similar procedures for gaining rsync access.
Then configure your mail or SpamAssassin servers using lists to do the lookups on your local list name server. Many people run the local DNS for their lists on their mail server(s), which tends to work well since it keeps everything on the same box. If your mail server is separate from your list name server, then set up DNS on the mail server to resolve using that list name server. The following documents may be helpful in setting up local caching:
A solution is to disable such site correction or modification features on servers or clients doing SURBL queries. Alternatively, consider using regular (non-modifying) nameservers for those systems. Often the best solution is to set up a local caching nameserver.
Note also that SURBL applications may be incompatible with DNS modification or proxy services that change the DNS query results of non-matches (NXDOMAIN results) for non-existent sites.
Note that as of 1/25/07, OpenDNS no longer modifies results for SURBL lists. It should now be safe to use OpenDNS with SURBL applications. If you find you are behind a firewall or proxy that is modifying SURBL DNS queries incorrectly, one solution is to set up a local caching nameserver. A local caching server can significantly improve performance also.
For SpamAssassin 2.63 and 2.64, SpamCopURI is a program that can be used with SURBLs.
Important Note: Matt Kettler says: DO NOT run SA 2.63 on a production server. Upgrade to 2.64 or 3.x because 2.63 has a MIME parsing bug that can be used to DoS your server.
For SpamAssassin 3.X, there is a suite of programs contained in the plugin URIDNSBL including some that can be used with SURBLs and some that can't:
For more information, please see SpamAssassin documentation and community.
For example, if you're using spamd, make sure it's started without the -L or --local flags, which force local tests only.
If you are running Amavis, make sure amavisd.conf has $sa_local_tests_only = 0. (Uncomment this line if it was commented out before, then set the value to zero to enable network tests.)
If you are using MIMEDefang, make sure you set $SALocalTestsOnly to zero:
# If boolean true, skip SA network tests
$SALocalTestsOnly = 0;
to enable SpamAssassin network tests from your mimedefang-filter.
Also make sure that you have a recent Net::DNS installed. Too old versions of Net::DNS seems to be a common reason for RBL checks not working, especially when upgrading from an older version of SpamAssassin.
urirhssub URIBL_JP_SURBL multi.surbl.org. A 64
body URIBL_JP_SURBL eval:check_uridnsbl('URIBL_JP_SURBL')
describe URIBL_JP_SURBL Contains a URL listed in the JP SURBL list
tflags URIBL_JP_SURBL net
score URIBL_JP_SURBL 3.0
Where body above was previously header.
Here is the changelog reference:
r54022 | felicity | 2004-10-07 22:21:30 +0000 (Thu, 07 Oct 2004) | 1 line bug 3734: uridnsbl rules work on body data, not header data, so change the rule type from header to body
SURBLs need to be used with programs that can extract domains from message body URIs, which is different from traditional list usage.
In contrast, most other lists have the IP addresses or domain names of unsolicited message senders, open relays, open proxies, etc. Programs that use these traditional lists generally check those list entries against connections or message headers. They generally do not check message bodies, i.e., the content of messages. Most prior efforts to look at message body URIs have converted the sites they find into IP addresses for comparison with a numeric list.
We feel that SURBLs directly address the web sites seen in unsolicited messages. We also feel that there is a significant performance advantage in comparing SURBL sites to message body sites, since there is no delay needed to resolve domain names into numbers.
However the act of resolving the domain can confirm for senders of unsolicited messages that your specific address was reachable, that you've opened their message, etc. And the name resolution can significantly delay the amount of time it takes for your provider's servers to process each message. (The delay is on the order of a few seconds to a few minutes; not a big deal to an end user, but a major bottleneck to the provider's servers which typically need to handle thousands of messages much more quickly.)
In contrast, SURBLs contain mostly domain names that have appeared in message body URIs. Typically those are the web sites that are being advertised. Using SURBLs doesn't require resolving the domains that appear in messages. That's safer, more private and much faster.
While some SURBL lists such as SC and AB use data from SpamCop, it's the Spamvertised sites that they use and not the sender IP addresses. Those are the web sites advertised in the message bodies which have been reported to SpamCop.
The difference is between detecting senders based on message headers, as the SpamCop Block List is commonly used for, versus detecting based on URIs advertised in message bodies, which is what SURBLs are used for. This is useful because senders of unsolicited messages frequently shift the IP addresses they send from, but they tend to advertise the same sites repeatedly.