Also known as Command and Control Servers (C&C), botnets are an interesting phenomenon that involves a group of compromised host machines that are controlled by a smaller number of command hosts, which sometimes are controlled by a "bot herder", the botnet's originator.

A recent article about detecting email spamming botnets was published by a series of Microsoft Researchers where they developed an innovative system called AutoRE. Naturally they tested the framework against MSN Hotmail, and their findings were pretty compelling.

AutoRE: Automatic RegEx Spam Filter

AutoRE is a framework for detecting email spam botnets. By inspecting the content of an email, AutoRE automatically generates signatures based on the discovered URLs and utilizes regular expressions (regEx) to account for polymorphic URL variants (e.g. unique tracking parameters, campaignID=2323). The unique combination of grouping IP addresses, sending time, and URL signatures with the addition of regEx signatures, they were able to capture an additional 16% - 18% of the spam that slips through traditional spam filters (e.g. spamhaus).

"To date, detecting and blacklisting individual bots is commonly regarded as difficult, due to both the transient nature of the attack and the fact that each bot may send only a few spam emails."

In addition, because AutoRE uses regular expressions, it is also capable of detect future botnets regardless of domain name, which many spammers will churn and burn through. All the while, performing these feats with a very high precision rate generating very few false positives.

Compared with complete URL (fixed string) based signatures, regular expression signatures are more robust and can detect 10 times more spam emails.

Low false positive rate: Using AutoRE signatures, we identified 580,466 spam emails with a false positive rate of 0.002. AutoRE’s false positive rate in detecting botnet hosts is less than 0.005.

How Does This Affect Me?

Well, as a legitimate email marketer it really shouldn't. Although this article is primarily geared toward preventing spam from botnets, there are some good tidbits that can be taken away.

You always want to do as much as possible to ensure your email campaign isn't perceived as spam. Here are a few precautions if you don't want to be caught by AutoRE:

• Avoid sending your emails from the following networks: Korea Telecom, Verizon Internet Service, France Telecom, China 169-backbone, China-backbone
  • These ISPs were identified be the Top 5 originators of botnet IP addresses
• Scrub your email list for invalid email addresses
  • Having a high ratio of invalid recipients to valid recipients is a signal that you could be a spammer
• Limit the number of recipients per email and Throttle the number of SMTP connections per second
  • A high number of recipients and connections per second are indications of aggressiveness
  • Botnets tend to be "bursty" in nature, typically sending the bulk of an email campaign within a 5 day period
• Limit the number of servers which you send emails from
  • Most botnets have tens to hundreds (10s-100s) of IP addresses. The largest in the test sample had 1384 IPs

Legitimate emails sent by a big company advertising a product or event could also be bursty. But they will be unlikely sent from hosts spanning more than a few [IP blocks].

As a consumer and webmaster, this would be a wonderful spam fighting tool to add to the arsenal. Although it's been very effective in tests, it hasn't been tested in real time on a production environment. We can only hope that they're able to make further advancements on this interesting framework.

Source & Citations

Authors:

Read more about the authors, Yinglian Xie and Fang Yu, and their research on combating spam by detecting dynamic IP addresses.

For more information, read the full article [PDF] at Computer Communication Review Online or at the ACM Digital Library Portal.

Abstract

In this paper, we focus on characterizing spamming botnets by leveraging both spam payload and spam server traffic properties. Towards this goal, we developed a spam signature generation framework called AutoRE to detect botnet-based spam emails and botnet membership. AutoRE does not require pre-classified training data or white lists. Moreover, it outputs high quality regular expression signatures that can detect botnet spam with a low false positive rate. Using a three-month sample of emails from Hotmail, AutoRE successfully identified 7,721 botnet-based spam campaigns together with 340,050 unique botnet host IP addresses.

Our in-depth analysis of the identified botnets revealed several interesting findings regarding the degree of email obfuscation, properties of botnet IP addresses, sending patterns, and their correlation with network scanning traffic. We believe these observations are useful information in the design of botnet detection schemes.