OEM Spam Filtering Engines

Customer

Founded in 1999 and with offices in US, Europe, and Asia, the customer is a leading provider of OEM spam filtering and anti-phishing softwar, competing with the likes of Symantec, McAfee, and Cloudmark. Over 5,000 companies and 10 million consumers worldwide rely on them for their services.

Business Case

The goal was to develop a spam filtering engine that offered information actuality, up-to-date categorization, scalability, reliability, framework flexibility and extensibility. The client needed an ability to scan as many URLs for as many categories/fraudulent practices/phishing/spam/viruses/etc. as possible.

Solution

The project has been a two year and a fifteen developer effort which produced an OEM spam filtering engine with the following parts:

SDK
Anti-Spam SDK is a software library that provides classes to communicate with the spam filtering engine. Functions are provided to return a 'spam score' for each message.
The SDK includes an API (Application Programming Interface) that allows developers to integrate Anti-Spam engine with other applications, along with more than 40 configuration options that allow OEMs to balance memory usage, throughput and detection.

For every message:
The four spam identification engines utilize more than one million filter checks to identify spam with an incredibly high accuracy rate, and a near-zero false positive rate.
Creates a fingerprint ID for every e-mail message and compares it to existing fingerprints in its databases.
Results are combined using a proprietary Bayesian statistical formula, into one final spam probability score.

Image Spam
The engine looks for image attributes that are unlikely to exist in legitimate email. These include, but are not limited to:
Jigsaw puzzle-style images.
CAPTCHA-style images that intentionally obscure content.
Images designed to emulate plain text.
Though the presence of these techniques does not guarantee that the message is spam, scoring algorithms can penalize these messages to ensure consistently high accuracy.
Engine’s image analysis is a subset of our approach to quantifying the reputation of each attribute of each message.
Engine treats images—and parts of images—as attributes that can be extracted and tracked over large numbers of messages.
Engine defines reputation of an attribute as the difference between the number of spam versus the number of legit messages for that attribute.

Phishing
The Engine combines its own home-grown reputation filters, along with global access to advanced data networks, to block phishing and other forms of email fraud.
Reputation analysis and email authentication help the system identify the rightful owners of IP addresses, domains, email address, and even message content.
Engine’s global data network includes near real-time reporting of phishing outbreaks.
The SDK also includes an API to identify and segregate phishing from other types of spam. This allows OEMs to reject, delete, quarantine, etc., phishing attacks before they reach customers' inboxes.

Features

Additional features include:
Scan IP addresses for spam/legit attributes.
Extensive Spyware Database
Verify owner of an IP/Domain address and Software
New Ximian Evolution plugin
Ability to prevent, detect, and disinfect zombie machines
Detection of viruses in real-time with and without signatures.
New Novell Groupwise Plugin
Custom override the weight of any rule.
Block spammers who spoof domain names.
Tune performance and accuracy by setting the size of the message to be read and to be scanned.
Networks checks throughput can be increased by being used only when most needed.

Benefits

At its default settings, the SDK catches more than 95% of spam with less than 0.005% 'false positives.' Virtually all of the false positives are non-English bulk emails such as newsletters and legitimate advertisements.

Tools and Technologies

Supported platforms include: Linux (Certified for Redhat, Mandrake, and Suse), Microsoft Windows,Solaris 8 (Sparc), Solaris Intel, FreeBSD, AIX, Mac OS X, HP-UX

Languages and Tools: C/C++, Perl, PHP, Apache, Sendmail, Gcc compiler, Gdb debugger, Gprof, Valgrind memory leak checker, Flex text parser