EW3D SpamProbe Use Instructions

Each user has their own database which SpamProbe uses to 'score' your email based on your individual preferences. You must feed information in the form of emails to this database. SpamProbe then uses your personal database to score email as spam or good and places good mail in your 'Inbox' and spam in the 'your_spam' box. You 'feed' the database by moving or copying email into the various other 'training' mailboxes which we subscribed to in the installation instructions.

The Mailboxes

• 'Inbox' is a standard Inbox available to you via your standard POP account and your IMAP account (same place on the server). Mail scored as not spam is delivered there.
• 'your_spam' is an IMAP mailbox residing in the IMAP area on our server. This is where mail scored as spam is placed.
• The 'training' boxes are placed in the directory named 'Corrections' in your IMAP folder.
• The following explains the mailboxes in the 'Corrections' folder:
  • Important note about the mailboxes in the 'Corrections' folder: Mail placed in these folders is automatically deleted after SpamProbe trains it to the database. This is done at particular hours of each day. Items placed here could be deleted almost immediately.
  • 'is_spam' is where you move mail which you consider to be spam
  • 'non_spam' is where you 'copy or move' mail which you consider to be good
  • 'not_scored' is where you 'copy or move' mail which could degrade your database (perhaps a joke from a friend containing a particularly spammy term.. perhaps the name of some drug highly advertised in spam?)

Using These Mailboxes to Train SpamProbe

SpamProbe is born with little knowledge, only the ability learn, and listen to corrections made by you
It is important to work with SpamProbe frequently just after we set it up. SpamProbe does a very poor job of scoring on the first emails. 'Training' it frequently and often at first, keeps you from having to overcome early mistakes. At first, you will get some spam in your 'Inbox' and some good mail in the 'your_spam' box. It is time to begin the 'training'.

How SpamProbe works
SpamProbe in a nutshell, reads every bit of text in every email, headers and body, and records that information into a database. Words and terms (word sets) are given a score, less = not spam, higher = spam. As soon as it is installed on your account it starts making these decisions. Good mail is scored as 'twice' good, spam is scored as 'once' spam. Therefore, SpamProbe is very good about avoiding false positives (good mail scored as spam). Again, it is pretty dumb at the start. So you do need to be sure all of the good mail is out of the 'your_spam' box and retrained as good. After around 2000 emails with your input, it gets pretty darned smart about your particular tastes.

The Training Mailboxes
First, do be aware that mail placed in the training boxes will be deleted automatically!

The training boxes all reside under the 'Corrections' folder of your IMAP account.

To train SpamProbe, lets first deal with some spam mail. I don't guess you want to save this spam mail. Lets simply move any spam in your 'Inbox' into the 'is_spam' mailbox. In your mail client, you may simply drag and drop the spam there. Do note that this is actually sending the mail back out onto the internet to this mailbox. It could take a bit of time if the email is large, or if muliple emails are selected at one time. The SpamProbe script will then execute at some later time and add details from this mail to the database. At this time, all mail in the 'is_spam' mailbox is automatically deleted.

Next, lets deal with some good mail. It might be in either your 'Inbox' or at times in the 'your_spam' box. First we need to remember that mail placed in the training boxes is automatically deleted. Therefore, if you wish to keep a copy, you will need to copy it to the 'non_spam' box, not move it. If this mail is in the 'your_spam' box, move it to your 'Inbox' in your POP account. This will store it locally for you in case you need it later. If you plan to delete this good mail anyway, you can simply move it to the 'non_spam' box and it will be deleted for you. In Thunderbird, you can 'right' click on a message header, select 'Copy To' and follow the fly outs to the 'non_spam' folder. You may also hold down the shift key to highlight more than one message at a time, reducing the number of 'Copy To's you make.

Now what is this 'not_scored' box? Why should I use it? Let's say a friend or family member sends you a joke which contains spammy words. A particular "enhancement drug" which I likely shouldn't name here, comes to mind. We do not want this email to degrade the system. That person's 'Good' email address and header information is in conflict with the spammy terms. SpamProbe argues over this email, Good vs. Spam. This type of mail can be placed in the 'not_scored' box so that it is removed from the database. This keeps the senders address from being degraded and keeps your term, in this case "enhancement drug" from becoming a weaker spammy term. If nothing is done with these emails, a person sending you a lot of jokes might wind up scored as spam (if they aren't right from the start). Then again, maybe you would want these jokes placed in the 'your_spam' box? Do remember, email in this box is also automatically deleted.

The your_spam Mailbox
Most of our SpamProbe users simply give up on finding good mail within the 'your_spam' box after the intial training period. We have created a script which automatically deletes mail from the 'your_spam' mailbox when it is 4 days old. During the training period, it is a good idea to review and delete each spam.... afterwards... well that is up to you. You should now have a full understanding of what happens to mail in these boxes.

Other General Information

Other Automated Maintenance
The SpamProbe database seems to average around 35 megs for most users. Some more active accounts will hit 70 to 100 megs. We run a script once per day which removes terms not seen for 30 days. Otherwise, the database would continue to grow. We've found this to be a good working solution, with very little degradation to scoring emails. So, if you do see a spam that seems to start up again later, this could be the reason. If that spam is sent constantly, it will be refreshed and not removed from the database.

How Good Is It?
In my opinion, nothing short of remarkable. I personally get about 3000 to 4000 messages a day (the joy of being a sys admin). For a period of six months, after my initial traing period of a few days, I did not find a single good email in my 'your_spam' box. At the same time, SpamProbe was catching often times in the 99.6% rate all spam. That amounts to 10 to 20 out of 3000 spams per day made it through. For most users, this will be more on the order of 2 to 4 per day. Many of these are 'new spams' which simply needs to be learned by SpamProbe.

EW3D Home | SpamProbe Setup Instructions | Mozilla Home Page