SPAM - My Solution

After spending countless hours combatting spam and all fun that goes with it, I finally found a solution that’s not only good for me, but good for the end users as well.

Now, keep in mind we’re a fairly small shop - only about 60 users - but just as with everyone else, I found around 40-50% of all e-mail was plain crap, and another 10% wasn’t really needed.

I tried tagging for a while using the blacklists in R6. That was nice - but then I needed to do something with it. I turned to Raymond Neeves’ Intercept to pull the mail into another database after it’d been tagged. In reality, most people could probably get by with this solution. However, it’s near impossible to do whitelists in this situation - and it’s not very flexable. It also smashed any attachments, which isn’t a good thing. So I started looking to some kind of agent solution.

Lotus, you really need to include some of these agents in your next release - I’ve compiled about 3 or 4 different peoples’ code into what I now use. Here’s the jist of it, and if anyone wants to see the final results, I’m happy to share the code I’m using as well.

Step one - tag EVERYTHING. Use just about any available DNS Blacklist site you can trust.

Step two - run a before delivery agent on each person’s mail file to check for the tag.

Give the user the options that best suit your company - most of the scripts I found gave out “Ignore, Move or Delete” as possible options. I don’t want people to be able to ignore, so I pulled that out. The only options available to my users are Move (to a folder called “Spam”) or have the mail deverted to a ‘holding tank’. The holding tank holds the e-mails for 2 weeks (figure about 4-5MB of crap per user per week) then automatically purges them. Also give users control of their own whitelist - this is the most important feature! Once you’ve got folks updating their whitelists with the addresses (or domains) they want to come in, most of your work takes care of itself.

And third - nightly the server runs another agent that looks in the spam folder on each users’ mail file for any mail that has been tagged and is older than 2 weeks old. I figure if they can’t move it out of the folder within two weeks, it must not be all that important. Most people will forget to check/clean out that folder, and you’re right back where you started from.

With this solution in place, I think I get one or two e-mails a week that slip thru… and I used to get 50 in my in-box nightly! I’m a happy camper. It wasn’t an easy solution to figure out (I’m not a Lotus Script guru, but I can read code) but it is a FREE solution, which is very important for me - and my company, who is not-for-profit.

This actually ends up being 3 different agents in each mail file (yes, you have to either make a new template or modify each persons’ mail file) but it works, and everyone ends up being happy.

Subject: Thanks for sharing this …

We"re also a small non-profit organisation, and SPAM is becoming a real problem. the DNS-blacklists alone are blocking about 60-70% of the spam. As I was looking for better solutions, I read your post with much interest.

Joris

Subject: SPAM - My Solution…

Hi,

sounds cool. can you please send me the code or put in here in the forum ?

thx.

Horst

Subject: SPAM - My Solution… code/agents…

Ok… here’s what I use.

First off, check out a few articles that really helped me out in this process. The first one is http://www.dominopower.com/issues/issue200305/00001039001.html - it talks about the basic modifications to the mail template you need to do. I use their modifications to the Calendar Profile with a few modifications (like removing the “Ignore” option).

Once you have the changes made to the Calendar Profile (FYI, that allows you to set your anti-spam settings from the Tools/Prefs section, but you should be able to figure that out from the article!), you need to add the agents. They have some agents in that article as well, but I’ve tweaked them a bit to better suit my needs.

The first agent is the one that looks for the tag and acts before new mail arrives. I changed the cases as well - in the artilce they reference using “M”, “I” and “D” - I killed the Ignore, and changed “M” to “Move” and “D” to “Remove” since I’m not DELETING the messages - I’m moving them to another database for keeping for two more weeks.

Here’s that agent:

Sub Initialize

Dim Session As NotesSession

Dim DB As NotesDatabase

Dim archiveDb As NotesDatabase 

Dim doc As notesdocument

Dim docProfile As NotesDocument

Dim strAction As String

Dim strFolder As String

Set Session = New NotesSession

Set doc = Session.DocumentContext

If doc.HasItem("$DNSBLSite") Then

REM Current Mail is marked as Spam

	Set DB = Session.CurrentDatabase

REM Get Mail Preferences with Spam Settings

	Set docProfile = db.GetProfileDocument("CalendarProfile")

	If Not docProfile Is Nothing Then

REM Check whether sender is listed in personal Whitelist

		If Not IsInWhiteList(db, doc, docProfile) Then

REM Get selected Action

			strAction = ""

			If docProfile.HasItem("SpamOptionsTX") Then

				strAction = docProfile.SpamOptionsTX(0)

			End If

			Select Case strAction

			Case "Move" 'Move to a folder

				strFolder = "Spam"

REM Get Folder name

				If docProfile.HasItem("SpamFolderTX") Then

					strFolder = docProfile.SpamFolderTX(0)

					If strFolder = "" Then

						strFolder = "Spam"

					End If

				End If

REM Move document to folder

				Call doc.PutInFolder(strFolder,True)

				Call doc.RemoveFromFolder( "($Inbox)" )

			Case "Remove"

				Set archiveDb = New NotesDatabase(db.Server,"SpamMail.nsf")

				If Not archiveDb.IsOpen Then

					Call archiveDb.Create( "", "", True )

				End If

				Call doc.CopyToDatabase(archiveDb)

				Call doc.RemovePermanently(True)

			End Select

		End If

	End If

End If

End Sub

You also need the “IsInWhitelist” function to check against the whitelist:

Function IsInWhiteList(DB As NotesDatabase, Doc As NotesDocument, docProfile As NotesDocument) As Integer

IsInWhiteList = False

Dim docWhiteList As NotesDocument

Dim strFrom As String

Dim strTemp As String

If doc.HasItem("SMTPOriginator") Then

	strFrom = Ucase(doc.SMTPOriginator(0))

Else

REM This is not an SMTP Message

	Exit Function

End If

If strFrom = "" Then

	If doc.HasItem("From") Then

		strFrom = Ucase(doc.From(0))

	Else

REM Unable to determine the sender

		Exit Function

	End If

End If

Forall strAllowed In docProfile.SpamWhiteListTX

	If strAllowed <> "" Then

REM WhiteList may contain complete mail addresses like ‘John@domain.com’ …

		If Instr(strAllowed, "@") > 0 Then

			If Ucase(strAllowed) = strFrom Then

				IsInWhiteList = True

				Exit Forall

			End If

		Else

REM … or just domain names like ‘domain.com

			strTemp = Mid$(strFrom, Instr(strFrom, "@")+1)

			If Ucase(strTemp) = Ucase(strAllowed) Then

				IsInWhiteList = True

				Exit Forall

			End If

		End If

	End If

End Forall

End Function

You might notice that I’ve got a Mail Box on my server called “SpamMail.nsf” - this is my holding tank, which autmatically kills messages after 14 days via the replication settings.

Now, I didn’t like the way the Whitelist worked either… it was nice and all, but it was rather confusing to use for most people. So I took some code from another issue of DominoPower that had another agent to add folks to a Whitelist automatically - MUCH more convient. I still added the Whitelist tab in the Calendar Profile tab, however, so people could see who they have in their Whitelist.

http://www.dominopower.com/issues/issue200305/00001044001.html

The agent I use lookslike this…

[Declarations]

Const SpamWhiteList =“SpamWhiteListTX”

Const DNSBLSite = “X_RBL”

[Initialize]

Sub Initialize

Dim Session As notessession

Dim DB As NotesDatabase

Dim coll As NotesDocumentCollection

Dim UIWs As NotesUIWorkspace

Dim UIDoc As NotesUIDocument

Dim doc As NotesDocument

Dim docProfile As NotesDocument

Dim intChoice As Integer

Dim strFrom As String

Dim strDomain As String

Dim item As NotesItem

Dim boolFound As Integer

Set Session = New NotesSession

Set DB = Session.CurrentDatabase

Set docProfile = db.GetProfileDocument("CalendarProfile")

If docProfile Is Nothing Then

	Messagebox "There is currently no profile for your mailfile available. Please click 'Tools' 'Preferences' "+_

	"to create a new profile.", 16, "Unable to find profile"

	Exit Sub

End If

If docProfile.HasItem(SpamWhiteList) Then

	Set item = docProfile.GetFirstItem(SpamWhitelist)

Else

	Set item = New NotesItem(docProfile, SpamWhiteList, "")

End If

REM Agent may be called from view or UIDocument

Set coll = DB.UnprocessedDocuments

If coll.Count = 0 Then

	Set UIWs = New NotesUIWorkspace

	Set UIDoc = UIWs.CurrentDocument

	If Not UIDoc Is Nothing Then

		Set doc = UIDoc.Document

		Set coll = Nothing

		Call coll.AddDocument(doc)

	End If

End If

If coll.Count = 0 Then

	Messagebox "Please select the sender you want to add to your Spam Whitelist.", 64, "No documents selected"

	Exit Sub

End If

Set doc = coll.GetFirstDocument

Do While Not doc Is Nothing

	If Not doc.HasItem(DNSBLSite) Then

		Messagebox "The selected document is not flagged as Spam Message. Therefor it's not necessary to "+_

		"add the sender to your Spam Whitelist.", 64, "No need to update Whitelist"

	Else

		If doc.HasItem("SMTPOriginator") Then

			strFrom = doc.SMTPOriginator(0)

		Else

			strFrom = ""

		End If

		If strFrom = "" Then

			If doc.HasItem("From") Then

				strFrom = doc.From(0)

			End If

		End If

		If strFrom = "" Then

			Messagebox "The sender of the mail could not be identified with fields 'SMTPOriginator' and 'From'." +_

			"Therefor it's not possible to add the sender to your Whitelist.", 16, "Sender not identified"

		Else

			strdomain = Mid$(strFrom, Instr(strFrom, "@")+1)

			If strDomain = strFrom Then

				intChoice = 7

			Else

				intChoice = Messagebox ("Instead of adding the single email address '"+strFrom+"' you may also add the "+_

				"whole domain '"+strDomain+"'. This is quite convenient if the domain belongs to a business partner and "+_

				"you often receive mails from different users within this domain '"+strDomain+"'."+Chr(10)+Chr(10)+_

				"Click 'Yes' to add the domain to your whitelist."+Chr(10)+_

				"Click 'No' to add the single email address to your whitelist."+Chr(10)+_

				"Click 'Cancel' for no update.", 291, "Update Spam Whitelist")

				Select Case intChoice

				Case 6

REM Add Domain

					strFrom = strDomain

				Case 7

REM Add Single Address

				Case Else

REM Cancel

					Exit Sub

				End Select

			End If

			boolFound = False

			Forall strAllowed In item.Values

				Select Case True

				Case Ucase(strAllowed) = Ucase(strFrom)

					boolFound = True

					Messagebox "Sender '"+strFrom+"' is already listed in your Whitelist. ", 64, "No need to update Whitelist"

					Exit Forall

				Case Ucase(strAllowed) = Ucase(strDomain)

					boolFound = True

					Messagebox "Your Whitelist already contains an entry for Domain '"+strDomain+"'. As the sender '"+_

					strFrom+"' also belongs to this domain it is not necessary to add this single emailaddress.", 64, _

					"No need to update Whitelist"

					Exit Forall

				End Select

			End Forall

			If Not boolFound Then

				Call Item.AppendToTextList(strFrom)

				If docProfile.Save(False, False) Then

					Messagebox "Address '"+strFrom+"' has been added to your Whitelist.", 64, "Whitelist updated"

				Else

					Messagebox "An error appeared while saving your updated whitelist. Please try again to update your "+_

					"whitelist in some minutes. If the problem still appears please contact your Support.", 16, _

					"Whitelist could not be updated!"

				End If

			End If

		End If

	End If

	Set doc = coll.GetNextDocument(doc)

Loop

Print "Agent finished"

End Sub

So that takes care of most of it. But this will STILL leave untold thousands of e-mails in people’s in-boxes if you don’t do something about the Spam folder. So nightly I run the following agent (it runs on all db’s in the mail\ dir on the server)…

[Declarations]

Dim db As NotesDatabase

Dim session As notessession

Dim doc As NotesDocument

Dim docnext As NotesDocument

Dim folder As NotesView

Dim docdate As NotesDateTime

Const daysold = -14

[Initialize]

Sub Initialize

Dim RemoveDate As New NotesDateTime("")

Dim dbdir As New NotesDbDirectory("")



Set directory = session.GetDbDirectory( "FMPANOTES1/FMPA" )

Set db = dbdir.GetFirstDatabase(DATABASE)



While Not(db Is Nothing)

	If Left(db.filepath,5)="mail\" Then

		Call db.Open("","")

		Set folder = db.GetView( "Spam" )

		Set doc = folder.Getfirstdocument

		Set docdate=New NotesDateTime("")

		Call RemoveDate.SetNow

		Call RemoveDate.AdjustDay(daysold)

		Do While Not(doc Is Nothing)				

			docdate.LSLocalTime=doc.Created

			Set docnext = folder.GetNextDocument(doc)				

			If doc.HasItem("$DNSBLSite") And (RemoveDate.TimeDifference(docdate)>0) Then

				Call doc.remove(True)

			End If

			Set doc=docnext

		Loop

	End If

	Set db= dbdir.GetNextDatabase

Wend		

End Sub

Note that you could also take the code from the before mail arrives agent and actually move the ‘expired’ e-mail to another database if you wanted… but I figure 2 weeks aught to be enough for anyone. Also, I check for both the spam tag AND message date, but you don’t have to check for the spam tag… if it’s in that folder, they must not want it, right?

I’m happy to answer questions as best I can… as I said, I’m not a coder, but I ended up taking a bunch of different code and making these agents do what I needed for my situation.

I hope this helps you as much as it did me!

Subject: SPAM - My Solution…

Thank you for sharing you experience.Be a hero…, share your code via the Sandbox.

This is my favorite place to find out new useful code. Your code will be appreciated.

Alexandre

Subject: RE: SPAM - My Solution…

I’ve never done that before. Actually, I’ve never had much luck finding what I need in the sandbox, so conseqently I don’t use it that much…

I’ll see what I can do to submit it.

Subject: SPAM - My Solution…

Hi,

Thought I should add our stratergy (researched and planned - to be implemented). I know of other sites that have already implemented with great success.

*Have an SMTP gateway/firewall (Postfix on Linux) which scans and routes only;

*Use ClamAV for viurs scanning http://www.clamav.net

*Use SpamAssassin for spam filtering http://www.spamassassin.org

*Use MailScanner to bind Postfix with ClamAv and SpamAssassin http://www.sng.ecs.soton.ac.uk/mailscanner

There are plenty of documents and examples on these sites - enjoy :o)

Subject: SPAM - My Solution…

I don’t like the idea of using Agents (before mail arrives). That can slow down large mail servers.

Here’s our solution.

All of our inbound Internet mail is first delivered to a Linux box. It’s running your standard Linux mail software (not Domino). This Linux box performs the following tasks before forwarding the inbound Internet mail to our Domino mail server.

  1. Bounce/Forward/Delete messages for ex-employees.

  2. Scan the mail for virus using macafee pro for linux. If found, send an fyi to the recipient, store the infected files for review by an administrator.

  3. Scan the mail for Spam using SpamAssassin (www.spamassassin.org). This software (still on the Linux box) looks for keywords or phrases and assigns the email a spam score. If the email scores high enough, then it’s most likely spam. The subject of the email becomes SPAM .

  4. Forward the new message on to the Domino mail server.

Each user’s mailbox has a MailRule. The rule says, if the subject contains SPAM then move the message to the folder “Junk Mail”. Any message is delivered directly to the “Junk Mail” folder instead of the Inbox. User’s can then sort thru their potential spam when they have time.

Subject: Is a rule better than a ‘before arrives’ agent??

I agree that there are scalability issues with running an agent on every message that comes in, but doesn’t a rule do pretty much the same thing??

-chris

Subject: RE: Is a rule better than a ‘before arrives’ agent??

The rule is evaluated by the mail router, and doesn’t require the creation of the Agent Manager context that a Before Mail agent does. Rules also store evaluative formula in a pre-tokenized state, so the router handles them quickly.

Subject: RE: Is a rule better than a ‘before arrives’ agent??

As for the amgr or the router executing the “before mail arrives” agent Julie the agent goddess specifically noted in one of her articles that this agent only is executed by the router. As for the the rules being better or worse I wish i could find something definitive on that. But I will take Julie’s word as definitive on the before agent.

Subject: SPAM - My Solution

For those of you that used the MailRule option, did you find a way to deploy that to all Notes Clients automatically???

Subject: RE: SPAM - My Solution

I don’t know of a way to create MailRules programmatically. (Maybe OpenNTF can build a LotusScript library (API) or something).

When you create a “move to folder” MailRule in a user’s mailbox, that specific folder’s unique ID is stored inside the MailRule. You can not therefore copy a MailRule from one user’s mailbox and paste it in to another mailbox. It will not work.

http://www-10.lotus.com/ldd/46dom.nsf/0/8cc83a66cc33d86985256bc3004d8792?OpenDocument

Subject: Agree

This solution may be optimal for small shops, but there are serious scalability questions as the infrastructure grows…

Subject: SPAM - My Solution…

For those of you finding this post, here’s my ‘update’ to using this same (but now even more enhanced, and proven!) code for 6.5:

http://www-10.lotus.com/ldd/nd6forum.nsf/55c38d716d632d9b8525689b005ba1c0/909652ac6b2ae08c85256db700774c69?OpenDocument