Duplicate E-mails

It’s probably one of the most annoying things; receiving the same e-mail multiple times. Actually I’m not talking about a single e-mail but basically all your e-mails and not just twice bot LOTS of times. This article discusses the most common configurations which lead to e-mail duplication and how to prevent them from happening again.

• Rules
• Multiple accounts
• Profile recreation
• Synchronization tools
• Virus scanner/firewall
• Tools to clean up the mess

Rules

Rules and more specific “Move to folder” rules is the most easiest one to get yourself some duplicates. This is because for each e-mail received Outlook will fire all configured rules against it. The following example will show this can lead to duplicates.

In this example there are two rules configured;

1. Apply this rule after the message arrives
   from Robert Sparnaaij
   move it to the Outlook folder
2. Apply this rule after the message arrives
   with Outlook in the subject
   move it to the Outlook folder

Now imagine that you receive an e-mail from me where I have the word Outlook in the subject; both rules with apply! So it will get moved two times creating a duplicate. The way to prevent this is to add the action “stop processing more rules” to the rule. If we would add that action to the first rule and the rule applies the action “stop processing more rules” will prevent that the second rule will be executed. Keep in mind that if you have even more rules configured that should always trigger -for instance that assigns messages to a category or marks it for follow up- you’ll need to sort your rules in the correct order as well.

Multiple Accounts

When you configure multiple accounts in Outlook make sure that these are indeed separate accounts and not just aliases on the same mailbox. In the case of an alias you would basically have the same account configured twice as all mails from either alias is being received in the same mailbox. When you also have the option configured to leave a copy on the server (POP3 account) both accounts will collect the same e-mail leaving you with duplicates. To read up on the difference between a separate mailbox and aliases click here.

To prevent duplicates you must configure the alias account not to receive e-mails since they are already being received by the original account;
Tools-> Options-> tab Mail Setup-> button Send/Receive-> button Edit…-> select the alias account-> uncheck Receive mail items

Profile Recreation

When you recreate your mail profile -for instance when the original got corrupted or when you reinstalled your PC and configure Outlook for the first time again- and you originally had Outlook configured to leave a copy on the server (POP3 account) it will collect all the e-mails from the server again even when you have received them in a previous configuration. This is because the newly created profile doesn’t “know” that these messages have been received before by another profile. So this means that when you recreate your mail profile and during setup you immediately configure it to use your old pst-file as the default delivery location it will create duplicates for the messages that are already received and that are still on-line as well. To prevent this from happening make sure start Outlook at least once with a clean pst-file before reconnecting the old one and setting it as the default delivery location. This way you’ll collect all the e-mails in a new pst-file.

For Outlook 2002/XP and 2003

1. Connect to your original pst-file by File-> Open-> Outlook Data File… and once opened move all the newly received e-mails to that pst-file.
2. Close Outlook and go to Control Panel-> Mail-> button Data Files…-> button E-mail Accounts-> button Next-> at the bottom you can select the original pst-file as the default delivery location and press Finish.
3. Press the button Data Files… and remove the other pst-file. If you want to keep your computer clean press the Settings button first and write down the location of the pst-file so you can physically delete the file as well and not just the connection to it.

For Outlook 2007

1. Connect to your original pst-file by File-> Open-> Outlook Data File… and once opened move all the newly received e-mails to that pst-file.
2. Close Outlook and go to Control Panel-> Mail-> E-mail Accounts-> tab Data Files. Select the the original pst-file and press Set as Default
3. Select the other pst-file and press Remove. If you want to keep your computer clean press the Settings button first and write down the location of the pst-file so you can physically delete the file as well and not just the connection to it.

Synchronization Tools

Synchronization Tools are great for instance to keep your contacts you have in Outlook and on your mobile device like PDA and mobile phone synchronized and up-to-date. Depending on the tool you are using synchronizing for the first time might be tricky and could cause some duplicates or “similar” items. For instance I used to store most of my mobile phone contacts by first name only and the ones in Outlook by their full name. So I had a contact named Edwin on my mobile phone and a contact named Edwin Sparnaaij (yep, trying to get my brother famous here ) in Outlook. When I would synchronize I would end up with an item named Edwin and an item named Edwin Sparnaaij in both Outlook and my mobile phone since the synch tool doesn’t know these are one and the same persons in real life.

To prevent this from happening make sure your contacts are in order before synchronizing for the first time or you’ll only duplicate the mess. Since my mobile phone was a big mess mainly because of the lack of a proper input device and contact storage properties on it (I had a Nokia 3210 at that time) getting things in order would be handier in Outlook. To make sure it doesn’t mix with your original Outlook contacts you can create a new mail profile first with a dummy account or no account configured at all (tip: call the profile Mobile Device so you can easily recognize it). Now you can safely synchronize with that mail profile and use Outlook to easily clean up your contacts on your mobile.

If you are a control freak (I won’t be last to admit this) and want to make sure that the initial synchronization indeed doesn’t create any duplicates move the cleaned up contacts from the Mobile Device mail profile to your original one.
Start Outlook with your original mail profile
Connect to the pst-file from the Mobile Device profile by File-> Open-> Outlook Data File…
Move the Contacts from the Mobile Device pst-file to your original Contacts folder

Now that you have created a “master copy” in Outlook make a back-up of your pst-file in case synchronization goes wrong after all. Also now that we have all contacts correctly in Outlook we don’t want it to still end up creating duplicates because the synchronization tool isn’t “intelligent” enough to recognize the similar items (this depends on the synchronization tool you are using so I’m going for a fail safe here). As we have a master copy in Outlook we can safely delete all the contacts on the mobile device. Now reconfigure your synchronization tool to use your original mail profile and synchronize; there is just NO WAY you can end up with duplicates now!

Virus scanner / Firewall

Another quite common evil which will end up creating duplicates are virus scanner that integrate with Outlook or too strictly configured firewall. To understand this I’ll explain a little bit how the e-mails are actually being received without a virus scanner or firewall;

When Outlook does a Send/Receive it will make a connection with the mail server and will ask the server which e-mails it is currently storing. The mail server will respond with a list and Outlook will verify with its local list if it has collected all the e-mails already. If so; Outlook will close the connection to the mail server. If not; Outlook will request the new e-mails from the server and when it has received all the e-mails it will close the connection with the mail server and update its own list of received e-mails.

The above illustrates a perfect Send/Receive cycle. If for some reason the connection gets terminated unexpectedly so that Outlook cannot close the connection to the mail server properly Outlook will consider it as a failed Send/Receive cycle and will not update its own list of received e-mails even when some or all of them have been received already. This means that at the next Send/Receive interval Outlook will check for new e-mails with an “outdated” list and collects the e-mails from the previous Send/Receive interval again and will not update its local list until the connection to the mail server closes properly.

This is where the virus scanner and firewall come into the picture. The exact method depends a bit on the virus scan solution chosen but basically when a virus scanner integrates with Outlook it will “intercept” the Send/Receive request from Outlook. So when Outlook does a Send/Receive and “thinks” it is talking to a mail server it is in fact talking to the virus scanner. This means that now the virus scanner is responsible for the connection. This means setting up the connection, requesting the list of mails stored on the server, managing the timeouts, etc… but also closing the connection properly and telling this to Outlook. Although this could go right of course many many times, keep in mind that you do have added a single point of failure to the Send/Receive process leaving you with another possibility that the connection to the mail server isn’t closed properly and already received e-mail being received again at the next Send/Receive interval.

Therefore it is another good reason to prevent your virus scan solution to integrate with Outlook. There is no direct need to integrate a virus scanner with Outlook as it is redundant anyway; it won’t add another level of security since it is one and the same tool as your on-access-scanner. Default Outlook security settings and an up-to-date virus scanner are enough to keep you safe! see the documentation of your virus scanner on how to disable its integration with Outlook.

Now that you know how the Send/Receive interval works you can imagine that a too strictly configured firewall could terminate the connection prematurely so Outlook will never have the chance to update its local list of received e-mails. In this case check disable your firewall and wait two Send/Receive intervals. If the second one doesn’t cause any duplicates you’ll know that your firewall isn’t properly configured. See the documentation of your firewall software on how to configure your firewall for e-mail retrieval with Outlook.

Tools to clean up the mess

If you are just swamped with duplicates you probably don’t want to clean up the mess manually. The ideal and easiest way to recover would be by restoring a back-up. If you don’t have a current back-up there is a wide range of duplicate remover tools to choose from.

SperrySoftware (discount code “BH93RF24″)
Duplicate Appointments Eliminator
Duplicate Contacts Eliminator
Duplicate Email Eliminator Across Folders
Duplicate Journals Eliminator
Duplicate Notes Eliminator
Duplicate Posts Eliminator
Duplicate Tasks Eliminator
Eliminator Bundle

MAPILab (discount code “4PM76A8″)
Duplicate Email Remover
Duplicates Remover