Mail List Archiving with SharePoint 2010

Recently exchange team has requested SharePoint team to evaluate mail list archiving with SharePoint strategy. The current mail list could include internal employee’s email, group email, and external emails such as gmais. If email has been send to the mail list, anyone included in this email list will be able to receive email notification. Emails are archived in exchange public folders you could browse. This blog covers the necessary design how to implement this archiving in SharePoint with answers to questions from the teams.

1. Current Mail List Archiving Process

Today when a mail list is archived one of the two below methods are used:

• MailAR – This is the original archiving solution that is hosted on a UNIX host.  There is a single account that is a member of many lists that are flagged for archiving and any time a message is sent to the list a copy is sent to MailAR where it is deposited into an archive.  This archive is accessible via a web client, the content is kept forever, and it has specific access controls so that only members of a list can see mail in the archive unless it is a public list.

• Exchange Public Folders – This was our first stab at trying to move away from MailAR because it was not reliable.  Similar to MailAR there is an account that is a member of each list that is to be archived and whenever an email is sent to the mail list a copy goes to the Exchange public folder.  The public folder is accessible through Outlook as well as through MyMail. See the picture below.

2. Mail List Archiving Requirements
Exchange public folders really aren’t working out like we would like and is missing a lot of important features which we really require.  Exchange team has been looking around for a suitable replacement, such as Jive, but hasn’t found any except for SharePoint.  The current requirements exchange team provided are as follows:

• Easily accessible from any popular browser (IE, Firefox, Safari)
• Content indexed and searchable
• Content completely discoverable for legal purposes
• Storage equivalent to Qualcomm mailboxes (5GB) and extendable when required.  Aging out data is a possibility and open for discussion.
• Message size limits in line with the messaging system (50MB per e-mail)
• Lock down archived data so that only current members of the list can see the archive contents (read only) unless it is a public mail list.
• Archived data is READ ONLY to everyone with the exception of IT staff such as MSG or SharePoint admins in cases where ITSEC requires a message to be removed.
• There shouldn’t be anything displayed on the landing page but the archives themselves (and only the archives the visitor has rights to open should be visible).

 

3. SharePoint Design to Support Mail List Archiving

There are several limitations and risks if we implement SharePoint to support mail list archiving. This includes but not limited to the following items.

• Email size – Current exchange support 50M and SharePoint default is 10M. If we increase the default size, it will impact SharePoint performance and storage requirement.
• Numbers of emails per folder – Current exchange support ‘unlimited’ number of emails. However, SharePoint has the list throttling setting to display only 2000 emails even millions of the emails have been stored. If we change the list throttling setting, it will impact performance.
• Total size of the emails per folder – Current exchange will allow 5G as the default. Exchange team provides the average growth rate for normal public folder is about 500M in two years. SharePoint may need to increase storage to support the growth.
• Total size of the ALL email folder - Current exchange has not provided the total number of public folders and we will need this input to design the hardware, storage, and possible storage archiving strategy.

 

4. SharePoint 2010 SP1 Capacity

Here is the SharePoint capacity limitation based on 2010 and SP1.

• Site collection 250,000 per Web application
• Content database 200G to 4T
• No limit for number of lists per site collection
• Items 30,000,000 per list
• List view threshold 5,000
• A new limit of 60 million items in any one SharePoint content database

5. SharePoint Architect Design

In order to support the mail list archiving with SharePoint requirements and minimize the maintenance cost, we would like to propose the following architect. Assumption is exchange email list will handle the external email and forward email to SharePoint.

• Build a separate web application that has separate site collection quota, list throttling, and content databases.
• Reuse current services including search index service to minimize maintenance cost. WE may need to add different services if the load increasing to a certain point that current services could not handle.
• Create separate only one site collection in one content database to host all public folders.
• Create multiple folders inside the same site collection to reflect current public folder.
• Create separate archive site collections to host aged public folders
• Apply policy or governance to archive emails from primary site collection to backup site collection

6. SharePoint Design for Mail Archiving Requirement

Here are answers to the requirement.

• Easily accessible from any popular browser (IE, Firefox, Safari) – OoB functions 

         SharePoint support these three major browsers. However, some functions may not work exactly same in Firefox and Safari comparing to IE.

• Content indexed and searchable – OoB configurations         

          The design will provide a search index services so content will be searchable. The result will be  based on user permissions.

• Content completely discoverable for legal purposes – Need process change and testing         Current search is through exchange. SharePoint support search but we may need to modify the process to feed search result to existing tools.

• Storage equivalent to Qualcomm mailboxes (5GB) and extendable when required.  Aging out data is a possibility and open for discussion. – Capacity planning

• SharePoint could support 5GB for one folder
• One site collection w/ 200G could hold 40 public folders
• Aging process can be done to purge to different site collections
• Current load is 500M in 2 years for one folder and the site could use for 10 years

• Message size limits in line with the messaging system (50MB per e-mail)
• As discussed to restrict to 10M to start
• Customization to handle large than 10M
• Lock down archived data so that only current members of the list can see the archive contents (read only) unless it is a public mail list.
• Configure through permissions
• May require some customizations
• Archived data is READ ONLY to everyone with the exception of IT staff such as MSG or SharePoint admins in cases where ITSEC requires a message to be removed.
• There shouldn’t be anything displayed on the landing page but the archives themselves (and only the archives the visitor has rights to open should be visible).
• Configure through permissions
• May require some customizations

7. SharePoint Implementation Task Summary

Based on the analyst and design described above, we would need to apply some customizations to SharePoint in addition to configurations. We may need to add servers to support the load based on performance testing result. 

Here are some the high level customization and configuration needed.

• Mail List Webapp setup and configuration with services
• Primary site collection setup and configuration
• Primary site collection home page customization w/ search options
• Public folder auto provisioning and approval process development
• Public folder and email capacity monitoring process development
• Archiving emails and move to backup site collection policy or workflow development
• Permission related configuration and customizations
• E-Discovery process testing
• Performance and load testing with system adjustment
• Potential two additional server configurations (one WEF and one application servers)

If you have done this on SharePoint, please share your experience.