Mail List Archiving with SharePoint 2010

Recently exchange team has requested SharePoint team to evaluate mail list archiving with SharePoint strategy. The current mail list could include internal employee’s email, group email, and external emails such as gmais. If email has been send to the mail list, anyone included in this email list will be able to receive email notification. Emails are archived in exchange public folders you could browse. This blog covers the necessary design how to implement this archiving in SharePoint with answers to questions from the teams.

1. Current Mail List Archiving Process

Today when a mail list is archived one of the two below methods are used:

• MailAR – This is the original archiving solution that is hosted on a UNIX host.  There is a single account that is a member of many lists that are flagged for archiving and any time a message is sent to the list a copy is sent to MailAR where it is deposited into an archive.  This archive is accessible via a web client, the content is kept forever, and it has specific access controls so that only members of a list can see mail in the archive unless it is a public list.

• Exchange Public Folders – This was our first stab at trying to move away from MailAR because it was not reliable.  Similar to MailAR there is an account that is a member of each list that is to be archived and whenever an email is sent to the mail list a copy goes to the Exchange public folder.  The public folder is accessible through Outlook as well as through MyMail. See the picture below.

2. Mail List Archiving Requirements
Exchange public folders really aren’t working out like we would like and is missing a lot of important features which we really require.  Exchange team has been looking around for a suitable replacement, such as Jive, but hasn’t found any except for SharePoint.  The current requirements exchange team provided are as follows:

• Easily accessible from any popular browser (IE, Firefox, Safari)
• Content indexed and searchable
• Content completely discoverable for legal purposes
• Storage equivalent to Qualcomm mailboxes (5GB) and extendable when required.  Aging out data is a possibility and open for discussion.
• Message size limits in line with the messaging system (50MB per e-mail)
• Lock down archived data so that only current members of the list can see the archive contents (read only) unless it is a public mail list.
• Archived data is READ ONLY to everyone with the exception of IT staff such as MSG or SharePoint admins in cases where ITSEC requires a message to be removed.
• There shouldn’t be anything displayed on the landing page but the archives themselves (and only the archives the visitor has rights to open should be visible).

 

3. SharePoint Design to Support Mail List Archiving

There are several limitations and risks if we implement SharePoint to support mail list archiving. This includes but not limited to the following items.

• Email size – Current exchange support 50M and SharePoint default is 10M. If we increase the default size, it will impact SharePoint performance and storage requirement.
• Numbers of emails per folder – Current exchange support ‘unlimited’ number of emails. However, SharePoint has the list throttling setting to display only 2000 emails even millions of the emails have been stored. If we change the list throttling setting, it will impact performance.
• Total size of the emails per folder – Current exchange will allow 5G as the default. Exchange team provides the average growth rate for normal public folder is about 500M in two years. SharePoint may need to increase storage to support the growth.
• Total size of the ALL email folder - Current exchange has not provided the total number of public folders and we will need this input to design the hardware, storage, and possible storage archiving strategy.

 

4. SharePoint 2010 SP1 Capacity

Here is the SharePoint capacity limitation based on 2010 and SP1.

• Site collection 250,000 per Web application
• Content database 200G to 4T
• No limit for number of lists per site collection
• Items 30,000,000 per list
• List view threshold 5,000
• A new limit of 60 million items in any one SharePoint content database

5. SharePoint Architect Design

In order to support the mail list archiving with SharePoint requirements and minimize the maintenance cost, we would like to propose the following architect. Assumption is exchange email list will handle the external email and forward email to SharePoint.

• Build a separate web application that has separate site collection quota, list throttling, and content databases.
• Reuse current services including search index service to minimize maintenance cost. WE may need to add different services if the load increasing to a certain point that current services could not handle.
• Create separate only one site collection in one content database to host all public folders.
• Create multiple folders inside the same site collection to reflect current public folder.
• Create separate archive site collections to host aged public folders
• Apply policy or governance to archive emails from primary site collection to backup site collection

6. SharePoint Design for Mail Archiving Requirement

Here are answers to the requirement.

• Easily accessible from any popular browser (IE, Firefox, Safari) – OoB functions 

         SharePoint support these three major browsers. However, some functions may not work exactly same in Firefox and Safari comparing to IE.

• Content indexed and searchable – OoB configurations         

          The design will provide a search index services so content will be searchable. The result will be  based on user permissions.

• Content completely discoverable for legal purposes – Need process change and testing         Current search is through exchange. SharePoint support search but we may need to modify the process to feed search result to existing tools.

• Storage equivalent to Qualcomm mailboxes (5GB) and extendable when required.  Aging out data is a possibility and open for discussion. – Capacity planning

• SharePoint could support 5GB for one folder
• One site collection w/ 200G could hold 40 public folders
• Aging process can be done to purge to different site collections
• Current load is 500M in 2 years for one folder and the site could use for 10 years

• Message size limits in line with the messaging system (50MB per e-mail)
• As discussed to restrict to 10M to start
• Customization to handle large than 10M
• Lock down archived data so that only current members of the list can see the archive contents (read only) unless it is a public mail list.
• Configure through permissions
• May require some customizations
• Archived data is READ ONLY to everyone with the exception of IT staff such as MSG or SharePoint admins in cases where ITSEC requires a message to be removed.
• There shouldn’t be anything displayed on the landing page but the archives themselves (and only the archives the visitor has rights to open should be visible).
• Configure through permissions
• May require some customizations

7. SharePoint Implementation Task Summary

Based on the analyst and design described above, we would need to apply some customizations to SharePoint in addition to configurations. We may need to add servers to support the load based on performance testing result. 

Here are some the high level customization and configuration needed.

• Mail List Webapp setup and configuration with services
• Primary site collection setup and configuration
• Primary site collection home page customization w/ search options
• Public folder auto provisioning and approval process development
• Public folder and email capacity monitoring process development
• Archiving emails and move to backup site collection policy or workflow development
• Permission related configuration and customizations
• E-Discovery process testing
• Performance and load testing with system adjustment
• Potential two additional server configurations (one WEF and one application servers)

If you have done this on SharePoint, please share your experience.

Emailing Documents as Attachments Workflow Activity for Sharepoint 2010

One of the most obvious use cases which is lacking from SharePoint is the ability to email documents as attachments directly from SharePoint, instead of just a hyperlink pointing to the selected document.

Currently, our SharePoint users requested to implement sending documents to external partners or suppliers through workflow approval. In order to enforce the security policies and protect IP, the documents will be converted to PDF with designed watermark using third party tool. The documents will be submitted to workflow approval process and the actions will be recorded for auditing purpose.

Now, the question is how we could incorporate the email documents as attachments into the workflow. The best option is to develop a workflow activity for reuse. Before jumping directly to the development, we found some useful commercial third party tools and free CodePlex functions we could use. 

Here is the summary for you to select if you do not want to develop yourself.

1. HarePointWorkflow Extension for SharePoint 2010 - provides over 180 new 'ready-to-use' activities for SharePoint workflow development. The product is fully supported by all Microsoft workflow development tools and there is no need to migrate and adapt current workflows to a new development environment, or to master and deploy new development tools. You can benefit from the power of the product in SharePoint Designer 2010, Microsoft Visual Studio 2010, and in Microsoft Visio 2010.

Here is the summary of the HarePoint workflow activity for email. 

2. Virto SharePoint 2010 Workflow Activities Kit – no-code set of SharePoint activities for creating custom Workflow using SharePoint Workflow Designer. Virto Workflow Kit that can be used as an add-on for MS SharePoint 2010 includes extended activities for sending e-mails and SMS as well as messaging to Twitter, Jabber and Miranda. 

3. CodePlex SP Designer Activity - Send Email withAttachment - The new version adds support for Folders and Document Sets. If a user selects a folder or document set to run the workflow on this activity will get all items from the folder / document set and add them as attachments to the email.

The advantages of commercial third party tools are:

o  No-code and Ready-to-use Activities

o  Easy Deployment (just add them into your SharePoint Designer)

o  Enhanced Capabilities for your Custom Workflow Creation

o  Vendor support

The advantages of CodePlex tools are:

o  Free

o  Easy to modify since it comes with source code

o  User community support

Sometime, we found third party or CodePlex tools are working well on single SharePoint server. Even some big vendor products are not working consistently on three tier architecture with separate web front servers and application servers. As a result, please carefully evaluate the third party tools.

If you need to add email attachments directly to the list, you could check the following tools

• Emailing Documents as attachments in SharePoint 2010from CodePlex
• Sample code to email attachments by John

 As always, you should implement email validation. The one in CodePlex should be very helpful.

We are in the process to select and implement this function and will share our experience when it’s done. If you have any experience on any of these tools, please let me know your input.

How to resolve issue to open Document Library as Compressed (zipped) Folder Allitems.zip?

We have users reported the following issue in production last night. Users try to open one Document library http://servername/sites/department/subsite/Products  and get the following dialog with the message.

Do you want to open or save this file?

• Name: Allitems.zip
• Type: Compressed (zipped) Folder
• From: SharePoint Webapp name 

If you click "open", you're redirected to a blank page in SharePoint and users could not access the documents inside the library.

When I open the AllItem.aspx through SharePoint designer, it display the unreadable content. This indicates that this file has been corrupted. 

There are several ways you could fix it quickly. Here are two simple ways for your references.

1.  Get backup for the site and copy the AllItem.aspx to the existing site. You may require DBA to get the backup.
2. Create a new view for the library and set as the default view. Delete the previous default view AllItem.aspx. Rename the new view as AllItems.aspx.

Option one is very straight forward and no future explanation. I'll explain the option #2 here in details.

Since you are not able to access the library, the URL to the library is the default view for the library. You could access the list edit view through the following URL

http://servername/sites/department/subsite/_layouts/listedit.aspx?List=0e24c314-0848-4241-a12e-a72367a142df

 

 

Please note the last parameter is the list GUID that you could get easily through SharePoint Designer when you click the list. See the following screen shot.

 

 

 

Now, you could create the new view and set as default. You should be access the library now!

In order to keep the original URL to access the library, you could delete the previous default view AllItems.aspx. Then rename the new view to previous AllItem.aspx.


Now users could access the library using the same URL now http://servername/sites/department/subsite/Products.