Managed metadata terms are not visible on document library item edit property dialog after migrated from one site to another using powshell commands

As we mentioned before, the major issue of using managed metadata terms is that the managed metadata column values become un-editableand terms are not available after the content migration either fromdifferent farm or different webapp. We have some solution that will be published later to deal with most of the situation or use cases as list before. During the testing of the solution on use cases #5 and #6 as listed below, we found there is issue on the powershell backup/restore or export/import commands to migrate terms associated with SharePoint document library. The terms are not visible on document library item edit property dialog window!

• Migrate sub site(s) from one site collection to a different site collection with the same Managed Metadata Service (MMS)
• Migrate sub site(s) from one site collection to a different site collection with the different Managed Metadata Service (MMS)

In order to explain the details, we will create one sub site with one custom list to compare with document library. We will also use saved template to compare with export/import process.

Here are details steps to reproduce this issue and explain the issue.

• Create a team site http://sbx08:3797/sites/MSource and a sub team site http://sbx08:3797/sites/MSource/Sub1
• One sub1,  modify Shared Documents library by adding one column to use global term and another column to use local term as in the screen shot

• One sub1, add one custom list with one column to use global term and another column to use local term as in the screen shot

• One sub1, you will see you are able to see terms when edit items on both Shared Documents library and custom list

 

• Export the sub1 sub site (web)

        export-spweb http://sbx08:3797/sites/MSource/Sub1 -path c:\webpackage.cmp -CompressionSize 100000000 -verbose

• Create a destination site collection http://sbx08:3797/sites/MDestination and a sub team site http://sbx08:3797/sites/MDestination/SubD1
• Import the sub1 sub site (web) to destination site collection and see the screen shot for the sub site on destination

       import-spweb http://sbx08:3797/sites/MDestination/SubD1 -path c:\webpackage.cmp -force -verbose

• When you try to edit properties the item inside Shared Documents library, you will find the managed metadata columns disappeared and NOT visible!

This is same if you use backup/restore powershell commands. However, the managed metadata columns on list are visible! Of cause, the managed metadata are not linked correctly as we mentioned before that will be addressed in future blogs.

 

In order to provide stranger evidence on this powershell issue, we will use the following steps to show the expected behavior.

• Save the sub1 http://sbx08:3797/sites/MSource/Sub1 to a template WITH content named MTemp1.wsp
• Save MTemp1.wsp solution to local disk as MTemp1.wsp
• Upload MTemp1.wsp to destination site collection http://sbx08:3797/sites/MDestination solution gallery
• Create a sub site http://sbx08:3797/sites/MDestination/SubD2 using MTemp1.wsp template

   

• The interesting thing is the Shared Documents library created from the template has different behavior compare to the one migrated by powershell. The managed metadata columns on list are visible as shown in the screen shot!
  

It’s very clear at this point that export/import or backup/restore powershell commands did not handle managed metadata columns on Shared Documents library correctly! Of cause, after these managed metadata columns successfully migrated, we still need to relink back to the correct managed metadata term stores that will be published in future blogs.

Even more intresting is if we migrate sub site from DocAve and set append to the destination site, we got the same result as export/import. However, if we select to overwrite destination from DocAve, we got the managed metadata columns but NO data displayed for Shared Documents as the following screen shot. The managed metadata columns are migrated and linked correctly using DocAve to migrate sub sites with same MMS!

If you edit the items inside Shared Documents, you are able to see the columns that is better than export/import process! We are checking with DocAve to see whether we could migrate the managed metadata for Shared Documents library.

We have worked with Microsoft and it seems like this issue could not be reproduced from older versions such as June CU 2010 or SP1. But could be reproduced on Feb CU 2012 and June CU 2012.

Thanks for our SharePoint developer Anitha who is working closely with me to identify some issues and find ways to verify the results. She has provided a work around and we could share with you later.

Managed metadata issue – User profile properties could not editable on SharePoint 2010

We run into an issues that some user profile properties could not editable on ALL non production SharePoint 2010 environments while those are editable in production. The simptom is you will get the following error on user profile edit page for some of the fields. 

 

Some SharePoint 2010 users have encountered the similar issues but I have not found good solution. I've tested this issue for a while and was able to find the root cause and workaround. Here are the details how to reproduce the issue and the woraround. Please note this workaround has side effect and we are looking for final solution from Microsoft.

• Set up two managed metadata service in SharePoint 2010 environment and mutiple webapps 

• One managed metadata service is used for specific Legal webapp and the other one is used for all other webapps.
• Each webapp will have only one managed metadata service associated with it.
• Both  managed metadata service configured identically as bellow screen shot 

• Refresh production content databases into non-production environemnt (Please refer other blogs on the issues of the linked managed metadata cloumns. You could see all blogs under managed matadata service category)

Now, you will have the issues not be able to edit user profile some properties! This does not seem to impact any SharePoint 2010 environemnt using single managed metadata service.

I did some testing and identified one woraround. The workaround is to unselect the “This servcies application is the default storage location for Keywords” on one of the managed metadata service as shown in the following screen shot. If you change the configuration like this, you are able to edit ALL user profile properties! 

The side effect is the all keywords will be samed into the one global managed metadata service term store that vialates the original requirement to limit users from other webapp to see keywrods from different webapp using different MMS.

As a result, we will work with Microsoft to see whether we could have a good solution to fix this issue.

Migrate SharePoint 2010 content with managed metadata challenges - Part II migrate site collection from one webapp to another webapp with the same MMS

As we mentioned in previous post, it is very challenge to keep the integrity of the managed metadata column values when you migrate the content. Foe the ten use cases we mentioned in previous blog, some of them are much difficulty that others. In this blog I’ll provide solution to use cases #3 listed before to fix managed metadata column values become un-editable when we migrate site collection from one webapp to another webapp with the same MMS.

Let us set up the site collation so we could demonstrate the issues and solution.

1. First, we will create a site collection with URL http://sbx08/sites/MMDDHW/SitePages/Home.aspx

2. Second, we set up the global managed term group named “QCGlobal” with one Term Set named Set1 with two terms. Then set up site collection local Term Set named “LocalMMS” with two terms. You could view both global and local term store setting for this site collection from the following screen shot

3. Third, we create a list named “MMD1” with one managed column “LocalMMD” linked to local term store and another managed column named “GlobalMMD” linked to global term store

4. Finally, add a new item and link the localMMD and GlobalMMD to local and global terms “Test1” and “Global1”

At this time, you could edit the item and verify both terms are linked to global or local term stores. Both columns are editable.

 

Let's migrate this site collection from this webapp to a different one with same MMS. I'm using site collection backup and restore method to demonstrate this. Here are the steps.

•  Back up the existing site collection using powershell

             Backup-SPSite -Identity http://sbx08/sites/MMDDHW -Path C:\MMDDHW.bck

• Create a new site collection http://sbx08:20339/sites/MMDDHW in destination webapp using blank template
• Restore the existing collection destination webapp using powershell

            Restore-SPSite –Identity http://sbx08:20339/sites/MMDDHW  -Path C:\MMDDHW.bck  -Force

Now if you browse the site, you will see identical site collection comparing to source site collection.

However, when you try to edit the item with managed metadata column, the global column is editable but the local term is grayed out not editable. The reason global term is editable because we migrated the site collection to the webpp with same MMS. IF you migrate to a webapp with different MMS, the global term will also not editable as shown in the following screen shot.

 

The question is how to fix the local terms. I follow the same procedure and script published by SergeyZelenov even the script is not meant for this purpose. Here are the procedures to fix local terms.

• Open Powershell ISE
• Open RestoreSPSiteLocalTermSet.ps1 powesshell
• Run Selection (F8) (Do not click Run!)
• Enter the destination site collection URL http://sbx08:20339/sites/MMDDHW and click OK
• You will see the following message

 PS C:\Users\harrycx> Restore-SPSiteLocalTermSetAccess

Site collection's access to term store group 'Site Collection - sbx08-sites-MMDDHW' restored successfully

 

Now, the issue resolved and both local and global columns are editable again!!!

There are some other ways you could use to migrate site collections from one webapp to another webapp. Here is the list of some of the ways with my suggestions.

• Export and import (Do not use and prefer backup and restore as we discussed before)
• Content DB detach and attach (Use it when site collection is larger than 15G)
• Metalogix migration tools
• DocAve migration tool (Prefer over Metalogix)

Does not matter what method you are using to migrate site collection, as long as it has been migrated to the webapp with SAME MMS, you could following the procedure listed above to fix the local terms. We will discuss other use cases including site collection migrated to webapp with different MMS later.

Migrate SharePoint 2010 content with managed metadata challenges - Part I use cases and issues

As we mentioned before, the Managed Metadata Service application is one of the most important and interesting new service applications in SharePoint Server 2010.  However, it’s very frustrating for SharePoint users and administrations not able to use the global and local managed metadata columns when you migrate the site collection from one farm to different farm, migrate site collection from one webapp to different webapp with same or different Managed Metadata Service applications.  The major issue after the content migration is the managed metadata column values become un-editable and terms are not available that Microsoft has not resolved!

It will become even worse if you migrate sub site from one site collection to another site collection with same or different Managed Metadata Service (MMS) application. The reason is the managed metadata column might be implemented through list column, site column, and content type. The site column and content types normally are on site collection level and will not be migrated with sub-site!

The major reason for this issue is the GUID changed and links broken as many people reported. Before you read the following sessions, you might need to review the previous post regarding the managed metadata column relationship. There are many efforts to resolve this but all of them have some limitation. We have worked with consultants and Microsoft product special service groups for a long time to provide a final solution to meet the different content migration managed metadata repair tool. In this blog, I’ll go through the major issue and most of the use cases so we could go to details how to resolve each of them in the future blogs.

Since the content migration could be on several different scopes like site collections, sub site, list, and item level, there will be many combinations of the migration if we consider the destination might have same MMS or different MMS. Here are some major use cases in our company that we try to find solution.

1. Migrate site collection(s) from one farm to another farm with same MMS
2. Migrate site collection(s) from one farm to another farm with different MMS (This can be done to combine #1 and #4)
3. Migrate site collection(s) from one webapp to another webapp with the same MMS
4. Migrate site collection(s) from one webapp to another webapp with the different MMS
5. Migrate sub site(s) from one site collection to a different site collection with the same MMS
6. Migrate sub site(s) from one site collection to a different site collection with the different MMS
7. Migrate list(s) from one site collection to a different site collection with the same MMS
8. Migrate list(s) from one site collection to a different site collection with the different MMS
9. Migrate item(s) from a list to another list in a different site collection with same MMS
10. Migrate item(s) from a list to another list in a different site collection with different MMS

Since you may link managed metadata column through list column, site column, and content type, you will have to test each use case all three implementations. To illustrate the content migration with managed metadata issue, we start with use case #1 to migrate one site collection from one farm to another. This is very common user cases we refresh all production content to one non-production environment so we could test upgrade process. This example is using managed metadata column through list column.

First we set up source site collection with url http://sbx10:3333/sites/mmd1, global terms, and local terms like in the following screen shot.

Second, we create two lists to use global terms and local terms separately. One named “Local Terms Test” using local terms and another list named as “Gobal Terms Test” using global terms. See following screen shots for details.

When you try to edit the items using global terms and local terms, you are able to change the values as in the following screen shot.

 

For migration, we demonstrated the common way for content migration that is very close to the one posted here.

1.  Site collection migration

• Back up site collection content DB from source farm
• Restore to destination farm as http://sbx08:3333/sites/mmd1

 2.  Managed metadata service migration

• Back up managed metadata service database from source farm
• Restore managed metadata service database to destination farm
• Configure managed metadata service on destination farm to use backed up managed metadata service database

After all these work, you would expect the site collection on destination farm with all the content and managed terms migrated. When you look at the destination site collection and list, it seems like all managed metadata values for both local and global terms are migrated to http://sbx08:3333/sites/mmd1 as in the following screen shot. 

However, the first issue is only global terms are available but NOT the local terms through manager terms link as in the following screen shot.

The second issue is when you try to edit either local or global terms, all of those terms are greyed out and uneditable as show in the following screen shots. One is for local terms and another one is for global terms.

 

  

Now, the migrated managed metadata are not linked to the correct term stores! We have tried several ways to fix the issues with limited success. Some methods listed below could fix the global terms but none of them could fix both global and local terms. Here is the list of the methods we tried from different blogs.

1. Backup and restore approaches suggested by Wictor and PointBeyond
   
   
2. Taxonomy APIs C# code to export and import managed metadata by Chris Khaw
   
3. Export/Import-SPMetadataWebServicePartitionData to export term store by Tajeshwar Sing and Microsoft DSE
   
4. Manually link back terms
   

We have found the global term store issues could be resolved much easier than local term store. The site collection migration is easier than sub site. The migration between same MMS is easier than different MMS.  Since there is no simple solution to cover all the test cases we discussed, we worked with Microsoft service group on building several tools to link back the migrated managed local AND global terms. I'll cover the details to each different uase case in future blogs.

LINQ could not access SharePoint 2010 managed metadate list columns

LINQ is a very simple and powerful way of querying data in the .Net programming language as we mentioned before. It allows the .Net language to query data natively against strongly typed classes. Yes against class with objects through DataContext generated by SPMetal.exe tool. SharePoint will convert the LINQ to CAML during the runtime. As a result, LINQ is an easy way comparing to older CAML to work with list data. Here is example to use LINQ to SharePoint 2010 with tricks and limitations.

1. Create a list named “City” with three columns “City”, “State”, and MMD that is managed meta column on site http://sbx08/sites/Harry/.

2. Create several entries and file in MMD columns as in the screen shot

3. Use SPMetal.exe located in ..\14\BIN to generate DataContext so you could use the objects in LINQ

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\BIN>spmetal /web:http://sbx08/sites/Harry/ /namespace:HarrySite /code:HarrySite.cs /language:csharp /useremoteapieapi

spmetal /web:http://sbx08/sites/Harry/ /namespace:HarrySite /code:HarrySite.cs /language:csharp /useremoteapieapi

4. Follow the procedure we mentioned before to add DataContext class. In the DataContext class HarrySite.cs, you will find the attributes “City” and State” for City list. However, there is no MMD attribute generated for this managed meta column. See the following  SPMetal generated code for City list.

[Microsoft.SharePoint.Linq.ContentTypeAttribute(Name="Item", Id="0x01", List="City")]

       public partial class CityItem : Item {

             

              private string _state;

             

              private string _city;

             

              #region Extensibility Method Definitions

              partial void OnLoaded();

              partial void OnValidate();

              partial void OnCreated();

              #endregion

             

              public CityItem() {

                     this.OnCreated();

              }

             

              [Microsoft.SharePoint.Linq.ColumnAttribute(Name="State", Storage="_state", FieldType="Text")]

              public string State {

                     get {

                           return this._state;

                     }

                     set {

                           if ((value != this._state)) {

                                  this.OnPropertyChanging("State", this._state);

                                  this._state = value;

                                  this.OnPropertyChanged("State");

                           }

                     }

              }

             

              [Microsoft.SharePoint.Linq.ColumnAttribute(Name="City", Storage="_city", FieldType="Text")]

              public string City {

                     get {

                           return this._city;

                     }

                     set {

                           if ((value != this._city)) {

                                  this.OnPropertyChanging("City", this._city);

                                  this._city = value;

                                  this.OnPropertyChanged("City");

                           }

                     }

              }

       }

If you try to use the column MMD from DataContext, it will NOT find the reference. This seems to be a bug and it is similar to the issue REST web service could not retrieve data form managed metadata reported before.

We will work with Microsoft and keep you posted.

Could not get Managed metadata or lookup field values through SharePoint 2010 REST web service

SharePoint 2010 provides a new RESTful web service to integrate with SharePoint data. When we use it to integrate with list data, we are not able to get managed metadata fields. After researching through different scenarios, we have concluded that REST interface could not get any lookup fields including user and group fields, normal lookup fields, or managed metadata fields. Indeed user fields and managedmetadata fields are look up fields.

We have created a customized list named “SOAP” with four columns. Title as string, Sponsor as user, ManagedMetadata as local term store fields, and descriptions as string. We have two list items added as in the following screenshot.

Please Sponsor is the looup fields as indicated in the following screenshot.

When we try to access this SOAP list through REST web service, we are not getting either Sponsor or ManagedMetadata values. Instead we are getting the lookup index values. Here are two ways either from REST returned xml or SharePoint Designer Data Source. Here are details.

1. Verify from REST ATOM returned xml data. Here are the steps to reproduce this.

• Configure to display REST data in xml format. For IE version 8 go to Tools –> Internet Options –> Content tab –> Feeds and web slices Settings.  In this dialog, clear the Turn on feed reading view check box as indicated in Randy's post.
• Invoke REST to display result http://sbx08/sites/Harry/_vti_bin/listdata.svc/SOA

 You will see result in xml in the following screenshot. The list item 1 Sponsor displayed as index "1" and Managedmetadata also displayed as index.

2. Verify from SharePoint designer Data Source values. Here are steps.

• Open site http://sbx08/sites/Harry with SharePoint designer
• Add REST Web service to the Data Sources similar to add SOAP web service
• Add a data view web part to the page using the data source 

Please note you need to enter user name and password in order to make REST data source work. The correct for my site is show in the screenshot.

I have inserted the REST data source into one page and here is the result from REST data source as in the screenshot. You will see it does not return Sponsor or Managedmetadata values. They are lookup index instead.

The same list will return both Sponsor or Managedmetadata values through SOAP web service interface as in the following screen shot using designer.

There are several attempts from different people try to resolve this through API or LINQ. However, it would be extremely difficult for end users to utilize REST web service if we could not get back managed metadata or any lookup fields.

 If you have any suggestions, please let me know.