Options to block or exclude a security group to access some selected Sharepoint site collections - Part I

Our security department has identified some SharePoint 2010 site collections need to restrict to users of some security groups. The requirement is to restrict the users belong to some security groups to access selected site collections with sensitive information even these users have been granted the permission through individual account, any AD groups, or email list groups.

After extensive research and testing, we have found that there is no out of box solution on SharePoint 2010 we could exclude a security group to access some selected Sharepoint site collections. Here are some possible options I would recommend to try to resolve this issue.

1. Siteminder policy configuration to exclude any users belongs to selected groups to access the SharePoint site URL identified.

If you have already integrated the siteminder with SharePoint, you could modify the policy to restrict the groups to access site URLs. As long as you have the groups needed to be restricted and the site URLS, it should be a quick configuration changes.

Since you could provide the root URL and all sub sites will be blocked, the pros of this approach is clean and simple. The sites need to be controlled or protected can be in any webapp. Even users have been added to the SharePoint site, they will not be able to access the site. However, current version siteminder SSO does not support multiple domain AD SharePoint. We have more than FIVE AD domains in our company and it was NOT a solution at this time for us.

2. Configure ADFS Identity provider or other gateway such as Vordel to deny certain group of users to access selected URLs.

If you have implemented ADFS identity provider or other gateway tool such as Vordel to work with with SharePoint, you might work with the team to configure on ADFS or Vordel side to deny group of users to access certain selected sites. The pros and cons are same as described using siteminder.

Since we have identified many issues for SharePoint with ADFS, we have on hiod the SharePoint ADFS implementation early February 2011. ADFS approach is NOT a solution at this time for us. At same time, we are evaluate the Vordel approach and may move to this solution whenever it's available.

3. Configure IIS URLscan tool to block user agents for SharePoint sites

Microsoft strongly recommends that all users upgrade to Microsoft Internet Information Services (IIS) version 7.0 running on Microsoft Windows Server 2008. IIS 7.0 significantly increases Web infrastructure security.The URLScan tool can be used to protect your Web server from attacks and exploits. When a client requests a page from a Web server, it typically sends over some HTTP headers that contain additional information about the request. You mighht use the [DenyHeaders] section to restrict some user groups to access to certain SharePoint sites. You may refer to the discussion for more details.

Based on our observation, this approach may be useful but we may have difficult time to identify the method to filter the certain users. This may not the solution at this time for us.

4. Configure Microsoft Internet Security and Acceleration (ISA) to restrict access

In this session we will use ISA to accomplish this task to restrict the users belong to some security groups to access selected site collections. Here are the steps.

    * Create a security group and add users to that group
    * Create a new Firewall Policy and a Domain Name Set
    * Verify your Firewall rule and fine tune settings.

Please refer to Microsoft TechCenter on ISA for details.

Since this approach requires the additional ISA installation and lots of testing, it may not the solution at this time for us.

5. Configure user policy at web application level to deny them access to SharePoint sites inside the webapp.

If you do not have siteminder integrated with SharePoint, you may consider to group or migrate all sites need to be controlled or protected to a separate webapp. Then creating a user policy at web application level which denies them access to SharePoint. Here are the steps.

•  Click Manage Web Applications
•  Click the Web Application you wish to set a deny policy on
•  Click User Policy in the Ribbon bar to display the Policy for Web Application dialog
•  Click the Add Users link
•  Leave the Zones prompt as (All Zones) and click Next to display the Add Users dialog
•  Enter the AD group into the Users prompt
•  Click the Deny All check box
•  Click Finish

This is also very clean process and same as the previous approach the users will not be able to access the site even they have been added to the SharePoint site. This approach is extensible to add aaitional groups need to be blocked to the webapp. However,there are two major issues need to be planned for this approach.

• You may need to migrate sites identified to be protected to the new webapp. This will break the URLs or bookmarks published to other applications. You may implement redirect or  alternate access mappings to minimize the impact.
• You may change your site provisioning process so future sites need to be controlled will be provisioned inside controlled webapp.

This may be a good long term solution for us and we may plan to implement this approach.

6. Customize SharePoint access

You might intercept the out of box SharePoint authentication or authorization package to exclude the groups of users to access selected SahrePoint sites.  One suggestion provided by Allen Wang is to apply custom master page for this site collection and insert a custom user control near the top of the master page to perform security trimming, or check in the user control. If the user is in one of these group, then redirect it to another page or something....

You will see this might work for the users to access the site directly. However, it may not block users to access the files directly. This is not safe since the users could modify the master page or remove the code from the page.

7. Customize SharePoint Policy Feature

Since we could manage Information Management Policy in SharePoint Server 2010, I'm thinking whether we could implement a policy and apply the policy feature on the SharePoint site to exclude certain users. If anyone think this is feasible, please let me know.

As a result, none of the above options are perfect as we could see at this point. We are exploring other options in Part II and if you have any good suggestions, please let me know.

Thanks for your input.

Query String URL Filter web parts error - No item exists and it may have been deleted or renamed by another user

We have several users using Query String URL Filter web parts to organize the content on the page. They have complained that the Query String Filter web part does not consistently working and sometimes they have the following error.

No item exists at http://sbx01/sites/Harry/SiteAssets/Filter2.aspx?ID=10.  It may have been deleted or renamed by another user.

Here is the way to reproduce this error.

1. Create a list like task list and display ID column.
2. Create a web part page and add two web parts. One for task list and another Query String Filter web part.
3. Add Query String Parameter Name as ID to filter the list by ID
4. Edit Query String Filter web part connections to "Send Filter Value to" task list
5. Select "Consumer Field Name" as ID
6. Verify the result by access URL like  http://sbx01/sites/Harry/SiteAssets/Filter2.aspx?ID=1

Please note the page created named Filter2.aspx. The syntax to pass the query string is ?ID=x and x represent to the number of the ID column.

If you try to access the URL while passing different numbers, sometime, you will get error "No item exists and it may have been deleted or renamed by another user". Sometime, you are fine for the first several items as someone else described here and MSDN site.

We tested the web part page and wiki pages. Both of them have the similar behavior. However, it's always working if you add the Query String Filter web part to the default view of the list page. One example is the task list page we are able to get the result http://sbx01/sites/Harry/Lists/Tasks/AllItems.aspx?ID=10.

After some testing and we were able to find the solution that is to avoid to use SharePoint out of box column name such as ID in the "Consumer Field Name" for Query String Filter web part connection. We could use MyID instead of ID and the page is working fine. This seems like to be same as described in version 2007.

From what we learned here, we should avoid to use SharePoint out of box column name  in the "Consumer Field Name" for Query String Filter web part connection. I'm hoping this has been documented somewhere in SharePoint user guide to avoid so much confusion for the end users.

Check permission levels given to user is None for Claims Based Site if user is given permission through AD group

We are running into a critical issue on the SharePoint 2010 Extranet implementation that is working with Microsoft as critical bug.  Here are the details on the issues and the steps to reproduce. Please let me know if you have any solution or workaround.

Issue description: Check permission levels given to user is None for Claims Based Site before user login if user is given permission through AD group.

Procedure to reproduce:
1. Create a webapp and select "Claims Based Authentication" with everything else as default as described in below screen shot.

2. Create a site collection based on any template such as "Team Site" template as described in below screen shot.

3. Add a AD group like ems.sp.team to any groups such as Site Member Group.Please note one user with ID "harryc" is one of the member of the AD group.

4. Check Permissions on user in group (shows none). The correct result should display this user has Member Group permission. Click Site Actions->Site Permissions->Check Permission-> Enter user ID "harryc" as displayed in the following screen shot.

 
We have run Powershell command gpupdate /force and the result is the same.

5. Use "harryc" to login once and repeat the step #4 described above. The permission check result is correct now as Contribute   Given through the "Harry Members" group. 

Since we are implement SharePoint Extranet and we will need to hide all users on the site collectiuons except those inside the site collection, we would need this check permission function working in order to complete this function.

We are on SharePoint 2010 RTM release without any CU updates. Please let me know if you have any solution or workaround.

Yesterday, Microsoft has reproduced this issue and the trick part is you could ONLY reproduce this if AD groups are at Windows 2003 Domain Functional Level. See the screen shot for the version.

If you are using Windows 2008 Domain Functional Level, you will not have this issue. With Windows 2003 Domain Functional Level, we are able to reproduce this issue on SharePoint RTM, April CU, October CU, SP1 + June CU. We have tested AD on Window 2008 and 2008 R2 Window servers for both Universal and Global AD groups.

Since this has been submitted as bug for Microsoft, it may be resolve in the future releases. At meantime, you may consider to upgrade you  Windows 2003 Domain Functional Level to 2008 version if you need to resolve the issue or ask your end users to login to the site at least once. I was thinking to develop a script to login all users on the site automatically, but since it will mass up the auditing, I have on hold the though at this time.

 If you have any better idea, please let me know.

Feature with Id is not installed in this farm error when using Client Object Model to activate features

We have a situation that we have to enable a site collection feature from Client Object Model (OM) instead of server side API. We run into the error when feature is been activated. Here is the summary to reproduce the issue.

After setting up SharePoint 2010 Client Object Model Console Application with 64 bit platform and .Net 3.5,  we run into the error when feature is been activated.

Here is the Error Message: Feature with Id '859d22a7-0c7b-476e-afd4-420fd2955260' is not installed in this farm, and cannot be added to this scope.

E:\Harry\DEV\Student\ClientOM\ClientOM\bin\Debug>ClientOM.exe
Unhandled Exception: Microsoft.SharePoint.Client.ServerException: Feature with Id '859d22a7-0c7b-476e-afd4-420fd2955260' is not installed in this farm, and cannot be added to this scope.
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
   at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
   at ClientOM.Program.Main(String[] args) in E:\Harry\DEV\Student\ClientOM\ClientOM\Program.cs:line 28


Here is the source code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint.Client;


namespace ClientOM
{
    class Program
    {
        static void Main(string[] args)
        {

            // Get the client context
            using (ClientContext clientContext = new ClientContext("http://sbx01/sites/Harry/"))
            {
                // Load the features
                Site clientSite = clientContext.Site;
                FeatureCollection clientSiteFeatures = clientSite.Features;               

                Guid FeatureId = new Guid("859d22a7-0c7b-476e-afd4-420fd2955260");
                clientContext.Load(clientSiteFeatures);
                //clientContext.ExecuteQuery();
                // Activate the feature
                clientSiteFeatures.Add(FeatureId, false, FeatureDefinitionScope.Site);
                //clientSiteFeatures.Remove(FeatureId, false);
                clientContext.ExecuteQuery();
            }

        }
    }
}


We can activate the feature through the UI and Powershell on the same server running the code. In addition, We can use the same code to deactivate the feature by commenting out the add method and uncommenting the remove method. Someone has also raised similar issue here.

We also noticed another similar code but have not test it. Many people also reported the same issue here.

We are wondered how can a site or web feature be activated using the Client Object Model?



Please let me know if you have solution. Thanks.

401 UNAUTHORIZED error when integrate Perl from Window server to SharePoint 2010 site through SOAP web service

We have engineers using perl on Linux and Unix running testing scripts. They would like to send the testing result directly from perl to SharePoint. The perl to SharePoint integration is critical for our engineers on their daily jobs.

Since we need to pass authentication, we are trying to pass this on window server first before moving to Linux or Unix. We have identified two good blogs on this topics and tried the approach.  However, we are getting 401 UNAUTHORIZED error against both NTLM and Kerberos SharePoint 2010 sites. We believe the NTLM perl package may not configured correctly or Perl may not connect to SharePoint Kerberos sites. Since we are not perl or SharePoint integration expertise, we could like to get some help.


We have tried on 2008 64bit window VM server before moving to Linux or Unix. Here are the steps we tried.

1. Install ActivePerl 5.12.3
2. Install SOAP::Lite using cpan command cpan -i SOAP:Lite since it’s difficult using ppm
3. Setup SharePoint site and list and grant site collection admin to the test user account
4. Tried the script from The SharePoint / Perl Connection (Part 1)
5. Error The exception is on line: $lists = $soap->GetListCollection();

          401 Unauthorized at E:\Share\ws\ws.pl line 27
          Client-Warning: Unsupported authentication scheme 'ntlm'.

    6. I also tried to add the following line:
          use Authen::NTLM; ntlmv2(1);
          But got error
          E:\Share\ws>ws1.pl
          Can't locate Authen/NTLM.pm in @INC (@INC contains: C:/Perl64/site/lib C:/Perl64
/lib .) at E:\Share\ws\ws1.pl line 3.

It seems like the NTLM.pm is inside C:\Perl64\lib\LWP\Authen.

    7. I also tried to install LWP::Authen::NTLM from CPAN http://search.cpan.org/~gaas/libwww-perl/lib/LWP/Authen/Ntlm.pm. I’m not familiar the perl make installation.


Here is the script from John Wefler and I modified with our server and site.
================================================
print "Start\n";

use LWP::UserAgent;
use LWP::Debug;
use SOAP::Lite on_action => sub { "$_[0]$_[1]"; };
import SOAP::Data 'name', 'value';
sub SOAP::Transport::HTTP::Client::get_basic_credentials { return ('user' => 'password') };
our $sp_endpoint = 'http://sbx01/sites/Harry/_vti_bin/lists.asmx';
our $sp_domain = 'sbx01.na.qulacomm.com:80';
our $sp_username = 'adminaccount';
our $sp_password = 'password';

$debug = 1;
if ($debug) {
    LWP::Debug::level('+');
    SOAP::Lite->import(+trace => 'all');
}
my @ua_args = (keep_alive => 1);
my @credentials = ($sp_domain, "", $sp_username, $sp_password);
my $schema_ua = LWP::UserAgent->new(@ua_args);
$schema_ua->credentials(@credentials);
$soap = SOAP::Lite->proxy($sp_endpoint, @ua_args, credentials => \@credentials);
$soap->schema->useragent($schema_ua);
$soap->uri("http://schemas.microsoft.com/sharepoint/soap/");

$lists = $soap->GetListCollection();
quit(1, $lists->faultstring()) if defined $lists->fault();

sub lists_getid {
    my $title = shift;
    my @result = $lists->dataof('//GetListCollectionResult/Lists/List');
    foreach my $data (@result) {
        my $attr = $data->attr;
        return $attr->{ID}
        if ($attr->{Title} eq $title);
    }
    return undef;
}
sub lists_getitems
{
    my $listid = shift;
    my $in_listName = name('listName' => $listid);
    my $in_viewName = name('viewName' => '');
    my $in_rowLimit = name('rowLimit' => 99999);
    my $call = $soap->GetListItems($in_listName, $in_viewName, $in_rowLimit);
    quit(1, $call->faultstring()) if defined $call->fault();
    return $call->dataof('//GetListItemsResult/listitems/data/row');
}
my $list_id = lists_getid('Shared Documents');
print "List ID is: $list_id\n";
my @items = lists_getitems($list_id);
foreach my $data (@items) {
    my $attr = $data->attr;
    print Dumper($attr);
}

print "END\n";
=============================================




Here is the error I got:

 

I also listed some reference I got from web for your reference.

• How to Call a .NET-based Web Service Using the SOAP::Lite Perl Library
• Make a SOAP client with Perl and SOAP::Lite
• Test NTLM installation
• Accessing a sharepoint using perl and webdav

Since the final goal is to use perl on Unix and Linux, we found LWP does not have NTLM implemented on Unix/Linux through CPAN. However, both wget and curl  ( command line web clients ) support NTLM, and curl supports Kerberos. So we tried to use perl with wget to communicate with SharePoint.

Here is an example command from RHEL 5 command line that gets results back from the Lists web service:

wget --post-file /tmp/listrequest --header 'Content-Type: application/soap+xml; charset="utf-8"' --http-user='NA\mchiles' --http-password="putyourpwhere"  http://sharepoint.qualcomm.com/_vti_bin/Lists.asmx

note that I put the file for the request in /tmp/listrequest.

If you save this xml body to this file,  it should work
##


<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Body><GetListCollection xmlns="http://schemas.microsoft.com/sharepoint/soap/"/></soap:Body></soap:Envelope>

##

We are still try to get more SharePoint functions done through perl and please let me know if we make some progress.

Why Web Anlytical Reports "Custom Reports" always display empty reports?

As we discussed in previous blog, Web Analytics service as part of Microsoft® SharePoint® Server 2010, is set of features to help you collect, report, and analyze the usage and effectiveness of your SharePoint Server 2010 deployment. Web Analytics features include reporting, Web Analytics workflow, and Web Analytics Web Part.

As  SharePoint administrators, we could like to leverage the SharePoint 2010 Customized Web Analytic Reports to output weekly usage reports for management on the following metrics, total number of page views, average number of page views per day, total number of daily unique visitors and average number of unique visitors per day.  Some key benefits of the custom reports are the ability display visual charts and graphs in excel worksheets and the ability to aggregate report information into one consolidated report per web app and subsequently sent out to management.You could refer here for details on different reports.

One issue we found in our environment is Custom Web Analytic Reports are not showing data. One of the consultant is able to generate the same Custom Web Analytic Reports with correct in his environment. We have tried everything from permissions to excel services without any luck. He are the steps to reproduce this error in our environments.

1. Navigate to any site and select "Site Actions" -> "Site Setting" -> "Site Collection Web Analytics Reports" ->on the left navigation select "Top Browsers" ->"Analyze" tab. You will notice a valid report with data shown in the aspx page.

2. Navigate to the ribbon group "Analyze" and choose "Customize Report". If you do not have this button, please refer the blog to enable this report.


3. Name the report, "Harry.xlsx" note the document library and the report will be generated and displayed. You could navigate to the report under http://siteURL/AnalyticsReports/Forms/AllItems.aspx? and open in the browser. We found notice no data present in the report as shown in the below screen.

 

We are not able to find any errors on the server and here is the log trace for your reference. It seems like call excel service succeed.

User=NA\harryc    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

Site=/sites/Harry    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

WcfSendRequest: RemoteAddress: 'http://sbx01:32843/484f9d239d0046a2af1afd67257e47d4/ExcelService.asmx' Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetRange' MessageId: 'urn:uuid:6d0562b4-f247-47c7-b98c-b221c40cccb0'    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
WcfReceiveRequest: LocalAddress: 'http://sbx01.na.qualcomm.com:32843/484f9d239d0046a2af1afd67257e47d4/ExcelService.asmx' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetRange' MessageId: 'urn:uuid:6d0562b4-f247-47c7-b98c-b221c40cccb0'    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

Entering monitored scope (ExecuteWcfServerOperation)    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

ECS RequestId=2012    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

User=0#.w|na\harryc    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

Session=1.V21.5If3FmRT7OZcBtXSYrDI590.5.en-US5.en-US73.+0480#0000-11-00-01T02:00:00:0000#+0000#0000-03-00-02T02:00:00:0000#-006036.57a3d5b9-15eb-47cc-a8b4-aae9bb4a67941.N    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
Document=http://sbx01/sites/Harry/AnalyticsReports/Harry.xlsx    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
Result=Success    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=4.05470527678797    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
WcfReceiveRequest: LocalAddress: 'http://sbx01.na.qualcomm.com:32843/484f9d239d0046a2af1afd67257e47d4/ExcelService.asmx' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetRange' MessageId: 'urn:uuid:6d0562b4-f247-47c7-b98c-b221c40cccb0'    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
Entering monitored scope (ExecuteWcfServerOperation)    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=0.222095266297812    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60
 
Result=Success    beb00800-6d2c-4fad-9f53-fe4ae1f8fc60

If you have any solution on this, please share with us.

Today as May 26, we had a good discussion with Microsoft DSE support and this issue has been resolved now. The trick is Web Anlytical Reports "Custom Reports" is based on excel service and it requires C2WT service in order to pass through the authentication. Here is the way to result the issues.

Log in to central admin -> Central Adminstration -> System Settings -> Manage services on server -> start "Claims to Windows Token Service". See screen shot for your reference.

You will see data generated for Web Anlytical Reports "Custom Reports" as below.

At this time, you should be able to enable the Web Anlytical Reports "Custom Reports" and display the data!

Issue using Content Query Web Part to display audio or video files on SharePoint 2010

As we described before there are several different ways you can play or video media file from a SharePoint 2010 site. There are prons and cons on each of the implementation. You could refer the blog for details.

When we utilize the our of box Content Query Web Part to display audio or video files on SharePoint 2010,  we found one issue that is impacting the usability. The issue is when you have a Content Query Web Part on a page, it overlay and block any other system pages or configuration options. As a result, users  could not configure the web part or could not perform other activities such as create list since those options are overlapped blocked by the black box of the Content Query Web Part. Here are some examples of the issue.

1. If you try to configure the Content Query Web Part, the menu is blocked by the web part and it's very difficult to identify the Configure Web part menu option. See the screen shot.

2. If you want to create other lists or pages from "Site Actions" -> "More Options", the whole Create page might be blocked by the Content Query Web Part.

Of cause, we could use different ways to play or video media file from a SharePoint 2010 site. However, some users just want to get this working to display YouTube videos as I setup even back in SharePoint 2007 version. At this time, I'm not able to resolve this issue.

Does anyone have similar issue and have any solution or workaround?