Procedure, tips, and verification scripts to build a server-to server trust between SharePoint Server 2013 and SharePoint Online for one-way outbound hybrid search

After replacing the STS certificate in the on-premises SharePoint farm described in previous blog, you should be ready to set up server-to-server authentication for hybrid environments. You need to create a trust relationship between your on-premises SharePoint farm and your SharePoint Online tenant, which uses Azure Active Directory as a trusted token signing service. This process described in technet will be done on one of the SharePoint on-premises WFE server that has online service management tools installed. Since there are several errors in the technet, we had to cross check other two instructions. One is published by Microsoft escalation engineer MANAS BISWAS and  Bill Baer. Here are the detailed steps, tips, and tricks.

1. First, you need to install the following online service management tools on ONE on-premises SharePoint Server 2013 web server.

• Microsoft Online Services Sign-In Assistant for IT Professionals BETA (64 bit version)
• Azure Active Directory Module for Windows PowerShell (64 bit version)
• SharePoint Online Management Shell (64 bit version)

There are some tricks and tips.

• Only install the BETA (64 bit version) of Microsoft Online Services Sign-In Assistant for IT Professionals. If you installed versions like RTW, you will not be able to install Azure Active Directory Module for Windows PowerShell. You will get MOSSIA is not installed error. At this time, you MUST install Microsoft Online Services Sign-In Assistant for IT Professionals with BETA (64 bit version).

• After Microsoft Online Services Sign-In Assistant for IT Professionals installed, the registries will be updated and server bounce is required. If the install did not prompt you to restart the server, reinstall and do a repair. It will then ask you to bounce the server.

By adding the required Windows PowerShell modules and snap-ins, the following process can occur in a single Windows PowerShell window on the on-premises SharePoint web server. 

2. Second, you will execute the following powershell on on-premises SharePoint WFE to create the trust and proxy.

# Set environment
Add-PSSnapin Microsoft.SharePoint.PowerShell
Import-Module Microsoft.PowerShell.Utility
Import-Module MSOnline -force –verbose
Import-Module MSOnlineExtended -force –verbose
Import-Module Microsoft.Online.SharePoint.PowerShell -force

# Set the certificate
$stscertpfx="E:\source\O365\O365DEV.pfx"
$stscertcer="E:\source\O365\O365DEV.cer"
$stscertpassword="Qualcomm1"
$spcn="*.qualcomm.com"
$spsite="https://sharepointdev.qualcomm.com/"
$spoappid="00000003-0000-0ff1-ce00-000000000000"


# Update the Certificate on the STS - You could skip this if you already done this
$pfxCertificate=New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $stscertpfx, $stscertpassword, 20
Set-SPSecurityTokenServiceConfig -ImportSigningCertificate $pfxCertificate

# Type Yes when prompted with the following message.
#You are about to change the signing certificate for the Security Token Service. Changing the certificate to an invalid, inaccessible or non-existent certificate will cause your SharePoint installation to stop functioning. Refer to the following article for instructions on how to change this certificate: http://go.microsoft.com/fwlink/?LinkID=178475. Are you sure, you want to continue?

#Restart IIS so STS Picks up the New Certificate - need to be done on all SharePoint servers
iisreset
net stop SPTimerV4
net start SPTimerV4

#To validate that the above commands has run successfully, you can run any of the following cmdlets. The certs should be matching
$pfxCertificate
(Get-SPSecurityTokenServiceConfig).LocalLoginProvider.SigningCertificate


#Do Some Conversions With the Certificates to Base64
$pfxCertificate = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList $stscertpfx,$stscertpassword
$pfxCertificateBin = $pfxCertificate.GetRawCertData()
$cerCertificate = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cerCertificate.Import($stscertcer)
$cerCertificateBin = $cerCertificate.GetRawCertData()
$credValue = [System.Convert]::ToBase64String($cerCertificateBin)

# Establish Remote Windows PowerShell Connection with Office 365
enable-psremoting

#When prompted with Are you sure you want to perform this action? type Yes for all of the actions.
new-pssession

# Log on as a Global Administrator for Office 365
Connect-MsolService

#When prompted, provide the Global Admin account for your Office 365 tenant. This would have been sent to your corporate e-mail address when you signed up for the tenant.

# Register the On-Premise STS as Service Principal in Office 365
New-MsolServicePrincipalCredential -AppPrincipalId $spoappid -Type asymmetric -Usage Verify -Value $credValue




# Add an SPN for your public domain name to Azure Active Directory
$SharePoint = Get-MsolServicePrincipal -AppPrincipalId $spoappid
$spns = $SharePoint.ServicePrincipalNames
$spns.Add("$spoappid/$spcn")
Set-MsolServicePrincipal -AppPrincipalId $spoappid -ServicePrincipalNames $spns
$spocontextID = (Get-MsolCompanyInformation).ObjectID
$spoappprincipalID = (Get-MsolServicePrincipal -ServicePrincipalName $spoappid).ObjectID
$sponameidentifier = "$spoappprincipalID@$spocontextID"

# Set the SharePoint authentication realm
$site=Get-Spsite "$spsite"
$appPrincipal = Register-SPAppPrincipal -site $site.rootweb -nameIdentifier $sponameidentifier -displayName "SharePoint Online"
Set-SPAuthenticationRealm -realm $spocontextID


# Configure an on-premises proxy for Azure Active Directory and establish in the On-Premise Farm a Trust with the ACS
New-SPAzureAccessControlServiceApplicationProxy -Name "ACS" -MetadataServiceEndpointUri "https://accounts.accesscontrol.windows.net/metadata/json/1/" -DefaultProxyGroup
New-SPTrustedSecurityTokenIssuer -MetadataEndpoint "https://accounts.accesscontrol.windows.net/metadata/json/1/" -IsTrustBroker -Name "ACS"



3. Third, you need to verify the O365 certs match the on-premises, proxy configured, and ready for hybrid search configuration.

A. Verify SharePoint server and STS service have the identical certificates. You could use the following powershell and verify the thumbprint.

# Verify whether the SahrePoint on-premises server certificate match the STS service certificate   
Add-PSSnapin *sh* -ea 0
$stscertpfx="E:\source\O365\O365Sbx.pfx"
$stscertcer="E:\source\O365\O365Sbx.cer"
$stscertpassword="Qualcomm1"
$spcn="*.qualcomm.com"
$spsite="https://sharepointdev.qualcomm.com/"
$spoappid="00000003-0000-0ff1-ce00-000000000000"

#Validated STS Token Signing certifciate thumbprint
$pfxCertificate=New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $stscertpfx, $stscertpassword, 20
$pfxCertificate

# This cert is from SharePoint STS service
Get-SPSecurityTokenServiceConfig).LocalLoginProvider.SigningCertificate 
 

B. Verify certificate uploaded to O365 not expired using the following commands.
Connect-MsolService
Get-MsolServicePrincipalCredential -AppPrincipalId "00000003-0000-0ff1-ce00-000000000000"

You need to press enter and see the output with expiration data for the cert.

C. Validate SPNs setup properly in O365 using the commands.
$app = Get-MsolServicePrincipal -AppPrincipalId "00000003-0000-0ff1-ce00-000000000000"
$app.ServicePrincipalNames

You can verify the domain for the SharePoint on-premises should be in the list.

PS C:\Users\SPdev1> $app = Get-MsolServicePrincipal -AppPrincipalId "00000003-0000-0ff1-ce00-000000000000"
$app.ServicePrincipalNames
00000003-0000-0ff1-ce00-000000000000/*.qualcomm.com
00000003-0000-0ff1-ce00-000000000000/*.sharepoint.com
00000003-0000-0ff1-ce00-000000000000
Microsoft.SharePoint

D. Validate User Profile Service Application Status and it should be "online"
$upa=Get-SPServiceApplication | where-object {$_.TypeName -match "User Profile Service Application"}
$upa
$upa.status

E. Validaye on premises ACS Proxy created and it should be the named you configured in previous script. In this case, it's "ACS".
Get-SPServiceApplicationProxy | ? {$_.Name -eq "ACS"} | fl

You could check SharePoint central administration->Manage Service Applications. You should see new ACS Azure Access Control Service Application Proxy listed.

F. You can also verify the trust from SharePoint central administration->Security->Manage trust. The ACS proxy should be created as in the following screenshot.

Next it is ready to continue the search hybrid configuration. Please note there are some errors in the Microsoft technet article and the procedure we verified is based on Microsoft escalation engineer Manas.

One tip is you could suppress the  users import like login name, password, and the confirmation when you run the above powershell commands. You could also encrypt the password in the file.

Here is the procedure to encrypt the password. Login as the user account to window server and run the following script and the password will be encripted in the file.

read-host -AsSecureString | ConvertFrom-SecureString | out-file C:\cred.txt

You could use this file to auto accept the password. Here is the automated commands that will not prompt any input. The highlighted parameters are the key for the automation.


$o365admin = "userID@domain.company.com"
$password = get-content C:\cred.txt | convertto-securestring
$Creds = New-Object System.Management.Automation.PSCredential($o365admin, $password)

enable-psremoting -Force

#When prompted with Are you sure you want to perform this action? type Yes for all of the actions.
new-pssession

# Log on as a Global Administrator for Office 365
Connect-MsolService -Credential $Creds

See Ultimate procedure to display SharePoint online hybrid search results in SharePoint Server 2013 for other steps to configure hybrid search. 

SharePoint 2013 issue to configure audience - Rule context is not detected. This page needs to be loaded from Audience page.

When we configure audience targeting for SharePoint hybrid search on one of the 2013 farm, we got the following error when configure the audience rule.

"Rule context is not detected. This page needs to be loaded from Audience page."

This issue seems to be introduced by the latest 2014 October CU. We have no such issue on SP1 with August CU but have this issue after 2014 Oct CU installed. This issue will also impacting to update the audience. You will get same similar error.

Before this issue fixed by Microsoft November KBs, the workaround is to use powershell to add audience roles. Here is the example to create audience and add roles.

$spsite = New-Object Microsoft.SharePoint.SPSite("http://sharepoit.mycompony.com");       
$spcontext = [Microsoft.Office.Server.ServerContext]::GetContext($spsite);       
$audmanager = New-Object Microsoft.Office.Server.Audience.AudienceManager($spcontext);

$audcollection = $audmanager.Audiences;

$audname = "O365 Hybrid Users";
$auddesc = "Office 365 hybrid test user group";
$newaud = $audcollection.Create($audname, $auddesc);
$newaud.AudienceRules = New-Object System.Collections.ArrayList;

$property = "DL";
$operator = "Member of";
$value    = "CN=O365.Users,OU=Distribution Lists,OU=Exchange,DC=na,DC=mycompany,DC=com";
$newrule  = New-Object Microsoft.Office.Server.Audience.AudienceRuleComponent($property, $operator, $value);
[Void]$newaud.AudienceRules.Add($newrule);
$newaud.Commit();

This is working for us as workaround but you might need to apply the November fix in the future. Please note this workaround only create the roles for audience, you will still get error when to set the audience compiling schedule.

Another issue is if you compile the audience by clicking the compile audience link manually, you do not see the new users added to the audience. However, if the scheduled job runs, it will compile the new users. We could not change the compiling schedule and even we enter the all necessary information, it will still display error like below.

We could need to work with Microsoft to resolve this.

Options and tips to customize the Office365 navigation links

The new Office 365 and SharePoint 2013 introduced a suite bar at the top of the screen and you might need to customize it by adding some links. This particular useful if you want to users to browse back to SharePoint on-premises in hybrid implementation. In this blog, I would provide three different options and the pros and cons for the implementation. You could decide the option that best fit for your company.

Although Office 365 navigation bar will not allow the customization that can be easily done on SharePoint on-premises by adding custom user actions. I’m able to find the following three different ways to “modify” O365 navigation links. Here are the summary of the implementation.

1. Add a logo with link to top left navigation bar from Office 365 Admin Dashboard
2. Add the links to Office 365 site global navigation bar through Powershell and CSOM
3. Add links to top Office 365 navigation bar by applying the custom master page

The three implementations are displayed in the following screenshot. 

All of them have different pros and cons. I’ll explain the procedure how to implement them and the tips.

Option 1 - Add a logo with link to top left navigation bar from Office 365 Admin Dashboard. You could follow the following procedure to add the logo with the link.

• Login to your Office 365 Admin Dashboard (https://portal.office.com/admin)
• Click the your company name link on the right side
• From the property page choose a logo, pick your URL for where you would want the Logo to link to; accent colors, text and link colors and other Office 365 logo colors

Pros:

• No code implementation and easy to configure
• Global to all O365 application

Cons:

• Not use friendly and obvious to end users
• May break when O365 upgrade 
• Only O365 global admin could do this

Option 2 - Add the links to Office 365 site global navigation bar through Powershell and CSOM. You could use the following Powershell script snip and CSOM to add links to SharePoint online site navigation.

 #Credentials to connect to office 365 site collection url
$username="harryc@myqualcomm.onmicrosoft.com"
$password="password"

#Load CSOM libraries
Set-Location $PSScriptRoot
Add-Type -Path (Resolve-Path "Microsoft.SharePoint.Client.dll")
Add-Type -Path (Resolve-Path "Microsoft.SharePoint.Client.Runtime.dll")

#Load O365 site
$siteURL = "https://myqualcomm-my.sharepoint.com"
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)
$Context = New-Object Microsoft.SharePoint.Client.ClientContext($siteURL)
$Context.Credentials = $credentials
$context.RequestTimeOut = 5000 * 60 * 10;
$web = $context.Web
$site = $context.Site
$context.Load($web)
$context.Load($site)
$context.ExecuteQuery()

#Add SharePoint on-premises link
$OnPremisesURL = "http://on-premises-shrepoint.com"
$OnPremisesTitle = "QC SahrePoint"

$Nodes = $Context.Web.Navigation.TopNavigationBar
$NavigationNode = New-Object Microsoft.SharePoint.Client.NavigationNodeCreationInformation
$NavigationNode.Title = $OnPremisesTitle
$NavigationNode.Url = $OnPremisesURL
$NavigationNode.AsLastNode = $true  
$Context.Load($Nodes.Add($NavigationNode))
$Context.ExecuteQuery()

Pros:

• Easy to implement
• User can modify and manage themselves from UI
• This can be done through UI by site collection admin also

Cons:

• Not global to all O365 application but the SharePoint site
• You need to run this against all site collections

Option 3 - Add links to top Office 365 navigation bar by applying the custom master page. You could modify the master page by adding the following lines of code just before the line OoB navigation bar. We use seattle.master default master as example. You could add just befor the following line.

<SharePoint:AjaxDelta runat="server" id="suiteBarDelta" BlockElement="true" CssClass="ms-dialogHidden ms-fullWidth noindex">

This is the new link we will add. You could use click F12 and then locate the navigation bar in the master page code by clicking the bar.

 

<div class="dd"><style>

.dd {
    color: white;
    color: white;
    display: block;
    font-size: 13px;
    left: 250px;
    margin-left: 100px;
    position: absolute;
    top: 16px;
    z-index: 999;
}
a.ddLink, a.ddLink:visited
{
    color: white !important;
}

</style><a class="ddLink" href="http://www.google.com"><Label>TestingURL</Label></a></div>

Pros:

• User friendly
• Can be done by site collection administrator

 Cons: 

• Only global to the sub-site level and you need to apply to all web inside all site collections
• When new site created, you need to find a way to apply this master page
• You need to enable both publishing features on site collection and web before you can change master page
• Future O365 master page change will not be pick up until you apply the change to custom master page

Based these the complexity of the options, pros, and cons, you could find the best fit for your implementation.

In order for you to understand the impact of the O365 changes to the above three different ways to “modify” O365 navigation links, I pushed the latest O365 changes that should be as of Dec. 12 2014 to my O365 that is updated two week ago. The option #2 master page will be overwritten by the new update. Option #1 changed the company icon to the middle. The option #3 I'm not sure why the links are duplicated. It might not be caused by the O365 update. The three options after the O365 update is shown in the following screenshot. 

Now, it seems to me that option 1 should be the best option with lest effort and less impact by the O365 updates.