I was woken up early morning yesterday and informed about an issue on our SharePoint Farm. All the web applications in the farm were down, and showing errors (as shown in the screenshot below) to any user trying to login.
My first guess was that it was a custom solution (such as a web part, event receiver, timer job) that we deployed that might have caused the issue. Upon investigation, it turned out that was not the case. I then looked at the ULS Logs. The following error was found:
"An exception occurred when trying to issue security token: The requested service, 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc/actas' could not be activated. See the server's diagnostic trace logs for more information.. "
We also saw the error shown in the screenshot below with the User Profile Service Application:
User Profile Application’s connection is currently not available. The Application Pool or User Profile Service may not have been started. Please contact your administrator”.
Upon checking the User Profile Service Application Pool, it was found to be running and the Services were running on the server.
This error told us that something was wrong with Authentication within SharePoint, which is probably why no user was able to log on SharePoint. This also told me that there was probably something deployed lower down the stack, probably either a SharePoint update or a Windows update that broke the STS. We started checking Windows Update and found that one of the updates applied to the server was: kb2756920
My first guess was that it was a custom solution (such as a web part, event receiver, timer job) that we deployed that might have caused the issue. Upon investigation, it turned out that was not the case. I then looked at the ULS Logs. The following error was found:
"An exception occurred when trying to issue security token: The requested service, 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc/actas' could not be activated. See the server's diagnostic trace logs for more information.. "
We also saw the error shown in the screenshot below with the User Profile Service Application:
User Profile Application’s connection is currently not available. The Application Pool or User Profile Service may not have been started. Please contact your administrator”.
Upon checking the User Profile Service Application Pool, it was found to be running and the Services were running on the server.
This error told us that something was wrong with Authentication within SharePoint, which is probably why no user was able to log on SharePoint. This also told me that there was probably something deployed lower down the stack, probably either a SharePoint update or a Windows update that broke the STS. We started checking Windows Update and found that one of the updates applied to the server was: kb2756920
On further checking we figured that one of the update kb2756920 for Windows Server 2008 r2 looks to resolve WCF Security issues with the STS. As explained in the blog at:
However, since this is a production box, installing SP1 would have been a rigorous process and we needed to bring up our farm asap. Uninstalling the update, would have left our STS with the very vulnerabilities that the update was designed to fix. Upon calling Microsoft, we were told we could apply a SharePoint hot fix
described in the link below: The hotfix is something the blog post does not mention and we learnt about it from Microsoft. It can be accessed here:
Just applying this hotfix on both the WFE and APP server and rebooting the servers brought the SharePoint Farm back up. Users were able to login successfully.
However, we still had issues with the User Profile Synchronization service. The User Profile Service Application we showing 0 profiles so we had to do a synchronization back to AD.
Several attempts to the the AD synch failed and we got errors each time. Upon analyzing the ULS logs once again, we got the following error message:
01/14/2013
11:47:13.52
w3wp.exe (0x0200) 0x1014 SharePoint
Foundation Runtime
tkau Unexpected
System.IO.FileLoadException: The located assembly's manifest definition does
not match the assembly reference. (Exception from HRESULT: 0x80131040)
at
Microsoft.Office.Server.UserProfiles.UserProfileConfigManager.InitializeIlmClient(String
ILMMachineName, Int32 FIMWebClientTimeOut) at
Microsoft.Office.Server.UserProfiles.UserProfileConfigManager..ctor(UserProfileApplicationProxy
userProfileApplicationProxy, Guid partitionID) at
Microsoft.SharePoint.Portal.UserProfiles.AdminUI.ProfileAdminPage.IsProfileSynchronizationRunning()
at Microsoft.SharePoint.Portal.UserProfiles.AdminUI.ProfileAdminPage._CheckProfileSynchronizationStateAndNavigateJS()
at
Microsoft.SharePoint.Portal.UserProfiles.AdminUI.ProfileAdminPage.OnPreRender(EventArgs
e) at
Microsoft.SharePoint.Portal.UserProfiles.AdminUI.SyncNow.OnPreRender(EventArgs
e) at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint,
Boolean includeStagesAfterAsyncPoint)
6e86cdbe-18a1-4a14-b83c-7134295e8a60
Upon searching the internet, we found the following blog post:
This blog gave us an idea to do an IISRESET on both the WFE and App Servers. AS soon as we did that, the User Profile Synchronization Service started and we were back up completely.
