Sometimes best practice – or rather, perceived best practice – goes horribly wrong. This fact of computing life is revealed with a vengeance in a recent blog by MCM Andrew Higgenbotham describing a difficult support problem with Exchange 2013 RTM CU2. The problem occurred after CU2 was installed when users could not connect to their mailboxes with Outlook Web App (OWA), which displayed the catch-all message “the website cannot display the page”.
As described in the blog, the OWA protocol log stated bluntly that the mailbox database users to which were attempting to connect did not exist. Yet the mailbox database was available, mounted, online, and ready to accept client connections on a multi-role standalone Exchange 2013 RTM CU2 server.
Support problems have a nasty habit of satisfying the old adage that “the devil is in the detail”. In this instance, the detail was the standalone nature of the server. It was not part of a Database Availability Group. Its multi-role nature should have simplified matters as both the CAS and mailbox roles were collocated and incoming connections handled by the CAS couldn’t have a much simpler transit than to the mailbox database mounted on the same server.
The standalone nature of the server became important when some detective work revealed that the administrator of the server had been reading up about Exchange 2013 best practice and had found recommendations outlining how to put servers into maintenance mode before applying updates. One step is to instruct mailbox servers not to accept new instructions to activate database copies, which is done by running the Set-MailboxServer cmdlet. For instance:
Set-MailboxServer -Identity ExServer1 -DatabaseCopyAutoActivationPolicy Blocked
I am guilty of making such recommendations, as when I wrote about installing Exchange 2013 RTM CU1. The problem is that preventing the activation of database copies is only something that is really applicable to mailbox servers that are members of a Database Availability Group (DAG). You don’t need to block activations when a server is standalone because by definition its mailbox databases are standalone too and have no association with any other database copies.
However, the administrator had duly blocked database activations before applying the Exchange 2013 RTM CU2 update. Cue confusion and user dismay when the update finished and OWA could not connect. For whatever reason, the CAS running on the same server refused to proxy inbound connections to the mailbox server because it believed that the target database was unavailable.
The resolution is simple as the block can be removed by running the command:
Set-MailboxServer -Identity ExServer1 -DatabaseCopyAutoActivationPolicy Unrestricted
I’m sure that the folks who troubleshot the issue over a night were mightily relieved when they found the answer.
No blame can be attached to the administrator for reading up on best practice. We all try and keep up to date with the state of the art as it applies to our professional activities and recommendations made by well-respected individuals are sometimes taken as fact instead of being what they really are: observations and recommendations based on the configurations and environment that the writer was using at the time. The longest surviving tenet of best practice pertinent to Exchange is to always test before deploying into production. This principle is as applicable to advice that you receive as it is to new updates issued by a software vendor.
I’ve learned from the situation that I have to be more specific when I describe problems and their solutions. The simple fact is that a DAG is a very different working environment to standalone servers. Remembering that is a very good thing; remembering to validate ideas sourced on the Internet or at conferences (even Exchange Connections) is even better.
Follow Tony @12Knocksinna
