Exchange 2010 introduced the concept of Datacentre Activation Coordination (DAC). DAC is a property found on a Database Availability Group (DAG). It still remains slightly mysterious to some folks out in the field, so its certainly worth bringing it back up for discussion.
Tim McMichael’s famous paper plate demo with lots of willing participants was one of my MCM 2010 highlights!
Typically we see the following questions around DAC:
- Does a DAG have DAC enabled by default
- How do I enable DAC
- How do I disable DAC
- What benefits does DAC give me with two AD sites
- Should I enable DAC for a DAG in a Single AD site
- Is enabling DAC on a DAG in a single AD site supported
- What benefits does DAC give me in a single AD site
Is DAC Enabled By Default
NO. You must explicitly enable it in Exchange Management Shell
To Enable DAC
Set-DatabaseAvailabilityGroup -Identity DAG-01 -DatacenterActivationMode DagOnly
To Disable DAC
Set-DatabaseAvailabilityGroup -Identity DAG-01 –DatacenterActivationMode Off
What Benefits Does DAC Give Me With Two AD Sites
TechNet has a good write up on DAC. DAC mode is designed to prevent split brain from occurring by including a protocol called Datacenter Activation Coordination Protocol (DACP). Why is this needed? Let’s go back in history for a quick story.
I used to have an Exchange 2007 CCR cluster in Toronto and a SCR node in Vancouver. When the power failed (as it often did) in Toronto, we would activate the SCR node and get service restored. That was all good, but when the power came back up in Toronto the CCR cluster had quorum, and since the cluster service was responsible for mounting databases, the databases were mounted. The network team would need to do their magic on the WAN, but that was not always quick. Spot the problem there? I have one instance of DB1 mounted and running in Toronto and another instance of DB1 mounted and running in Vancouver! There is no network access between sites, and AD cannot replicate with the changes. Holy database divergence Batman!! Since the DBs had diverged, say hello to all databases being reseeded over the WAN back to Toronto. I used to work around this by setting the BIOS on the physical Exchange boxes not to power on after power loss, and there were copious stickers instructing the NOC not to power on without permission. When it came time to power on, servers were started one at a time and the cluster service was stopped to prevent the databases from mounting. Pretty ugly. Pretty manual.
If only we had something else that was smarter than the cluster service and was aware of such a situation…
That’s where DAC comes in. Think of DAC as application level quorum, over and above what is present in the Windows failover cluster service.
DAC is designed to prevent databases mounting is such a situation. Even if some servers have obtained quorum at the Windows cluster level they are still not allowed to mount databases. Remember that Exchange 2010 does not install Exchange specific resources into the cluster, and Exchange itself is responsible for database failover and mounting. This is why Active Manager was added. But don’t go looking for a Windows service called Active Manager, all of this code runs within the Replication Service. DAC makes Active Manager store a bit in memory (you can’t see it so don’t go looking) and when the server starts up the bit is set to zero. Databases can only be mounted if the bit is set to one. How does the bit get set to one? Either that server:
-
Can contact all the servers on the StartedMailboxServers list – i.e. have access to all necessary nodes
-
Can contact another server on this DAG with the bit set to 1.
A database can only be mounted on the server if the bit is set to 1. Tim describes this as the “Mummy, may I ?” bit. In other words, Active Manager starts up and seeks permission before it blindly mounts databases.
Taking our Toronto / Vancouver scenario into the Exchange 2010 world we would have a DAG that spans two Active Directory sites (no more LCR/CCR/SCR complexity). There would be two Mailbox servers in Toronto and a third in the Vancouver site. We must manually enable DAC.
When the Toronto side went down we would perform the necessary manual steps to activate the Vancouver DR infrastructure. And when the power came back in Toronto we see a difference. The servers could start up and if they could form a cluster with quorum, the cluster service would be up and running. Because the DACP bit is set to 0 on start-up and the Toronto servers cannot get over the WAN to a Vancouver server with the DACP bit set to 1, the Toronto servers cannot change their bit to 1. Since the Toronto servers have a DACP bit of 0, they are unable to mount databases even if the cluster service has quorum.
When the WAN comes back, the servers can then communicate and the Toronto server’s DACP bit will go to 1. They can then mount databases. However the Toronto database copies need to get synchronised with the changes written to the log stream in Vancouver. If all the necessary log files are present the databases will update and the full reseed operation is prevented. Also note there is none of that nasty database divergence stuff!
DAC also simplifies the process of restoring service into the DR data centre. See the section at the bottom for more details on this.
Should I Enable DAC for a DAG In A Single AD Site
At this point the value of DAC can be seen, and this is how it was with Exchange 2010 RTM – DAC was only supported with 3+ member DAGs in multiple AD sites.
Exchange 2010 SP1 added more high availability features, and extended DAC so that it supports a DAG in a single data center. This is buried in the New High Availability and Site Resilience Functionality in Exchange 2010 SP1 article.
In Exchange 2010 SP1, DAC mode has been extended to support two-member DAGs that have each member in a separate data center. DAC mode support for two-member DAGs uses the witness server to provide additional arbitration. In addition, DAC mode has been extended to support DAGs that have all members deployed in a single Active Directory site, including single Active Directory sites that have been extended to multiple locations.
What Benefits Does DAC Give Me In A Single AD Site
In addition to the datacentre resilience aspects there is also another benefit with DAC. If DAC is not enabled then we need to kick it old school and use failover cluster techniques to manage service restoration. If DAC is enabled we do not have to do this, and we can control all of the work from the native Exchange management shell.