Welcome to Part 3 of Exchange 2010 Site Resilient DAGs and Majority Node Set Clustering. In Part 1, I discussed what Majority Node Set Clustering is and how it works with Exchange Site Resilience when you have one DAG member in a Primary Site and one DAG member in a Failover Site. In Part 2, I discussed how Majority Node Set Clustering works with Exchange Site Resileince when you have two DAG members in a Primary Site and one DAG member in a Failover Site. In this Part, I will show an example of how Majority Node Set Clustering works with Exchange Site Resilience when you have two DAG members in a Primary Site and two DAG members in a Failover Site.
Part 3
Real World Examples
Each of these examples will show DAG Models with a Primary Site and a Failover Site.
4 Node DAG (Two in Primary and Two in Failover)
In the following screenshot, we have 4 Servers. Four are Exchange 2010 Multi-Role Servers; two in the Primary Site and two in the Failover Site. The Cluster Service is running only on the four Exchange Multi-Role Servers. More specifically, it would run on the Exchange 2010 Servers that have the Mailbox Server Role. When Exchange 2010 utilizes an even number of Nodes, it utilizes Node Majority with File Share Witness. If you have dedicated HUB and/or HUB/CAS Servers, you can place the File Share Witness on those Servers. However, the File Share Witness cannot be placed on the Mailbox Server Role.
So now we have our five Servers; four of them being Exchange. This means we have five voters. Four of the Mailbox Servers that are running the cluster service are voters and the File Share Witness is a witness that the voters use to maintain cluster quorum. So the question is, how many voters/servers/cluster objects can I lose? Well if you read the section on Majority Node Set (which you have to understand), you know the formula is (number of nodes /2) + 1. This means we have (4 Exchange Servers / 2) = 2 + 1 = 3. This means that 3 cluster objects must always be online for your Exchange Cluster to remain operational.
But now let’s say one or two of your Exchange Servers go offline. Well, you still have at least three cluster objects online. This means your cluster will be still be operational. If all users/services were utilizing the Primary Site, then everything continues to remain completely operational. If you were sending SMTP to the one of the servers in the Failover Site or users were for some reason connecting to the Failover Site, they will need to be pointed to another Exchange Server that is operational in the Primary Site or the Failover Site. This of course depends on whether the user databases are being replicated from a mailbox database failover standpoint.
But what happens if you lose a third node in which all DAG members in the Failover Site go offline including the FSW? Well, based on the formula above we need to ensure we have 3 cluster objects operational at all times. At this time, the entire cluster goes offline. You need to go through steps provided in the site switchover process but in this case, you would be activating the Primary Site and specify a new Alternative File Share Witness Server that exists in the Primary Site so you can active the Exchange 2010 Server in the Primary Site. The DAG will actively use the File Share Witness since there will be 2 Exchange DAG Members remaining which is an even number of nodes. And again, when you have an even number of nodes, you will use a File Share Witness.
But what happens if you lose two nodes in the Primary Site as well as the FSW due to something such as Power Failure or a Natural Disaster? Well, based on the formula above we need to ensure we have 3 cluster objects operational at all times. At this time, the entire cluster goes offline. You need to go through steps provided in the site switchover process but in this case, you would be activating the Failover Site and specify a new Alternative File Share Witness Server that exists (or will exist) in the Failover Site so you can activate the Exchange 2010 Servers in the Failover Site. The DAG will actively use the Alternate File Share Witness since there will be 2 Exchange DAG Members remaining which is an even number of nodes. And again, when you have an even number of nodes, you will use a File Share Witness.
Once the Datacenter Switchover has occurred, you will be in a state that looks as such. An Alternate File Share Witness is not for redundancy for your 2010 FSW that was in your Primary Site. It’s used only during a Datacenter Switchover which is a manual process.
Once your Primary Site becomes operational, you will re-add the two Primary DAG Servers to the existing DAG which will still be using the 2010 Alternate FSW Server in the Failover Site and you will now be switched into a Node Majority with File Share Witness Cluster instead of just Node Majority. Remember I said with an odd number of DAG Servers, you will be in Majority Node Witness and with an even number, the Cluster will automatically switch itself to Node Majority with File Share Witness? You will now be in a state that looks as such.
Part of the Failback Process would be to switch back to the old FSW Server in the Primary Site. Once done, you will be back into your original operational state.
As you can see with how this works, the question that may arise is where to put your FSW? Well, it should be in the Primary Site with the most users or the site that has the most important users. With that in mind, I bet another question arises? Well, why with the most users or the most important users? Because some environments may want to use the above with an Active/Active Model instead of an Active/Passive. Some databases may be activated in both sites. But, with that, if the WAN link goes down, the Exchange 2010 Server in the Failover Site loses quorum since it can’t contact at least 2 other cluster objects. Again, you must have three cluster objects online. This also means that each cluster object must be able to see two other cluster objects. Because of that, the Exchange 2010 Server will go completely offline.
To survive this, you really must use 2 different DAGs. One DAG where the FSW is in the First Site and a second DAG where its FSW is in the Second Site. In my example, users that live in the First Active Site would primarily be using the Exchange 2010 DAG Members in the First Active Site which would be on DAG 2. Users that live in the Second Active Site would primarily be using the Exchange 2010 DAG Members in the Second Active Site which would be on DAG 1. This way, if anything happens with the WAN link, users in the First Active Site would still be operational as the FSW for their DAG is in the First Active Site and DAG 2 would maintain Quorum. Users in the Second Active Site would still be operational as the FSW for their DAG is in the Second Active Site and DAG 1 would maintain Quorum.
Note: This would require twice the amount of servers since a DAG Member cannot be a part of more than one DAG. As shown below, each visual representation below of a 2010 HUB/CAS/MBX is a separate server.
The Multi-DAG Model would look like this.
Prashant says
Hi Bob,
Quick Question.
Primary Site 2 Mailbox + 1 FWS = 3
Secondary Site 2 Mailbox = 2 ( No Alternate Witness Share configure)
( 4 / 2 ) +1 = 5
What have simulated are below:
1. WAN link goes down cluster stays in primary site, which working correctly.
2. if we reboot the FWS server, the cluster its getting automatically moved to secondary site (which should not be). can you tell what should i do to keep the cluster in primary site if the Node Votes are 2 in each site.
FSW is on CAS server on primary site.
Prashant says
Hi,
I do have a query, on this. If i do have a scenario of single DAG for DC and DR site. In such way.
DAG 01 consist of 2 HUB/CAS + 2MBX at DC site and 1HUB/CAS + 2 MBX at DR site.
what happen if my 2 MBX at DC site gets fails?? will the HUB/CAS at DC can connect me to MBX at DR? if so than how can i do this solution with out manual intervention.
Moe says
I have my primary Data Center with three Exchange servers (MB/CAS/HUB) roles on all of them with FSW that is not in use, since they are ODD number of servers. I am designing a DR site with two Exchange server (MB/CAS/HUB) on one server and (MB/HUB) on the other server, as per this article I would need alternate FSW at DR site, right?
5/2 + 1= 3 to get cluster majority?
Michael Newman says
Great post. Does this apply to Exchange 2013 as well?
Bob says
Thanks a lot
Bob says
Hi,
It's really valuable info on this page. Great article. Can you confirm that am I right?
I have three Data Centers:
Primary Data Center
2 Mailbox + 1 Witness
Backup Data Center 1
1 Mailbox
Backup Data Center 2
1 Mailbox
So it is 4 node + Witness. I need 3 of them online to keep cluster online (4/2)+1. So in this particular situation if my Primary Data Center will go offline and remain two node in BDC1 and BDC2 they also will gone offline? So to correct this situation I should add 1 witness server to BDC1 and 1 witness to BDC2 so finally I will have 7 voting elements and at least 4 must be online. So if now PDC site will go offline other backup data centers will work? am I correcty?
Elan Shudnow says
Correct, in this particular situation if the Primary Datacenter goes offline, Backup Datacenter 1 and Backup Datacenter 2 also go offline as the cluster no longer has enough cluster resources (3) to maintain majority. A cluster cannot have more than one file share witness. But what you can do is add more mailbox servers in the backup datacenters.
So for example:
Primary Datacenter (2 Mailbox Servers + 1 Witness)
Backup Datacenter 1 (2 Mailbox Servers)
Backup Datacenter 2 (2 Mailbox Servers)
You now need (6 / 2) + 1 = 4 resources to always stay online whether than may be 3 Mailbox Servers and the FSW or just 4 Mailbox Servers. So now if the Primary Datacenter goes offline, you lost 3 resources but you still have 4 resources online (2 mailbox servers in Backup Datacenter 1 and 2 mailbox servers in Backup Datacenter 2).
Now even though you are adding additional mailbox servers which means extra licensing, it does not mean that you have to add database storage to these servers and large amounts of memory. They can be just sitting there for quorum purposes and not necessarily for database mounting.
Baldeep says
thanks for such a good post gys….
kane Hipolito says
Wow thanks for the info you share in here now I know what I'm going to do.. keep it up guys..
VZ 58
Adam says
Thanks very much for the Info – Very good
wolffparkinsonwhite says
Good catch. Of course the principle is the same but I realize that the wording doesn't reflect the Visio properly since I say the first DAG would have its Primary FSW in the first site, etc…
svoller says
so from Raymonds scenario you could actually have fully active/active DC's with 2 DAG's each with 2 members (one at each site) and FSW at the primary and secondary sites within each DAG to achieve full resilience.
Site 1
DAG1 = MBX1(Active databases for Site 1)+FSW
DAG2 = MBX2 (Passive Copy of MBX3)
Site 2
DAG2 = MBX3(Active databases for Site 2)+FSW
DAG1 = MBX4 (Passive copy of MBX1)
Typical failure scenarios would be:
If the server in Site A failed, clients would be redirected to server in site B via autodiscover ?
If the server in Site B failed, clients would be redirected to server in site A via autodiscover ?
If WAN link went down then the active servers would stay active in each DAG as they have the FSW in the site to maintain quorom.
So the question is what is the minimum number of mailbox servers in the 2 DAG scenario to achieve a fully active/active datacentre setup with resilience to wan link failure ? And in the event of a single server failure at either site in the dag or wan link failure clients wont loose connectivity to thier mailbox.
raymond says
No problem Elan. Thanks for a great series of posts!
I have a question along the same lines as @svoller above. I am designing an active/active solution for a client. I understand that best practice for active/active sites is two DAGs, with the FSW located at the site where the active users are – but having 4 servers for their two DAG-protected sites is a bit of a bitter pill to swallow for the client. The client actually has 3 active sites (the 3rd much smaller) – if they went with a single server per site, I understand if a WAN link goes down, a site will be isolated and its MBX will go offline.
But if the WAN is down, the site won't be able to send external mail anyway, so this may actually be acceptable to the client. But my concern is when the WAN link comes back up. I assume DAC kicks in, and the DB copies on it become passive copies (the DBs having failed over to other site(s)) and resume replication from the active copy from another site. Will the CAS in the site proxy Outlook connections automatically to the active copy (now hosted in another site) when the WAN comes back up? I'm just trying to understand how easy it would be for the client to recover from loss of quorum in this scenario, once the WAN is back up.
Elan Shudnow says
Well the clients that come up with contact Autodiscover. Because the WAN link is up, DNS will synchronize, and clients in the "downed but now back up site" will get autodiscover DNS pointing to the primary site. Let's say Site B is site that was activated during failure. Because Autodiscover will see the active copy is activated in Site AB, the clients will get the InternalURLs for Site B. And because the DNS record for the RPC Array for their database will be pointing to Site AB, they will do MAPI connects to Site B.
If Site A is your typical Primary Site, when you transition things back to Site A, autodiscover will now see that the user's mailbox is in Site A and start handing them out InternalURLs for Site A. And you would have also changed the RPC Array FQDN in DNS back to Site A.
svoller says
I read it again and it seems that for active/active with local site resilience it requires 2 Dags and 3 servers per dag.
Is my understanding correct as below ?
Site1 = DAG 1, 2 Servers
Site 2 = DAG 1, 1 Server (Copies only)
SIte 1= Dag 2, 1 Server (Copies only)
Site 2 = Dag 2, 2 Servers
thanks again for the usful info
Elan Shudnow says
You only need two DAGs in an Active/Active if you care about 1 location staying online even if there is a WAN failure as only 1 site will ever have the Majority. So if you require both sites to stay alive even during a WAN outage between the two locations, that's where you'd have 2 DAGs with majority of the voters being in one location and another DAG with its majority of the voters being in the other location.
DAC is what prevents split-bran so when the servers come back up, they have to be able to talk to the other servers that are operational before it can mount its databases thus preventing split brain.
svoller says
Elan
Thanks for the article it is most informative and very useful although I think I will need to re-read this again to get things clear.
Is there any other options with Active/Active sites other than 2 DAG's ? Or is it that to run active/active you must always have 2 DAG's no matter how many MBX servers that you have to ensure that you dont get a split brain syndrome?
How does DAC mode affect the design if at all ?
raymond says
Hi Elan, I suspect your last diagram (Multi-DAG model) may be wrong. I think the labels for DAG1 and DAG2 should be switched? The FSW for the First Active Site is in DAG2, not DAG1 (and vice versa for the Second Active Site).
Elan Shudnow says
Good catch. Of course the principle is the same but I realize that the wording doesn't reflect the Visio properly since I say the first DAG would have its Primary FSW in the first site, etc…
Thanks for catching this.