Tip: Speed Up Snapshot Based Exchange DAG Backups (Veeam, CommVault, etc.)

I was struggling with performance issues with my Exchange 2013 DAG backups, mainly due to storage contention during the snapshot commit (remove snapshot). We have a node that contains only passive copies that we are using strictly for backup. I first tried to put the node into maintenance mode, which helped a little. I decided to try suspending the database copies during the backup and that made a huge difference.

Time to delete snapshot on passive node:
Normal: 1.5 to 3 hours
Maintenance Mode: 45 minutes to 1 hour
Mailbox Database Copy Suspended: 2-3 minutes!!!

Analysis:  Having a Node of the DAG that only contains passive copies of the databases helps reduce the impact of the backup on production users. Putting that node into maintenance mode prevents log files from being played into the databases, but they are still copied to the VM. Suspending the database copies prevents the logs from even being sent to the node, so there are very few changes during the backup. Since from what I have seen, writes typically impact storage performance in a greater capacity, the backup itself (just reading the data) goes pretty well, then deleting the snapshot (combining the snapshot data back into the main disk), really thrashes the storage. By minimizing changes to the disk during the backup, the snapshot commit has very little to do and goes very quickly.

Note: We use Veeam, and I tested a restore of an Incremental Backup using this method and was able to restore and view and restore emails with the Veeam Explorer for Microsoft Exchange with the database suspended.

I first tried this process manually and after it worked, I created a script to do the suspend and resume. My scripts are running on the Exchange Server itself, but you could certainly do this within the Veeam job under the Advanced Settings, Advanced Tab, then Job Scripts.

StartBackup.ps1
get-mailboxdatabasecopystatus -server passivenode|Suspend-MailboxDatabaseCopy -SuspendComment "Backup" -Confirm:$False

StopBackup.ps1
get-mailboxdatabasecopystatus -server passivenode|Resume-MailboxDatabaseCopy -Confirm:$False

Disclaimer: Attempt at your own risk. While I have tested this and it is working well for me, attempt first in a lab...yeah...like everyone has one of those. At the very minimum, run through your own backup/restore test to make sure it doesn't impact your ability to do the restores you want.

Don't put your NETLOGON share and GPOs at risk...check this ASAP!

A very important Active Directory upgrade is being missed by organizations far and wide. Many aren't even aware it is a step that needs to be taken, others make the assumption that it is just done for them. What I'm talking about is the migration from FRS to DFSR for SYSVOL replication. Last year Microsoft announced that they are removing FRS from Windows Server, but that announcement seemed to be largely ignored.  Here's why this is important:

1. If you have a domain that has ever been at a Windows Server 2003 Domain Functional Level or prior, then you have used FRS for SYSVOL replication.  This would be the vast majority of domains being used today.
2. Migrating from FRS to DFSR for SYSVOL replication is not automatic, regardless of whether you upgrade your Domain Controllers or raise your Functional Levels.
3. FRS is antiquated and unreliable for replication.
4. If you haven't migrated, FRS is replicating your NETLOGON share (usually filled with login scripts and other miscellaneous items) and all your Group Policy Objects.

I apologize for the ridiculous font and size, but I'm afraid that most people don't understand the necessity in migrating from FRS to DFSR because they don't really understand what is contained within the SYSVOL. Most will know it sounds familiar, but don't realize that NETLOGON and all their GPOs are contained within this folder.  I don't put cheap gas in my Ferrari *cough* Kia. Okay...usually I do, but I don't want FRS replicating stuff in my Active Directory that is so critical.

So please just check and see if you are still on FRS, or have migrated to DFSR.

Note: If you aren't running at least Windows Server 2008 Domain Functional Level, then you are definitely using FRS.

• On a Domain Controller, Open PowerShell and run "get-addomain|fl Name,DomainMode"
• You are looking for Windows2008Domain or higher 

• Next, run "netdom query fsmo" to find your PDC Emulator.

• On the PDC Emulator DC, run dfsrmig /getglobalstate and dfsrmig /getmigrationstate
• If you have been migrated, you are looking for a global state of Eliminated

• If you see a message that the DFSR migration has not initialized, or get global states of Start, Prepared, or Redirected, then you definitely have some work to do.

I highly recommend these two articles as excellent sources of FRS to DFSR migration information.

http://blogs.technet.com/b/filecab/archive/2014/06/25/streamlined-migration-of-frs-to-dfsr-sysvol.aspx

https://windorks.wordpress.com/2014/11/24/migrating-frs-replicated-sysvol-to-dfsr/

Very cool free AD Replication Status Tool from Microsoft

Not sure how I missed this, but today I stumbled across the Active Directory Replication Status Tool when reading an article by Ned Pyle from Microsoft about FRS to DFSR migrations.

This tool satisfies my top 3 requirements for a utility:

1. It is free
2. It has useful information in a clean and easy to use interface
3. It isn't bloated unnecessarily (12MB post installation!)

The tool can be downloaded here: https://www.microsoft.com/en-us/download/details.aspx?id=30005

There really isn't much to say about the tool, as it is very self-explanatory when you open it. Select Forest, Domain, or Select Targets to specify just a few Domain Controllers.  When you choose Refresh Replication Status, look for the Last Sync Message, Last Successful Sync, and Consecutive Failure Count columns to see how things are doing. You can click Errors Only at the top, and hopefully that will show you an empty screen.

This certainly will not substitute for a good AD Health Check. Neither will this tool fix any issues you find...like...I don't know...someone powers off a Domain Controller they are having problems with and the KCC is like, "What the heck, dude?"

For the "Real men use the command line" crowd, there is nothing this tool does that you can't do yourself with repadmin, powershell, and some time.  You can stroke your red swingline stapler and ignore this blog post.

I do think using screenshots of this tool for documentation purposes could certainly up a SysAdmin's street cred...and who doesn't need that?