Project Coconut: Part 5 - Its mailbox migration time!

May 6th, 2008

Project Coconut entries:

Now that the Exchange server is installed and configured it is time to migrate the mailboxes. You can take a look at the existing mailboxes in the Exchange organisation using the Exchange Management Console or using the Get-Mailbox cmdlet in the Exchange Management Shell.

mailboxmigration003

[PS] C:\>Get-Mailbox

Name                      Alias                ServerName       ProhibitSendQuo
                                                                ta
----                      -----                ----------       ---------------
Administrator             Administrator        labex1           unlimited
User1                     User1                labex1           unlimited
User2                     User2                labex1           unlimited
User3                     User3                labex1           unlimited
User4                     User4                labex1           unlimited
John Adams                johnadams            labex1           unlimited
Paul Anderson             paulanderson         labex1           unlimited
Gary Billups              garybillups          labex1           unlimited
Frank Curtin              frankcurtin          labex1           unlimited
Harry Kindle              harrykindle          labex1           unlimited
Carl Munro                carl munro           labex1           unlimited
WARNING: Object lab.internal/Lab Corp/Users/Carl Munro has been corrupted and
it is in an inconsistent state. The following validation errors have occurred:
WARNING: "carl munro" is not valid for Alias. Valid values are: Strings formed
with characters from a to z (uppercase or lowercase), digits from 0 to 9, !, #,
 $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may
be embedded in an alias, but each one of them should be preceded and followed
by at least one of the other characters. Unicode characters from U+00A1 to
U+00FF are also valid in an alias, but they will be mapped to a best-fit
US-ASCII string in the email address which is generated from such an alias.

The first thing you might notice is the warning about an invalid alias on one of the mailbox users. The EMC and EMS will warn you of any such problems with mailboxes and you should take steps to fix them before you attempt the mailbox migration. With just one invalid alias it is easy to fix, but if you have dozens or even hundreds of them you’ll need some scripting assistance. Fortunately the Microsoft Exchange Team has come up with a script to make this easy.

Consider Transaction Logging

Bulk mailbox migrations generate a lot of transaction logging on the target server, as each mailbox item moved is a line in a log file. This can cause disk space problems on even very large capacity volumes, which is a problem you definitely want to avoid as running out of disk space will cause the databases in that storage group to dismount.

There are two approaches you can take to work around this:

  • use circular logging during the mailbox migrations (not recommended)
  • migrate the mailboxes in smaller batches and use backups to truncate the resulting log files (recommended)

I generally recommend the second approach as circular logging carries some risks. By migrating the mailboxes in smaller batches you allow your support staff to better handle any unforeseen problems that arise because they will only affect smaller numbers of users. It also makes the migrations easier to schedule in limited non-business hours. And finally it encourages a post-migration backup of the database which is just a plain good idea. In real life I tend to aim for a 2-3 hour window for mailbox moves in the evening and then launch a backup and go to sleep.

How to Migrate in Batches

The Exchange Management Console makes it very simple to migrate users in batches by simply using filters to limit your view of mailboxes to just those you wish to migrate. For example, if I planned on migrating all users with surnames starting with “A” I could apply such a filter easily.

mailboxmigration002

Similarly I could plan to move users according to which Mailbox Database they are on, and filter on that criteria.

mailboxmigration004

You can also use the Exchange Management Shell to move users who match certain criteria. To move all users with surnames starting with “A” using the Exchange Management Shell I can pipe a Get-User query into the Move-Mailbox cmdlet.

[PS] C:\>Get-User | where {$_.RecipientType -eq "UserMailbox" -and $_.LastName -
like "A*"} | Move-Mailbox -TargetDatabase "Mailbox Database" -Confirm:$false

Using -Confirm:$false ensures that I am not prompted for each mailbox move.

mailboxmigration005

In the next part of Project Coconut I’ll discuss some of the post-migration issues you may encounter, and go through the process of removing the legacy Exchange servers from the organisation.

Project Coconut: Part 4 - Configuring Standby Continuous Replication

May 5th, 2008

Project Coconut entries:

Project Coconut, now with even more blown deadlines!

What is Standby Continuous Replication?

Standby Continuous Replication (SCR) is a great new feature of Exchange Server 2007 SP1. SCR is basically a way to maintain an up to date replica of your storage groups and mailbox databases on a second server in case of a disaster. Typically this would be a server in a DR site somewhere in your network.

SCR1

You can also use it to host replicas for multiple mailbox servers around your network, providing a single shared standby server.

SCR2

SCR uses the same transaction log shipping and replay that is used in Local Continuous Replication and Cluster Continuous Replication. Each time a transaction log is written and closed off (they are each 1mb in size) it is copied to the standby server and then replayed into the standby database. If you are considering using SCR in your own environment here are a few planning points:

  • SCR is available in both Standard Edition and Enterprise Edition of Exchange Server 2007 SP1.
  • SCR is available in both clustered and non-clustered Mailbox Servers (this applies to both the source and the target server).
  • The target server must be installed with the Mailbox Server role and have the same disk layout as the source server for a given storage group replica. For example if a storage group is configured to use L:\SG1Logs for logging and H:\SG1Data for the database, these paths must be available on the target server. You should also ensure that equivalent disk space is provisioned on the target server.
  • Unlike CCR and LCR you cannot use a the passive database copy on an SCR target server to run your backups.
  • An SCR source server can have more than one target, and an SCR target server can be the target for more than one source server.
  • SCR can effect your backups.

How to configure Standby Continuous Replication

To demonstrate SCR I have two Exchange Server 2007 servers running in the lab.

  • LABEX2 - member server running Exchange Server 2007 Client Access, Hub Transport, and Mailbox Server roles. This will be the SCR source server.
  • LABDC1 - domain controller running Exchange Server 2007 Mailbox Server role. This will be the SCR target server. After installing the server I deleted the default Storage Group and Mailbox Database as they are not needed. Note: in a production environment I wouldn’t recommend using a domain controller for this.

Note the Storage Group and Mailbox Database folder paths on LABEX2. These folders must be available (the system will create them itself) for SCR to use on LABDC1. If the drive letter does not exist on the target server, or the folder path is in use already, then SCR will fail.

[PS] C:\>Get-StorageGroup -Server LABEX2 | fl name, systemfolderpath, logfolderp
ath

Name             : First Storage Group
SystemFolderPath : C:\FSGLog
LogFolderPath    : C:\FSGLog

[PS] C:\>Get-MailboxDatabase -Server LABEX2 | fl name, edbfilepath

Name        : Mailbox Database
EdbFilePath : C:\FSGData\Mailbox Database.edb

When enabling SCR, the first of the commands can be run from either the source or target server, but the remainder must be run on the target server. For that reason it is easier to just run all of the commands on the target server.

First we enable replication.

[PS] C:\>Enable-StorageGroupCopy "LABEX2\First Storage Group" -StandbyMachine LA
BDC1

If we take a look at the status of SCR at this time we can see that First Storage Group has changed from “Not Configured” to “Disabled”.

[PS] C:\>Get-StorageGroupCopyStatus -Server LABEX2 -StandbyMachine LABDC1

Name                      SummaryCopySt CopyQueueLeng ReplayQueueL LastInspecte
                          atus          th            ength        dLogTime
----                      ------------- ------------- ------------ ------------
First Storage Group       Disabled      0             0
Second Storage Group      NotConfigured 0             0

When SCR is first enabled it does not immediately generate the target database. There is a delay of 50 transaction logs which must first be shipped to the target server. These will then not replay until the “ReplayLagTime” interval has elapsed, which by default is 24 hours.

To speed up the process of creating the SCR replica on the target server you can manually seed the replica.

[PS] C:\>Suspend-StorageGroupCopy "LABEX2\First Storage Group" -StandbyMachine L
ABDC1

Confirm
Are you sure you want to perform this action?
Suspending Storage Group Copy "First Storage Group".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help
(default is "Y"):y

[PS] C:\>Update-StorageGroupCopy "LABEX2\First Storage Group" -StandbyMachine LA
BDC1

[PS] C:\>Resume-StorageGroupCopy "LABEX2\First Storage Group" -StandbyMachine LA
BDC1

The Update-StorageGroupCopy operation can take some time if the number of transaction logs and the size of the database are large, particularly if the seeding is occurring over a slower WAN link. Keep this in mind when seeding your SCR target servers.

Maintaining Standby Continuous Replication

When an Exchange Server 2007 server is an SCR target you cannot backup the replicated databases. These can only be backed up on the source server. Therefore if the SCR target server is not performing any other roles in your Exchange organisation then you only need to perform backups of the operating system and system state. In some cases you don’t need to backup anything at all, as Exchange Server 2007 has excellent restoration procedures for recovering a failed server with its full configuration intact (as this is stored in Active Directory, not on the server itself).

However, because of the potential impact on your normal Exchange backups, it is important to monitor the health of Standby Continuous Replication. At the very least you should use Get-StorageGroupCopyStatus to check the health of each Storage Group’s replication, and monitor the SCR target server for MSExchangeRepl warning or error events in the Application Log.

In the next part of Project Coconut I’ll demonstrate a few shell commands to help with the migration of mailboxes to the new Exchange Server 2007 server.

Macbook Air vs Lenovo X300

May 2nd, 2008

Meet the nerds behind Exchange Server 2007

April 30th, 2008

The Microsoft Exchange Server team has thrown together a 5 minute video so that you can see the team members talk about their roles and how much they love their jobs.  You can watch the video here.

exchangeteam

Want to go work with these guys?  Check out the Exchange Team career site.

Exchange Server 2007 replication problems can lead to backup issues

April 25th, 2008

exchange-2007-logo2 Exchange Server 2007 has some very useful replication features such as Local Continuous Replication (in RTM and SP1) and Standby Continuous Replication (in SP1 only).  These features can provide nice and simple disaster recovery options by replicating your storage group logs and databases to another location, be it another disk/LUN with LCR or another mailbox server with SCR.

However, if you choose to implement LCR or SCR you should be aware of the implications for your backups.  A normal Exchange backup will truncate the transaction logs for your storage group, however when LCR or SCR are deployed this truncation may not work as you first expect.

One example of this is when using SCR to replicate your storage group to another mailbox server.  The source mailbox server ships each transaction log to the SCR target where it is replayed into a replica of the database.  In this scenario there are two queues created - the copy queue and the replay queue.

[PS] C:>Get-StorageGroupCopyStatus -StandbyMachine SERVER2 

Name                      SummaryCopySt CopyQueueLeng ReplayQueueL LastInspecte
                          atus          th            ength        dLogTime
----                      ------------- ------------- ------------ ------------
First Storage Group       Healthy       2             3232         25/04/200... 

The copy queue length is the number of transaction log files yet to be shipped to the standby server.  The replay queue length is the number of transaction log files yet to be replayed into the replica database (in this case the default 24 hour replay delay has been used).

Now that the source server knows it is shipping logs to a standby server it will be sure not to remove any transaction logs that are yet to be shipped.  This would mean that if replication is suspended or fails for any particular reason, and the copy queue length begins to grow quite long, that the source server will not truncate the logs during a normal backup.

However, even when SCR appears healthy and has a zero length copy queue there may be problems with the RPC calls between the servers to notify of log truncation events.  If the source and target cannot verify with each other that the logs have shipped and replayed then the source server will not truncate the logs during a normal backup.

To monitor for these types of scenarios it is wise to check for MSExchangeRepl warnings and errors on the target server.  For example:

Event Type:    Warning
Event Source:    MSExchangeRepl
Event Category:    Service
Event ID:    2137
Date:        19/02/2008
Time:        6:31:25 AM
User:        N/A
Computer:    SERVER2
Description:
Log truncation request to the Information Store using RPC has failed for storage group ‘SERVER1\First Storage Group’. Error code: 4294966264.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

When such SCR issues are interfering with backup operations one of the simplest ways to resolve the issue is to reseed the SCR target.

[PS] C:\>Suspend-StorageGroupCopy "SERVER1\First Storage Group" -StandbyMachine
SERVER2 

[PS] C:\>Update-StorageGroupCopy "SERVER1\First Storage Group" -StandbyMachine
SERVER2 -DeleteExistingFiles 

[PS] C:\>Resume-StorageGroupCopy "SERVER1\First Storage Group" -StandbyMachine
SERVER2

In the near future I’d like to nail down some more specific scenarios, causes and remedies, but for now I’ll just summarise by saying if you are having backup problems in an environment where LCR or SCR are deployed, look to your event logs for MSExchangeRepl errors and consider reseeding your replicas to get your backups behaving normally again.