551
Released: July 28, 2003 Instructor Guide Exchange Server 2003 Admin Troubleshooting

Exchange Server Admin Troubleshooting

  • Upload
    chawahi

  • View
    5.254

  • Download
    3

Embed Size (px)

Citation preview

Page 1: Exchange Server Admin Troubleshooting

Released: July 28, 2003

Instructor Guide

Exchange Server 2003 Admin Troubleshooting

Page 2: Exchange Server Admin Troubleshooting

ii Exchange Server 2003 Admin Troubleshooting

Contents

Module 1: Setup Changes Document Overview................................................................................................1 Setup Changes .........................................................................................................2 Setup Architectural Changes ...................................................................................3 Setup Actions Require New Active Directory Permissions ....................................7 New Setup Prerequisite Checks:............................................................................21 Lab 1.1: Finding renamed, moved, or deleted groups ...........................................26 Cluster-related prerequisite checks........................................................................31 Exchange System Manager-only installation prerequisites ...................................33 2000 to 2003 Setup and Upgrade Scenarios blocked.............................................36 New Features/Components in Setup:.....................................................................39 Setup Changes .......................................................................................................44 Security improvements to setup: ...........................................................................49 Troubleshooting Exchange Server 2003 setup failures: ........................................53 General Log Flow..................................................................................................57 Lab 1.2: Logparser and examination of progress logs...........................................68 Lab 1.3: Applying troubleshooting concepts .........................................................70 Appendix A: Answers ...........................................................................................74 Acknowledgments .................................................................................................76 Module 2: New Administrative Features New Administrative features in System Manager and Active Directory.................1 New Exchange System Manager Features...............................................................2 New Diagnostics......................................................................................................7 Public Folder Store and Public Folder Tree...........................................................15 Exchange HTTP Virtual Server.............................................................................20 Mailbox..................................................................................................................26 Module 3: Move Mailbox Wizard Dependencies...........................................................................................................4 General Overview....................................................................................................4 Troubleshooting.....................................................................................................10 Tools ......................................................................................................................18 Labs .......................................................................................................................20 Acknowledgments .................................................................................................23 Module 4: Troubleshooting the Recovery Storage Group What is the Recovery Storage Group?.....................................................................3 Creating a Recovery Storage Group ........................................................................7 Adding a Mailbox Store to the Recovery Storage Group ......................................10 Active Directory Attributes ...................................................................................17 Lab 4.1: Create a Recovery Storage Group and review the Active Directory attributes ................................................................................................................20 Overriding the Recovery Storage Group ...............................................................25 Restoring the data ..................................................................................................27 Recovering Exchange 2000 Server Mailbox Stores ..............................................33 Exmerge.................................................................................................................35

Page 3: Exchange Server Admin Troubleshooting

Known issues with Exmerge .................................................................................48 Exercise 2: Recovery Storage Group Scenario 1...................................................50 Exercise 3: Recovery Storage Group Scenario 2...................................................54 Details for Exercise 3: Disaster recovery after a Store Crash using the Recovery Storage Group........................................................................................................55 Acknowledgments .................................................................................................58 Module 5: Clustering Resource Dependencies...........................................................................................1 Cluster Service Account Permissions......................................................................5 MsExchange_NodeState..........................................................................................9 DNS registration/Kerberos ....................................................................................12 AntiAffinityClassNames .......................................................................................16 Mount Point Drives ...............................................................................................22 Creating an Exchange Virtual Server ....................................................................33 Upgrading an Exchange Virtual Server to Exchange 2003 ...................................56 Removing an Exchange Virtual Server .................................................................64 Lab 5.1 : Clustering ...............................................................................................89 Module 6: Deployment Tools and ADC Tools Overview .................................................................................................................1 Lesson 1: Deployment Tools...................................................................................2 Lesson 2: ADC Tools ............................................................................................39 Lab 6.1 Exchange 2003 ADC replication featuring Deployment and ADC Tools71 Appendix A: Sample log files ..............................................................................86 Acknowledgments ...............................................................................................107 Module 7: Exchange Directory Components Overview .................................................................................................................1 Lesson 1: Changes to the ADC................................................................................2 Lab 7.1: Troubleshooting LDAP queries...............................................................12 Lesson 2: DSAccess Usage and troubleshooting...................................................16 Lesson 3: Changes in DSAccess ...........................................................................28 Lab 7.2: Exomatic tool ..........................................................................................37 Lesson 4: Other Directory Changes.......................................................................38 Lab 7.3: Per-Attribute change troubleshooting......................................................59 Lab 7.4: Post-Setup and SRS replication troubleshooting.....................................59 Appendix A: Numeric registry keys used by DSAccess and their values .............68 Acknowledgments .................................................................................................70 Module 8: Cross-Forest/Multi-Forest Overview .................................................................................................................1 Lesson 1: The Cross-forest Specification ................................................................2 MIIS Components ...................................................................................................8 Lab 8.1: Getting to know MIIS 2003 and GAL Sync Management Agent ..........26 Lesson 3: Cross-Forest SMTP Mailflow ..............................................................32 Lesson 4: InterOrg Public Folder Replication .......................................................35 Lab 8.2: Cross Forest Practice.............................................................................47 Appendix A GAL Sync Log and Error Messages .................................................50 Appendix B: GAL Sync Mapping Types ..............................................................54 Appendix C: GAL Sync Provisioning Rules .........................................................67 Acknowledgments .................................................................................................76

Page 4: Exchange Server Admin Troubleshooting
Page 5: Exchange Server Admin Troubleshooting
Page 6: Exchange Server Admin Troubleshooting

Contents

Document Overview ............................................................................................... 1 Setup Changes......................................................................................................... 2 Setup Architectural Changes................................................................................... 3 Setup Actions Require New Active Directory Permissions .................................... 7 New Setup Prerequisite Checks: ........................................................................... 21 Lab 1.1: Finding renamed, moved, or deleted groups........................................... 26 Cluster-related prerequisite checks ....................................................................... 31 Exchange System Manager-only installation prerequisites................................... 33 2000 to 2003 Setup and Upgrade Scenarios blocked ............................................ 36 New Features/Components in Setup: .................................................................... 39 Setup Changes....................................................................................................... 44 Security improvements to setup: ........................................................................... 49 Troubleshooting Exchange Server 2003 setup failures:........................................ 53 General Log Flow ................................................................................................. 57 Lab 1.2: Logparser and examination of progress logs .......................................... 68 Lab 1.3: Applying troubleshooting concepts ........................................................ 70 Appendix A: Answers ........................................................................................... 74 Acknowledgments................................................................................................. 76

Module 1: Setup Changes

Page 7: Exchange Server Admin Troubleshooting

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 8: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 1

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Document Overview

This module discusses differences in the setup process between Microsoft Exchange 2000 Server and Microsoft Exchange Server 2003. In addition to discussing bug-level changes, students will focus on troubleshooting the Exchange Server setup progress logs.

Topic 1 Setup changes from Exchange 2000 Server

Topic 2 Troubleshooting Exchange Server 2003 setup

Topic 3 Learning measure/Labs

Prerequisites Experience with installing Exchange 2000 into Exchange Server 5.5 sites. Experience with creating an Exchange Virtual Server (EVS) on Windows

2000 clusters

Page 9: Exchange Server Admin Troubleshooting

2 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Setup Changes

This topic discusses differences between the setup architecture from the last product, as well as new features and work items in the setup process. Those accustomed to supporting Exchange 2000 Server will expect some of the same product features and behaviors to exist in Exchange 2003. The goal of this topic is to cover any “gotchas” in differences between the two products that would otherwise cause difficulty in support.

Page 10: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 3

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Setup Architectural Changes

In Exchange Server 5.5, many customers established administration models so that Exchange administrators were able to administer only Exchange, and domain administrators handled almost everything else. Yet Exchange 2000 Server required the installer to be given blanket permissions to the enterprise forest and the Exchange Server 5.5 directory – to the dismay of many companies migrating from, or coexisting with, Exchange Server 5.5. In order to separate these roles once more, the product group established the following “Full Administrative Group Administrator” setup changes so that network/domain admin roles could be separated from Exchange administrator roles. These changes were so extensive that the process flow of setup is nearly re-architected.

Setup /forestprep creates a placeholder object When Exchange 2003 setup is run explicitly in ForestPrep mode (using the /forestprep switch), and there is no existing Exchange organizational object within the configuration naming context, setup will create a “temporary” organization with a hard-coded name. (That name is a GUID: “{335A1087-5131-4D45-BE3E-3C6C7F76F5EC}”.) Setup can delegate the first Exchange administrator on this object, create the Exchange configuration underneath it, and so on. At a later time, when setup is run to install the first server in the organization – by someone who is an Exchange administrator – setup can rename the existing placeholder object, either to a user-specified name or to match the name of an Exchange 5.5 organization. The final naming is decided by the answer to the “Installation Type” screen. Improving upon Exchange 2000 setup, the organization name deferral was designed so that

• Administrators are not forced to make the organization name decision during forestprep.

• Enterprise/schema admins are not forced to be given Exchange Server 5.5 admin site permissions to run forestprep.

Conversely, Exchange 2003 installers (who are admins of an Exchange 5.5 site) are not required to have enterprise/schema admin permissions when later installing the first Exchange Server 2003 machine. Installers are also no longer

Page 11: Exchange Server Admin Troubleshooting

4 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

required to have the Active Directory Connector (ADC) installed when running forestprep.

Troubleshooting temporary org object creation: Should there be any problems creating this GUID, it will most likely be a permissions issue, caught at the pre-requisite stage with a descriptive error message. If this is the case, one should ensure that the logged-on user has full control privileges on the cn=Microsoft Exchange,cn=services,cn=configuration,dc=<forest root DN> container. (By default, Enterprise Admins has this permission). Although it is possible to manually-create the temporary org object, it is neither recommended nor supported since it would also require manually creating scores of child objects and setting their permissions appropriately.

“Installation Type” prompt moves to server setup mode In Exchange 2000 Server, running setup with the /forestprep switch whilst in a clean forest (where there is no Exchange organization object) would always prompt the installer with the “Installation Type” screen. This page of the setup wizard would ask if a new Exchange organization needed to be created or if setup should join an existing Exchange 5.5 organization. Therefore, Exchange 2000 setup /forestprep not only extended the schema; for the 5.5-joining case, it would also connect and perform intensive sync operations (via a temporary config CA) with the Exchange 5.5 directory. This is why with Exchange 2000 setup, the platinum-osmium synchronizer ran twice: once during explicit forestprep and again during normal server setup. (The exception is if only setup.exe is run without switches, thereby setting the forestprep component to “Install” mode so that the platinum-osmium synchronizer runs only once.)

Page 12: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 5

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Figure 1.1: The “Installation Type” prompt is no longer shown during /forestprep mode.

In Exchange Server 2003, the “Installation Type” prompt has moved to the server setup mode. That is, the prompt will only occur when running setup.exe without switches, and it will only occur once: when the first Exchange Server 2003 machine is being installed into a forest with no pre-existing Exchange organization object. (The Exchange organization object is located at (cn=<orgname>,cn=Microsoft Exchange, cn=services, cn=configuration, dc=<dn of the forest root>.) If the installer chooses to create a new organization, the placeholder orgname is renamed to whatever the installer desires. If the installer chooses the Exchange 5.5 coexistence option, the temporary orgname is renamed to match the Exchange 5.5 organization name. In Exchange Server 2003, the 5.5 (Osmium) synchronization process with Active Directory will occur only once, so only a permanent config CA comes into existence. (i.e. no temporary config CA will exist). Table 1.1 outlines the different states of the organizational object that can exist in Active Directory:

Page 13: Exchange Server Admin Troubleshooting

6 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Setup Action/ Detected State

setup /ForestPrep setup (install a server)

No organization object

Create temporary org

Ask user for org type/name; create org

Temporary organization object {335A1087-5131-4D45-BE3E-

3C6C7F76F5EC}

N/A Ask user for org type/name; rename temporary org

Named organization object (exists in place of GUID)

N/A N/A

Table 1.1: Creation flow for Exchange Organization object in Active Directory This architectural change does not affect manual creation of first Administrative Group through System Manager (per 215930). However, when customers launch Exchange System Manager to manually create their administrative group, they might be surprised to see the GUID, {335A1087-5131-4D45-BE3E-3C6C7F76F5EC}.

Note: When the temporary organization object exists, you must not run Exchange 2000 Server setup. Although it does not get blocked through a pre-requisite check, later in the setup process the Exchange 2000 Server setup wizard does not understand the GUID organization object, and the installation is likely to fail catastrophically.

Server Setup mode no longer stamps organization-level permissions Previously, the Exchange 2000 Server SETUP program would re-stamp Exchange Organization permissions on each server install. The drawback was that this action would overwrite any custom changes to the permissions structure, such as removing the permission for all users to create top level public folders. So if a customer kept having his/her top-level permissions reset, this was a perceived security risk.

In Exchange Server 2003, the setup process has changed so that it will only stamp default permissions on the Exchange Organization object once (on the first server install/upgrade) and will not re-stamp permissions for subsequent installations. Although this resolves the workaround for security, the previous behavior was a useful support tool for quickly fixing customers who have inappropriately modified their Active Directory permissions on containers that cause operational problems in Exchange. A typical problem would be a paranoid administrator removing required access control lists (ACLs) on various objects underneath the “Microsoft Exchange” container. So in order to correct the problem, or to revert back to Exchange 2000 Server settings, one must now manually correct the Active Directory permissions by applying the permissions listed in Table 1.4 under the section entitled “New per-object permissions changes during setup.” If the customer does not mind that the security settings revert back to the Exchange 2000 Server configuration, then run Exchange 2000 setup to “join” a new Exchange 2000 server object to the existing Exchange 2003 organization.

Page 14: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 7

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Setup Actions Require New Active Directory Permissions

Because there are several setup modes and component options, setup will require different combinations of Active Directory permissions, depending upon the detected topology. For example, setup operations dealing with a Site Replication Service (SRS) still require Exchange Full Administrator at the Organization level. Table 1.2 outlines the required permissions of the person being logged on.

Setup Action Active Directory Permission(s) required Install first Exchange 2003 server in a domain Exchange Full Administrator at Organization level

Install first Exchange 2003 server into a 5.5 site (SRS-enable) Exchange Full Administrator at Organization level

Uninstall/reinstall Exchange 2003 with an SRS Exchange Full Administrator at Organization level

First “ForestPrep” in forest [with schema update] or

ADC’s Setup when older schema is detected or

ADC’s setup used with the explicit “schemaonly” switch

Enterprise Admin [+ Schema Admin]

Subsequent “ForestPrep” Exchange Full Administrator at Organization level

“DomainPrep” Domain Administrator

Install a server to have first instance of a Groupwise/Lotus Notes connector Exchange Full Administrator at Organization level

Install, maintain or remove server containing Key Management Server Enterprise Admin

Install, maintain or remove server with SRS enabled Exchange Full Administrator at Organization level

Install additional server (non-SRSs, clusters EVSs) Exchange Full Administrator at Admin Group level + machine account added to Domain Servers group

Run maintenance mode on any server (except Key Management Server or SRS enabled) Exchange Full Administrator at Admin Group level

Remove a server (no SRS present) Exchange Full Administrator at Admin Group level + remove machine account from Domain Servers group

Page 15: Exchange Server Admin Troubleshooting

8 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

after setup

Remove last server in org Exchange Full Administrator at Organization level

Apply service pack Exchange Administrator at Admin Group level

Table 1.2: Setup Matrix Several of the above actions require “Exchange Full Administrator” at the organizational level. Although it is possible to manually create and grant Exchange Administrator-like permissions through ADSI Edit, it is not recommended because the specific combination of permissions and inherited rights settings are not easy to set, and setting “Full Control” on the organization object would be overkill. The recommended methods for granting Exchange Full Administrator at the org level are to either:

Rerun /forestprep so that the Exchange setup wizard will prompt for an additional account to be granted Org permissions, or

Use the Exchange System Manager’s delegation wizard by right-clicking on the top-most organization object.

The proper method of granting Exchange Full Administrator at the Admin Group level is to launch Exchange System Manager’s delegation wizard by right-clicking on an Administrative Group name.

In Exchange 2000, you needed to be a full admin at the organization level to install, maintain, or remove any server. Unfortunately, customers desired to deploy with well-separated admin groups and delegate administrators on those administrative groups who would be able to handle routine tasks -- like installing and maintaining servers. (This had been the 5.5 model, of course.)

Many efforts from our customer experience team and customers, themselves, expended considerable ingenuity in trying to find ways to work around this requirement in Exchange 2000 setup, but all in vain -- even if you managed to bypass the permission prerequisite, setup would still fail, since it refreshed org-level settings and permissions during every server install; and without org-level rights, you wouldn't have access to those objects.

In Exchange 2003, full admin-group level admins can now install, maintain, and remove most servers within their own administrative group. However, there are still exceptions: You still need full org admin permissions when installing the SRS or first Exchange 2003 server into a domain. In the latter case, the first server installed into any given domain must set the access control entries (ACEs) for that domain’s "Exchange Domain Servers" group on the org-level object, which means that setup needs full org permissions.

Page 16: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 9

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

New Per-Object Permissions Changes During Setup: In addition to new permissions requirements, Exchange 2003 setup modifies Access Control Entries that were set by Exchange 2000. Tables 1.5-1.6 describe these Active Directory object-level access control list (ACL) changes, and tables 1.7-1.8 describe the NTFS-ACL changes. However, interpreting the tables requires a key:

Key to Reading the tables Permissions that are listed in the tables with a double strike-through are removed by Exchange 2003 setup. They represent permissions that were set in Exchange 2000, but which have since been deprecated from the security model.

Each table begins with the distinguished name (also known as DN) of the object it applies to. After that, the table lists when the right is stamped: during the ForestPrep phase, while installing a server, etc.

In some cases, the ACL is not stamped on the usual property (ntSecurityDescriptor), but on some other property – e.g., “msExchMailboxSecurityDescriptor”. The directory service, of course, cannot enforce security that is not specified in the NT security descriptor; in most cases, these ACLs will be picked up and replicated to store ACLs on appropriate objects by the store service. There is, unfortunately, no tool for viewing these ACLs as anything other than raw binary data.

The columns of the table are as follows:

Account The security principal granted or denied the permissions.

A Checked if this is an allow ACE.

D Checked if this is a deny ACE. Allow and Deny are mutually exclusive.

I Checked if this ACE inherits to child objects.

Right The permissions allowed or denied. Extended rights are given in italics.

On Property/Applies To In some cases, the permission applies only to a given property, property set, or object class; if so, that is specified here.

Reason The reason this permission is required.

Table 1.3: Legend for columns of charts 1.5-1.9 The rights are generally listed in the table by the names used on the ADSIEdit Security property page, under the “Advanced” view, on the “View/Edit” tab. The ADSIEdit Security property page lists a much more condensed view of the rights. LDP.exe displays the access mask directly, as a numerical value. The setup code refers to the rights by predefined constants.

The following table summarizes the relationships between these values:

Page 17: Exchange Server Admin Troubleshooting

10 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

ADSIEdit Advanced Page,

Binary value

ADSIEdit Summary Page

View/Edit Tab

#define

(“Mask” in LDP)

WRITE_OWNER |

WRITE_DAC |

READ_CONTROL |

DELETE |

ACTRL_DS_CONTROL_ACCESS |

ACTRL_DS_LIST_OBJECT |

ACTRL_DS_DELETE_TREE |

ACTRL_DS_WRITE_PROP |

ACTRL_DS_READ_PROP |

ACTRL_DS_SELF |

ACTRL_DS_LIST |

ACTRL_DS_DELETE_CHILD |

Full Control

Full Control

ACTRL_DS_CREATE_CHILD

0x000F01FF

List Contents +

ACTRL_DS_LIST |

Read All Properties +

ACTRL_DS_READ_PROP |

Read

Read Permissions

READ_CONTROL

0x00020014

Write All Properties +

ACTRL_DS_WRITE_PROP | Write

All Validated Writes

ACTRL_DS_SELF

0x00000028

List Contents

ACTRL_DS_LIST 0x00000004

Read All Properties

ACTRL_DS_READ_PROP 0x00000010

Write All Properties

ACTRL_DS_WRITE_PROP 0x00000020

Delete DELETE 0x00010000

Delete Subtree

ACTRL_DS_DELETE_TREE 0x00000040

Read Permissions

READ_CONTROL 0x00020000

Modify Permissions

WRITE_DAC 0x00040000

Modify Owner

WRITE_OWNER 0x00080000

All Validated

ACTRL_DS_SELF 0x00000008

Page 18: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 11

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Writes

All Extended Rights

ACTRL_DS_CONTROL_ACCESS 0x00000100

Create All Child Objects

Create All Child Objects

ACTRL_DS_CREATE_CHILD 0x00000001

Delete All Child Objects

Delete All Child Objects

ACTRL_DS_DELETE_CHILD 0x00000002

ACTRL_DS_LIST_OBJECT 0x00000080

Table 1.4: Bit values for tables

Permissions Modified On Active Directory Objects in the Configuration Naming Context Microsoft Exchange Container cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=<domain>

Account A D I Right On Property/Applies To Reason

During ForestPrep phase

List Contents Authenticated Users X

Read All Properties

Allow DomainPrep to read Full Org Admins

Designated Admin Account X X Full Control Allow Full Org Admin to administer org

During server install

Read Permissions

Read All Properties

Exchange Domain Servers X X

List Contents

Allow Exchange servers to read config info

During ADC setup

Exchange Services X X Full Control Allow ADC servers to create/delete objects to keep Exchange config up to date

ADC Connection Agreement Container cn=Active Directory Connections,cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=<domain>

Account A D I Right On Property/Applies To Reason

During server install

Exchange Domain Servers X X Full Control

Organization Container cn=<org>,cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=<domain>

Account A D I Right On Property/Applies To Reason

During ForestPrep phase

Read All Properties Authenticated Users X

ACTRL_DS_LIST_OBJECT

Allow DomainPrep to read Full Org Admins

Page 19: Exchange Server Admin Troubleshooting

12 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Designated admin account X X Send As Exchange admins are not allowed to open mailboxes

Designated admin account X X Receive As Exchange admins are not allowed to open mailboxes

During server install

Enterprise Admins X X Send As NT admins are not allowed to open mailboxes

Enterprise Admins X X Receive As NT admins are not allowed to open mailboxes

Domain Admins of root domain X X Send As NT admins are not allowed to open mailboxes

Domain Admins of root domain X X Receive As NT admins are not allowed to open mailboxes

Everyone X X Create top-level public folder

Everyone X X Create public folder

Everyone X X Create named properties in the information store

Read Permissions Applies to object class:

Read All Properties msExchPrivateMDB

List Contents

Everyone X X

ACTRL_DS_LIST_OBJECT

Read Permissions Applies to object class:

Read All Properties msExchPublicMDB

List Contents

Everyone X X

ACTRL_DS_LIST_OBJECT

Read Permissions Applies to object class:

Read All Properties mTA

List Contents

Everyone X X

ACTRL_DS_LIST_OBJECT

ANONYMOUS LOGON X X Create top-level public folder

ANONYMOUS LOGON X X Create public folder In Windows 2003 “Everyone” no longer includes “Anonymous Logon,” so we must grant those rights explicitly

ANONYMOUS LOGON X X Create named properties in the information store

Read Permissions Applies to object class:

Read All Properties msExchPrivateMDB

List Contents

ANONYMOUS LOGON X X

ACTRL_DS_LIST_OBJECT

Read Permissions Applies to object class:

Read All Properties msExchPublicMDB

List Contents

ANONYMOUS LOGON X X

ACTRL_DS_LIST_OBJECT

Read Permissions Applies to object class:

Read All Properties mTA

List Contents

ANONYMOUS LOGON X X

ACTRL_DS_LIST_OBJECT

Exchange Domain Servers X X All Extended Rights

Exchange Domain Servers X X Create All Child Objects

Exchange Domain Servers X X Write Property Property Set: Maintain mail-

Page 20: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 13

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Public Information enabled config objects (e.g., MAD.EXE)

Property Set: Exchange Domain Servers X X Write Property

Personal Information

Maintain mail-enabled config objects (e.g., MAD.EXE)

Applies to object class: Exchange Domain Servers X X Full Control

siteAddressing

When enabling an SRS (ACE is removed when SRS is disabled)

Create All Child Objects

Delete All Child Objects

MACHINE$ X X

ACTRL_DS_LIST_OBJECT

SRS must be able to create/delete admin groups

Address Lists Container cn=Address Lists Container,cn=<org>,cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=<domain>

Account A D I Right On Property/Applies To Reason

During server install

Authenticated Users X X List Contents

Addressing Container cn=Addressing,cn=<org>,cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=<domain>

Account A D I Right On Property/Applies To Reason

During server install

List Contents

Read All Properties

Authenticated Users X X

Read Permissions

Recipient Update Services Container cn=Recipient Update Services,cn=Address Lists Container,cn=<org>,cn=Microsoft Exchange,cn=Services,cn=Configuration...

Account A D I Right On Property/Applies To Reason

During server install

Exchange Domain Servers X X Full Control

Administrative Group cn=<admin group>,cn=Administrative Groups,cn=<org>,cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=<domain>

Account A D I Right On Property/Applies To Reason

During server install (set on attribute msExchPFDefaultAdminACL)

Authenticated Users X X Create public folder

Default TLH cn=Public Folders,cn=All Folder Hierarchies,cn=<admin group>,cn=Administrative Groups,cn=<org>,cn=Microsoft Exchange...

Account A D I Right On Property/Applies To Reason

During server install (set on attribute msExchPFDefaultAdminACL)

Authenticated Users X X Create public folder

Page 21: Exchange Server Admin Troubleshooting

14 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Connections Container cn=Connections,cn=<routing group>,cn=Routing Groups,cn=<admin group>,cn=Administrative Groups,cn=<org>...

Account A D I Right On Property/Applies To Reason

During server install

Exchange Domain Servers X X Full Control

Servers Container cn=Servers,cn=<admin group>,cn=Administrative Groups,cn=<org>,cn=Microsoft Exchange,cn=Services...

Account A D I Right On Property/Applies To Reason

During server install, or during Exchange 2003 setup /ForestPrep

Exchange Domain Servers X X Receive As No server needs to read mail except on its own store

During server install (ACEs defined in schema defaultSecurityDescriptor)

Authenticated Users X List Contents

Server Object cn=<server>,cn=Servers,cn=<admin group>,cn=Administrative Groups,cn=<org>,cn=Microsoft Exchange,cn=Services...

Account A D I Right On Property/Applies To Reason

During server install (if the server is NOT a cluster Virtual Machine)

MACHINE$ X X Full Control Server must be able to maintain its own config

During server install (if the server IS a cluster Virtual Machine)

NODE1$

NODE2$

etc...

X X Full Control Every node in a cluster that owns an EVS must be able to maintain the EVS config

Exchange Domain Servers X X Full Control EVS must be able to maintain its own config, but setup can’t tell which specific server to grant control to

During server install (ACEs defined in schema defaultSecurityDescriptor)

Authenticated Users X Read Properties

When EDSLOCK script is run; ACE is REMOVED by Titanium ForestPrep

Exchange Domain Servers X X Receive As No server needs to read mail except on its own stores

Protocols Container cn=Protocols,cn=<server>,cn=Servers,cn=<admin group>,cn=Administrative Groups,cn=<org>,cn=Microsoft Exchange...

Account A D I Right On Property/Applies To Reason

During server install

Everyone X X List Contents

Everyone X X Read metabase properties

System Attendant Object cn=Microsoft System Attendant,cn=<server>,cn=Servers,cn=<admin group>,cn=Administrative Groups,cn=<org>...

Page 22: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 15

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Account A D I Right On Property/Applies To Reason

During server install (set on attribute msExchMailboxSecurityDescriptor)

Read Permissions

fsdspermUserSendAs

LocalSystem X X

fsdspermUserMailboxOwner

Read Permissions

fsdspermUserSendAs

Exchange Domain Servers X X

fsdspermUserMailboxOwner

5.5 Service Account Read Permissions

(if given) fsdspermUserSendAs

X X

fsdspermUserMailboxOwner

MTA Object cn=Microsoft MTA,cn=<server>,cn=Servers,cn=<admin group>,cn=Administrative Groups,cn=<org>...

Account A D I Right On Property/Applies To Reason

During server install or when enabling an SRS

5.5 Service Account

(if given)

X X Send As Required to send/receive mail from 5.5 servers

5.5 Service Account

(if given)

X X Receive As Required to send/receive mail from 5.5 servers

Table 1.5: Configuration Naming Context permission changes

Permissions Modified On Active Directory Objects in Domain Naming Context Domain Container dc=<domain>

Account A D I Right On Property/Applies To Reason

During DomainPrep phase

Property Set: Exchange Enterprise Servers X X Write Property

Public Information

Maintain mail-enabled user attributes

Property Set: Exchange Enterprise Servers X X Write Property

Personal Information

Maintain mail-enabled user attributes

On property: Exchange Enterprise Servers X X Write Property

groupType

On property: Exchange Enterprise Servers X X Write Property

displayName

Exchange Enterprise Servers X Manage Replication Topology Allow Recipient Update Service to track replication changes

Exchange Enterprise Servers X X List Contents Duplicates permissions granted to “Pre-Windows

Page 23: Exchange Server Admin Troubleshooting

16 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

2000 Compatible Access” group

Exchange Enterprise Servers X Read Permissions “

Read Permissions Applies to object class:

Read All Properties user

List Contents

Exchange Enterprise Servers X X

ACTRL_DS_LIST_OBJECT

Read Permissions Applies to object class:

Read All Properties group

List Contents

Exchange Enterprise Servers X X

ACTRL_DS_LIST_OBJECT

Applies to object class: Exchange Enterprise Servers X X Modify Permissions

group

Maintain ACLs for groups with Hidden membership

During DomainPrep phase (if running against Whistler schema)

Read Permissions Applies to object class:

Read All Properties InetOrgPerson

List Contents

Exchange Enterprise Servers X X

ACTRL_DS_LIST_OBJECT

We need same perms on InetOrgPersons as on Users

Domain Proxy Container cn=Microsoft Exchange System Objects,dc=<domain>

Account A D I Right On Property/Applies To Reason

During DomainPrep phase

Exchange Enterprise Servers X X Full Control Add/delete/modify proxy objects

Exchange Domain Servers X X Full Control Add/delete/modify proxy objects

Authenticated Users X X Read Permissions Allow access to PF objects

Authenticated Users X X Read Property garbageCollPeriod Allow access to PF objects

Authenticated Users X X Read Property adminDisplayName Allow access to PF objects

Authenticated Users X X Read Property modifyTimeStamp Allow access to PF objects

During DomainPrep (ACEs defined in schema defaultSecurityDescriptor)

Read Permissions

Read All Properties

List Contents

Authenticated Users X

ACTRL_DS_LIST_OBJECT

Set by the Recipient Update Service

All delegated org-level and admin-group level Full Admins

X X Full Control

Read Permissions

List Contents

All Validated Writes

All delegated org-level and admin-group level Admins

X X

Read All Properties

Page 24: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 17

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Write All Properties

Create All Child Objects

Delete All Child Objects

Read Permissions

Read All Properties

List Contents

All delegated org-level and admin-group level View-Only Admins

X X

ACTRL_DS_LIST_OBJECT

AdminSDHolder Container cn=AdminSDHolder,cn=System,dc=<domain>

Account A D I Right On Property/Applies To Reason

During DomainPrep phase

Read Property Property Set: Exchange Enterprise Servers X X

Write Property Public Information

This ACL is applied to users with domain admin rights

Read Property Property Set: Exchange Enterprise Servers X X

Write Property Personal Information

Read Property On property: Exchange Enterprise Servers X X

Write Property displayName

Exchange Enterprise Servers X X List Contents “

Pre-Windows 2000 Compatible Access Group cn=Pre-Windows 2000 Compatible Access,cn=Builtin,dc=<domain>

Account A D I Right On Property/Applies To Reason

During DomainPrep phase

On property: Exchange Enterprise Servers X X Write Property

member

The Recipient Update Service must add all Exchange Domain Servers groups to every domains’ Pre-W2K group

Exchange Enterprise Servers Group cn=Exchange Enterprise Servers,cn=Users,dc=<domain>

Account A D I Right On Property/Applies To Reason

During DomainPrep phase

All existing org-level Full Admins X Full Control Admins running setup must be able to add/remove machine accounts from group

Exchange Enterprise Servers X Full Control

Set by the Recipient Update Service

Page 25: Exchange Server Admin Troubleshooting

18 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

All delegated org-level Full Admins X X Full Control

Exchange Domain Servers Group cn=Exchange Domain Servers,cn=Users,dc=<domain>

Account A D I Right On Property/Applies To Reason

During DomainPrep phase

All existing org-level Full Admins X Full Control Admins running setup must be able to add/remove machine accounts from group

Exchange Enterprise Servers X Full Control

Set by the Recipient Update Service

All delegated org-level Full Admins X X Full Control

Table 1.6: Domain Naming Context permission changes

Page 26: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 19

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

File System Permissions Modified During Setup When setting ACLs in the file system, setup generally first examines the ACL to see if there are any explicit (i.e., non-inherited) ACEs on the folder. If there are, then setup assumes that one of two cases applies:

1. Setup has previously stamped ACLs on this folder, and there is no need to do so again.

2. An administrator has manually adjusted permissions to his or her liking, and setup should not overwrite those settings.

The effect is that, in the default case, setup stamps file system permissions on a clean install, but does not modify them on reinstalls.

Installation Directory C:\Program Files\Exchsrvr (by default; may be chosen during setup)

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

For this folder, setup reads the ACL from the “Program Files” folder and duplicates it; the permissions shown below are those that exist by default on Program Files. Authenticated Users X X Read & Execute

Server Operators X X Modify

Administrators X X Full Control

CREATOR OWNER X X Full Control

TERMINAL SERVER USER X X Modify

SYSTEM X X Full Control

Mailroot Directory ...\Exchsrvr\Mailroot

Account A D I Right On Property/Applies To Reason

During server install

Everyone X X Full Control

ANONYMOUS LOGON X X Full Control

Exchweb Directory ...\Exchsrvr\exchweb

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

Authenticated Users X X Read

Exchweb\bin Directory ...\Exchsrvr\exchweb\bin

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

Authenticated Users X X Read & Execute

Exchweb\bin\auth Directory ...\Exchsrvr\exchweb\bin\auth

Page 27: Exchange Server Admin Troubleshooting

20 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

ANONYMOUS LOGON X X Read

Exchweb\img Directory ...\Exchsrvr\exchweb\img

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

ANONYMOUS LOGON X X Read

Exchweb\controls Directory ...\Exchsrvr\exchweb\controls

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

ANONYMOUS LOGON X X Read

Exchweb\cabs Directory ...\Exchsrvr\exchweb\cabs

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

ANONYMOUS LOGON X X Read

Exchweb\views Directory ...\Exchsrvr\exchweb\views

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

ANONYMOUS LOGON X X Read

Exchweb\help Directory ...\Exchsrvr\exchweb\help

Account A D I Right On Property/Applies To Reason

During server install (if no pre-existing explicit ACEs)

ANONYMOUS LOGON X X Read

Table 1.7: NTFS changes to Installation Directory and Subdirectories

Page 28: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 21

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

New Setup Prerequisite Checks:

Marker Checks During server setup, if the installer chooses to join an Exchange 5.5 site, additional marker checks are enforced. This means that setup will check to see if the deployment tools have been executed as far as step 2 in the ADC Tools snap-in. (That step should have written the completion marker, ADCUserCheck, to the description attribute of cn=Microsoft Exchange, cn=services, cn=configuration, dc=<forest root DN> object in the configuration naming context.) If the marker exists, setup will continue; otherwise, the following error is displayed:

To ensure that an admin reads and performs the preparatory steps using the deployment and ADC tools, rather than attempting to bypass the process blindly, setup enforces this check when the first Exchange 2003 joins an admin group containing any Exchange 5.5 directories (which include SRSs). Marker checks are not performed on additional installs into mixed AGs where the 1st Exchange 2003 has already joined an Exchange 5.5 site.

Note that the string “- Error: ADC Tools were not run in your organization.” Is a variable string (%s) which can be replaced if other conditions are satisfied. For example, if the ADCUserCheck marker exists, but other markers do not, then the error message follows this format:

“Setup detected one or more of the following conditions that may affect your Exchange deployment. Microsoft recommends resolving these conditions before continuing this installation:\r\n%s\r\nPlease refer to your Exchange

Page 29: Exchange Server Admin Troubleshooting

22 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Server 2003 Deployment Tools documentation on your CD for information about correcting this problem.”

Where the %S string indicates that something has not yet finished replicating, or has not been run from the deployment tools. Specifically, depending upon the status of the other completion markers, ADCObjectCheck and PubfoldCheck the %s string will change accordingly. However, the failure to pass ADCObjectCheck and PubfoldCheck markers will only warn the installer of that specific problem, but will not prevent setup from continuing as in the ADCUserCheck case.

If the customer is halted with the blocking error message, use ADSI Edit or LDP.exe to view the description attribute. This is where any of the three completion markers may exist. If ADCUserCheck is present, check to see if its timestamp is older than two weeks. Note that if you’re not using credentials of a person who has full exchange org permissions, you may not be able to see this attribute. If you do not have the marker present, there are three ways to populate it:

Manual entry through ADSIEdit Running exdeploy.exe from command line, using the /adcusercheck switch.

(If 5.5-Active Directory objects are not in sync, this method will populate the %S string with a warning indicating that objects have not replicated. However, setup will not be blocked.)

Running ADC Tools’ Step 2 button, or Step 4 (Verify button)

Although setup enforces the prerequisites, it is a non-setup “glue” DLL (originally from deployment tools) that passes the prerequisite result back to setup. Walksdll.dll is the “glue” because it is a wrapper that is called not only by setup, but also from the deployment tools. Since setup shares the wrapper, you may find that the DLL exists in two places on the CD: within the setup\i386 folder, and also within \support\exdeploy. Upon launching setup, the markers are checked using this logic:

Troubleshooting Tip

Page 30: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 23

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

References to “Greenfield scenario” or “Pure TI or pure TI/PT” in the diagram above means that Pure Exchange 2003 or Exchange 2000/2003 admin groups do not require marker checks.

Note

Page 31: Exchange Server Admin Troubleshooting

24 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Server prerequisites for server FQDN == any SMTP domain on a recipient policy In the UNIX world, and especially at university-run UNIX mail servers, it was common practice to host users whose e-mail addresses contained domain names matching the fully-qualified domain names of the mail servers themselves. (For example, the server whose FQDN was mailserver.univ.edu hosted a mailbox with SMTP address [email protected]). When these customers deployed Exchange 2000 in the same fashion, mail flow would become inoperable between Exchange 2000 servers. This behavior is by design per KB Article Q288175. This new prerequisite prevents Exchange 2003 from being installed into an existing organization when the FQDN of the server (listed on the networkAddress/ncacn_ip_tcp attribute) matches any SMTP addresses on the recipient policy.

Setup checks if domain prepped GC is available for DSAccess Setup will iterate through all GCs in local and adjacent sites, checking if their domains have been domain prepped. If no suitable GC has been found with the SACL, setup will not continue.

Setup checks for stopped SRS On upgrades or reinstalls of machines that are supposed to have their site-replication service enabled, setup performs a prerequisite check to ensure this directory service is running so that setup can write to it, if necessary. To manually determine if a site replicate service is supposed to be enabled on a machine, look for the existence of the “Microsoft DSA” object underneath the server object in Active Directory. (CN=Microsoft DSA,CN=<servername>,CN=Servers,CN=<Admin Group Name>,CN=Administrative Groups,CN=<Organization Name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<DN of forest root>). If such an object exists, setup will perform this prerequisite check and will block from installing unless the “Microsoft Exchange Site Replication Service” is set to either “Manual” or “Automatic” and that the service is started.

Setup will not install until all ADC services are upgraded to Exchange 2003 version This check ensures that no Windows 2000 ADC services exist. The reason behind this is because Windows 2000 ADCs, when running public folder connection agreements, have been known to cause corruption on public folders. This prerequisite is checked on each run of Exchange 2003 setup.exe when no switches are specified. Although it may not seem necessary to execute this prerequisite check when the org is native mode, existing ADC installations will be checked, nevertheless.

Setup checks for Exchange Domain Servers/Exchange Enterprise Servers Customers that renamed or moved their Exchange Domain Servers or Exchange Enterprise Servers groups outside of their default “Users” containers caused various problems with Exchange 2000. To prevent this from happening, Exchange Server 2003’s setup has two improvements:

The setup /domainprep modifies the description attribute of these groups to include the string “DO NOT move or rename.”

Page 32: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 25

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

A prerequisite was added to normal setup (not domainprep) to check for the renaming or movement of these groups. This check only applies to subsequent (not the first) server installations, or re-installs of the first Exchange 2003 server, in the forest. However, this prerequisite check cannot run during setup /domainprep because there is no way for domain admins (lacking Exchange permissions) to query the Recipient Update Service object for the domain, to which the objectGUIDs or SIDs of Exchange Domain Servers/Exchange Enterprise Servers groups are linked. Consequently, rerunning setup /domainprep will still cause the 0X80072030 error, which is documented in KB Article 818470.

Page 33: Exchange Server Admin Troubleshooting

26 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Lab 1.1: Finding renamed, moved, or deleted groups

If the customer has a very large directory that is difficult to search visually, you can search for the objectGuid of the Exchange Domain Servers/Exchange Enterprise Servers groups by following these steps:

1. Power-on the virtual Machine “Solo” (Administrator/password) 3. Ask a lab partner or instructor to hide either Exchange Domain Servers

group or Exchange Enterprise Servers group in one of the organizational units (OUs), and rename it. This will simulate supporting a large OU hierarchy with thousands of users, where it would be painstakingly difficult to determine where the object was moved.

4. If you were to run setup at this time, you would receive the prerequisite message blocking setup.

5. Use ADSI Edit or a similar tool to view the properties of the domain Recipient Update Service object (CN=Recipient Update Service (STANDALONE),CN=Recipient Update Services,CN=Address Lists Container,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<forest root DN>)

6. Locate the following attributes on the domain Recipient Update Service, since they contain the GUIDs for the Exchange Enterprise Servers and Exchange Domain Servers groups, respectively: msExchDomainLocalGroupGuid, msExchDomainGlobalGroupGuid. Copy the values they contain. Let us assume that msExchDomainLocalGroupGuid was {1E519285-D987-42C8-BE35-8DC57F85F270}

7. Convert the GUIDs from string to Hex format. In the above example, {1E519285-D987-42C8-BE35-8DC57F85F270} becomes \85\92\51\1E\87\D9\C8\42\BE\35\8D\C5\7F\85\F2\70

8. In Active Directory Users and computers, right-click on the domain object, and choose FIND. Do a custom search, and select the advanced tab.

9. Enter an LDAP query similar to the following: objectGUID=\85\92\51\1E\87\D9\C8\42\BE\35\8D\C5\7F\85\F2\70 Where “\85\92\51\1E\87\D9\C8\42\BE\35\8D\C5\7F\85\F2\70” would be replaced by the values you converted in step 7.

10. Hit the FIND button, and you will be presented with the new name of the group (if it has been renamed).

11. To determine the OU in which it resides, choose the “object” property sheet to determine its changed location. If there are no objects found, this means the group(s) have been deleted. Rerunning domain prep recreates these groups.

After completing this exercise, students should be able to recognize the

following: 1) Does the msexchdomainlocalgroupguid correspond to the Exchange

Domain Servers group? (Y/N) 2) Recognize patter of reversed bits when converting GUIDs from string

format to hexadecimal string

Page 34: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 27

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

3) How easy it is to perform custom LDAP queries without any special tools installed.

Page 35: Exchange Server Admin Troubleshooting

28 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

New Setup Prerequisite Checks (2 of 2)

Disasterrecovery: Setup checks for existence of server object Running /disasterrecovery is useless if there is not a corresponding server object in Active Directory. This is because the purpose of a disasterrecovery setup is to restore a server based on its configuration stored in Active Directory. If a customer attempts this setup mode without first having created the server from a prior installation, Exchange setup assumes that the installation must be brand new, and therefore provides a prerequisite failure message indicating that they must abandon this switch and run setup normally to create the server object.

Setup checks for W3SVC to be installed Since Windows 2003 no longer installs the World Wide Web Publishing service by default with IIS, Exchange setup must ensure that it is installed through this prerequisite.

Setup checks for correct ASP.Net and .Net Framework versions Because there can be various versions of ASP.Net/.Net framework installed from different packages, setup ensures that 1.1.4322 is installed, or else a prerequisite is fired.

Setup now checks for 5.5 permissions on SRS upgrade/reinstall This prerequisite prevents a delegated Exchange administrator from setting “upgrade” or “reinstall” actions on the messaging and collaboration component when the admin does not have permissions to the SRS. This is an improvement over Exchange 2000 setup, where if the prerequisite check didn’t fire, customers would encounter this error message later in setup: “Could not bind to the Microsoft Exchange Directory server Name_of_Ex2000_server. You do not have the permissions required to complete the operation.” The installer should have at least “permissions admin” role in Exchange 5.5 organization, site, and configuration containers.

Page 36: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 29

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Exchange Domain Servers group now added to “Pre-Windows 2000 Compatible Access” group Due to how the Exchange Enterprise servers group was only a domain local group in Exchange 2000 implementations, servers would not always get all the read access they needed in multi-domain forests. ACLs and attributes couldn’t be read, leading to various potential issues. As a workaround, Exchange Server 2003 setup adds the Exchange Domain servers group to the Pre-Windows 2000 Compatible Access built-in group. This is performed during the domain prep mode of setup. Additionally, an access control entry is added to the Pre-Windows 2000 compatible access group, allowing the local domain’s Exchange Enterprise Servers group to modify the membership. So when a Recipient Update Service is designated for a domain, it will add all other domains’ Exchange Domain Servers groups to the Pre-Windows 2000 Compatible Access group.

Prerequisites for Windows 2000 SP3 GC’s Exchange Server 2003 requires that it only uses domain controllers that are Windows 2000 SP3 or later. To enforce this requirement, setup uses the process (below) to search for well-versioned domain controllers, or else halt the deployment.

Page 37: Exchange Server Admin Troubleshooting

30 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Page 38: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 31

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Cluster-related prerequisite checks

Required Resource States When manipulating the Exchange Virtual Server (EVS), here are the scenarios and prerequisites:

INSTALLING EVS: - network name resource must be online

REMOVING EVS: - network name resource must be online

- System Attendant resource must be offline

UPGRADING EVS: - network name resource must be online

- System Attendant resource must be offline

Setup blocks removal of cluster node if EVS is running on that node Previously, Exchange 2000 Server administrators were able to uninstall the last node of a cluster, without first removing the virtual server/system attendant resource. Neglecting the proper removal of the EVS would orphan the virtual server object in Active Directory. To prevent the orphaning, a new prerequisite in Exchange 2003 will determine if the node is a possible owner for any Exchange virtual server resources and halts if they are.

Setup /disasterrecovery is now blocked on cluster nodes The disasterrecovery switch was never supported on Exchange 2000 Server clusters. However, this was a support hit to Microsoft Product Support Services, as customers would continually attempt to run setup.exe /disasterrecovery on cluster nodes and fail catastrophically with 0x80005000 errors on the Information Store service. To prevent this from happening, a prerequisite check

Page 39: Exchange Server Admin Troubleshooting

32 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

blocks this setup switch if the machine is a node of a cluster, thus customers may only run normal setup. Additionally, the normal setup routine on a cluster node no longer presents a message indicating that setup will install the cluster-aware version, whereas the Exchange 2000 setup version would popup that dialog.

Clusters now require Kerberos-enabled Network Name resource A new requirement of Exchange Server 2003 clusters is for the network name resource to be Kerberos-friendly. If this prerequisite fails on a Windows 2003 server, ensure that from within cluster administrator, the network name resource properties shows that the Kerberos setting enabled. If the cluster is Windows 2000, look for the RequireKerberos property by using cluster.exe:

Cluster.exe res <resource name> /priv

If the listing shows that RequireKerberos is 0, you must set it to 1 by

1. Ensuring the network name resource offline 2. Type the following at a command prompt:

Cluster.exe res <displayname_of_network_name_resource> /priv RequireKerberos=1:DWORD

Preventing Exchange 2003 clusters from being the first non-legacy server in a pure Exchange 5.5 site Non-legacy in this heading refers to Exchange 2000 (6.0) or Exchange 2003 (6.5) servers. Previously, customers could run setup and join Exchange 2000 clusters as the first 6.x servers in Exchange 5.5 sites. However, this was an unsupportable situation because the SRS is supposed to reside on the very first 6.x server in a 5.5 site. Since the SRS is not a clusterable component, customers painstakingly needed to uninstall their cluster, install a non-clustered Exchange 2000 server, and then redeploy their cluster. To prevent this scenario for Exchange Server 2003, setup currently prevents the installation of the first Exchange 2003 server joining an Exchange Server 5.5 org on a cluster by graying out "Join an existing Exchange 5.5 Organization" choice on the “Installation type” page. Once a mixed site (having an SRS) has been established, the creation of the System Attendant resource allows the EVS to join the mixed site.

Clusters require Q329938 hotfix or Windows 2000 SP4 With the new Kerberos authentication requirements for clusters, a prerequisite check scans the operating system version of the target server. If the operating system is Windows 2000 SP3, then setup will look in the registry to see if the key "HKLM\SOFTWARE\Microsoft\Updates\Windows 2000\SP4\Q329938" is present. If the operating system is below or above Windows 2000 SP3, this prerequisite passes. (In the case that it is below SP3, another prerequisite will fire a warning about the operating system service pack level.)

Page 40: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 33

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Exchange System Manager-only installation prerequisites

For both Exchange 2000 Server and Exchange Server 2003, the component selection screen allows for the granularity to install the System Management Components without the messaging and collaboration components. This is what is called an “Exchange System Management-only” install mode, and Exchange administrators use this mode to administer their Exchange servers from their workstations.

Previously for Exchange 2000 System Manager-only installs, customers were only required to have the Windows 2000 administration package (which includes Active Directory Users and Computers) to be installed onto their Windows 2000 Professional edition operating systems. On Windows XP operating systems, Exchange 2000 System Manager could not be installed without hotfix q815529. This was due to the fact that the Exchange 2000 setup engine, using a prerequisite check, searched for the GUID of the Windows administration package. When the Exchange 2000 Server setup engine was built, it only knew to check for the Windows 2000, and not Windows 2003, administration package.

For a successful Exchange Server 2003 System Manager-only mode installation, the following operating system prerequisites must be met:

Windows XP SP1: Internet Information Services Snap-In component (In Add/Remove

Programs) SMTP Service component (In Add/Remove Programs) SMTP Service should be disabled after service is installed (reason for

disabling is that SMTP snap-in is only needed, and not the service itself. Additionally, leaving SMTP service running leaves open another possible point of attack)

WWW Service (SMTP requires this) should be disabled after service is installed (reason being that it is a security threat)

Windows 2003 AdminPack (provides NNTP snap-in and Active Directory Users and Computers snap-in)

Page 41: Exchange Server Admin Troubleshooting

34 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Windows XP SP2 (planned): Internet Information Services Snap-In component (In Add/Remove

Programs) SMTP snap-in is now provided as part of IIS Manager component Windows 2003 AdminPack (provides NNTP snap-in and Active Directory

Users and Computers snap-in)

Windows 2003 Internet Information Services Manager component (In Add/Remove

Programs)

Windows 2000 SP3 Professional Internet Information Services Snap-In component (In Add/Remove

Programs) Windows 2000 Server AdminPack (provides SMTP snap-in, NNTP snap-in

and Active Directory Users and Computers snap-in)

Windows 2000 SP3 Server Internet Information Services Snap-In component (In Add/Remove

Programs) SMTP Service component (In Add/Remove Programs) Should disable service after installed (only need the SMTP snap-in) NNTP Service component (In Add/Remove Programs) Should disable service after installed (only need the NNTP snap-in)

Applies to all scenarios: Setup prerequisites against installing admin-only on a workstation that does

not belong to a domain Exchange Server 2003 Forestprep required before installing System

Manager Although the Exchange Server 2003 System Manager may manage any Exchange Server 5.5 and Exchange 2000 Server servers in the organization, it may not manage the following components that were retired in Exchange Server 2003:

Instant Messaging service Key Management Server Chat Service Lotus cc:Mail Connector MS-Mail Connector Microsoft Mail Directory Synchronization Connector Schedule+ Free/Busy Connector

Therefore, some customers will need to retain a full Exchange 2000 server or Exchange 2000 Exchange System Manager-only installation in their organization in order to manage the services above.

Exchange 2000 System Manager may only be used to view (read-only)

Exchange System Manager Compatibility

Note

Page 42: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 35

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

property sheets on Exchange 2003 servers.

Page 43: Exchange Server Admin Troubleshooting

36 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

2000 to 2003 Setup and Upgrade Scenarios blocked

Attempts to upgrade in the following situations are blocked:

If the server does not have Exchange 2000 Server SP3 installed or Windows 2000 SP3 installed, then the prerequisite check fails. For clusters, setup will remotely check each node to ensure other nodes in the cluster are at the proper service pack level.

Attempts to in-place upgrade Exchange 2000 Server SP2 to Exchange Server 2003 are blocked This prerequisite fires unless Exchange 2000 Server SP3 or greater are installed.

In-place upgrades from English Exchange 2000 Server to Korean, Chinese, or any other double-byte character set (DBCS) of Exchange Server 2003 are blocked if the Groupwise connector is already installed. This is because the Groupwise connector in Exchange Server 2003 does not support Japanese character sets or any DBCSs. Once the Groupwise connector is uninstalled, an English version of Exchange 2000 may then be in-place upgraded to a DBCS version of Exchange 2003.

In-place upgrade of Exchange 2000 back-end server is blocked if there exists an Exchange 2000 front-end in the same Administrative group. Beta versions are not checked; the prerequisite only enforces the major version (6.5 versus 6.0) and not the minor versions (6944 versus 6895). The reason for pr-requisite is because front-ends must be upgraded first, in order to prevent various problems with Outlook Web Access. This block is only enforced when both front-end and back-end are in the same administrative group which means there is still an unchecked scenario: When front-ends and back-ends exist in different administrative groups, then customers will not encounter the prerequisite block, so Outlook Web Access users will experience error messages throughout the web interface (for example, script errors in Internet Explorer).

Exchange 2000 servers with Key Management Server component Chat server component, or Instant Messaging server components are blocked from being upgraded unless those components are removed from the Exchange 2000 server. This is because Exchange Server 2003 has dropped support for those components, and would not be able to upgrade these components properly. Key

Page 44: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 37

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Management Server administration is being replaced by Windows 2003’s Certificate Server feature. Instant Messaging server and Chat server functionality can be replaced by the features within the Microsoft Office Real-Time Communications Server 2003 product.

When upgrading an Exchange 2000 Server cluster to Exchange Server 2003, the Microsoft Distributed Transaction Coordinator (MS DTC) resource is required. In most cases, Exchange 2000 Server setup would have created that resource. However, there are some scenarios in which Windows 2000 did not allow Exchange 2000 Server setup to create the MS DTC resource, and so a blocking prerequisite message is displayed when upgrading to Exchange Server 2003 setup. To create the MS DTC resource on a Windows 2000 cluster, simply type Comclust.exe on each node of the cluster, and the MS DTC resource is added automatically (205796). Note: You should not use cluster administrator to create the MS DTC resource manually.

Setup Blocks for upgrades or installs In-place upgrade from Exchange Server 5.5 is blocked This stops customers from attempting an in-place upgrade from Exchange Server 5.5 to Exchange server 2003, as this path is unsupported.

Setup blocked if Windows 2003 POP3 service is installed A new feature of the Windows 2003 operating system is a lightweight Post Office Protocol (POP3) server service. Due to port conflicts and questionable supportability of two mail systems on a single machine, Exchange Server 2003 setup prevents the two from coexisting, by means of a prerequisite check: “-You must remove the Windows POP3 Service component in order for Setup to continue.” To remove this Windows 2003 feature to bypass the prerequisite check, go to Add/Remove Programs, then Add/Remove Windows Components, and select the details of the “E-mail services” category.

If MIS is installed, a prerequisite blocks install/upgrade To prevent collisions between different versions of mobility components, this prerequisite ensures that Mobile Information Server doesn’t already exist on the machine being setup with Exchange 2003. If this prerequisite is fired when the customer has already removed Mobile Information Server, check for the existence of the registry key "Software\\Microsoft\\Exchange\\DMI\\EventMessageFile" and remove it if it exists. Furthermore, the prerequisite will fire if the Mobile Information Server Exchange Event sink is registered in “HKLM/Software/Classes/Wnotify.MoExSink” Although Mobile Information Server and Exchange Server 2003 may not reside on the same machine, there is no problem with these two products coexisting within the same forest on different servers.

Setup disallows /disasterrecovery to convert an EVS to a standalone Setup checks if the Exchange server object was previously an Exchange virtual server. If it was, and the installer attempts to run /disasterrecovery on a non-clustered machine with the same name as the EVS’s old network name resource, setup will halt. In the past, Exchange 2000 would not check for this, and some servers would be installed without message transfer agents (MTAs). If a new, standalone server must be installed using the same name as the old EVS, then one must (a) delete the Exchange server object from the

Page 45: Exchange Server Admin Troubleshooting

38 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

configuration Naming Context, then (b) rerun setup without the /disasterrecovery switch.

Setup prevents any Exchange 2003 SKUs from installing on a Windows 2003 Web Edition server (212624) The Windows 2003 Web Edition is targeted as a low-cost Web-application server. Ideally, this edition of Windows 2003 would be a fitting platform for installing Outlook Web Access front-end servers. Nevertheless, the Web Edition of Windows is not a full-featured server product containing all the component services on which Exchange 2003 depends. Therefore Exchange 2003 setup is designed to check specifically to ensure that customers do not attempt to install on this scaled-down operating system. Although customers might feel that the Web Edition would be ideal for Exchange 2003 front-end servers, technical reasons preclude this from happening at this time.

In place upgrades of front-end servers from Enterprise to Standard edition You cannot in-place upgrade an Exchange 2000 Server enterprise front-end to become an Exchange Server 2003 standard front-end.

On a single machine, there is no direct upgrade path from Exchange 5.5 to Exchange 2003. To reduce the test matrix, and to prevent Exchange 5.5 disaster-recovery scenarios on failed upgrades, a decision was made to bar in-place upgrades from 5.x. Instead, customers should install Exchange Server 2003 on separate servers, and then use the move-mailbox method.

If you recall that Exchange 2000 may not be installed on Windows 2003 server, you may also deduce that upgrading the operating system where Exchange 2000 server resides on Windows 2000 may not work either. The upgrade path for the operating system would require that one must first in-place upgrade the Exchange version to Exchange Server 2003 before in-place upgrading the operating system. Otherwise, the Exchange 2000 Server setup will be prevented, since Windows 2003 server will proactively prevent its setup routine.

Page 46: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 39

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

New Features/Components in Setup:

Clusters create computer objects One new, notable feature when running on a cluster is that the Exchange 2003 System Attendant resource indirectly creates a computer object in the Computers OU of the domain. This may sound odd, since previously for Windows 2000 clusters, the existence of a pre-existing computer object would prevent the network name resource from starting, and these two events would be logged:

Event ID 1052 Source: Cluster Service Clussvc Category: 2056 Type: Stop Description: Cluster network name resource "<ResourceName>" could not be brought online because the name could not be added to the system.

Event ID 1069 Source: Cluster Service Clussvc Category: 4 Type: Stop Description: Cluster resource "<ResourceName>"

With Exchange 2003 installed on Windows 2000, the existence of the network name will not prevent the Exchange virtual server’s network name from starting. This change is a side-affect of the resource’s private property requiring Kerberos support, because when the system attendant is created, it sets the requirekerberos property on the network name resource. With Windows 2003 (regardless of whether Exchange 2003 is installed), the default behavior is such that ANY network name resource will automatically create a corresponding computer object.

Page 47: Exchange Server Admin Troubleshooting

40 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Outlook Mobile Access This is the mobility component that is automatically installed with every Exchange 2003 installation, but disabled by default. (To enable it, use Exchange System Manager | Global Settings | Mobile Services | Properties) Similar to Outlook Web Access, the installer has no choice here, as it is not a selectable component on the setup wizard’s component selection screen. Outlook Mobile Access is the first Exchange Server 2003 component written in C#, and therefore needs ASP.NET 1.1 installed.

Some parts of Outlook Mobile Access are installed through scripts (.INS files within the Exchange setup source directory). For example, omabrowseinstall.exe is called by a script (.INS) file. But since omabrowseinstall.exe is a separate module from the main setup program, it is not tied to the rich error/diagnostic reporting provided by the backoffice setup engine. Thus, we cannot get much information from it if errors are encountered. At the time of this writing, “OmaBrowseInstall.exe counters /create” is called by a script, and may sometimes fail if other software is installed on the same machine. To avoid a catastrophic failure that would have caused customers disasterrecovery measures in the upgrade scenario, the setup team has updated the counter creation to ignore failures during counter installation, but unfortunately a warning is not logged to the application log during installation.

The methods to determine (post-setup) if there was a problem with installing Outlook Mobile Access counters is to

a) Examine the application event log for an exit code in the setup progress.log, such as the one below:

[09:36:52] ++++ Starting interpreter on file z:\titanium\rtm\6944.1\server\rtl\usa\setup\i386\exchange\browse.ins ++++ -- ID:31258 -- [09:36:52] Interpreting line <CreateProcessSafe:D:\Program Files\Exchsrvr\OMA\Browse\bin;"D:\Program Files\Exchsrvr\OMA\Browse\bin\OmaBrowseInstall.exe" counters / create;180000> -- ID:31259 -- [09:36:52] Process created ... waiting (180000) [09:36:58] Ignoring exit code fffffffffd

b) Wait for the following event ID to be generated when a mobile user attempts to access their mailbox through Outlook Mobile Access:

Event Type: Warning Event Source: MSExchangeOMA Event Category: (1000) Event ID: 1101 Date: 6/12/2003 Time: 2:33:58 PM User: N/A Computer: BTSLABFE Description: Outlook(R) Mobile Access Browse Application could not initialize its performance monitor counters because of the following error: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.

Page 48: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 41

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

.NET Framework, ASP.NET 1.1 and ASP.NET Device Update-2 These components are installed automatically if running Exchange Server 2003 setup on a Windows 2000 server lacking the .NET Framework. However, since ASP.NET is a component of Windows 2003, it must be installed as a component. One may verify if .NET Framework is installed on a machine by ensuring it is on the list of installed applications: Start/Control Panel/Add or Remove Programs/Change or Remove Programs/Microsoft .NET Framework - <version>. However, there is no information about version of the ASP.NET there.

It is possible to have a several .NET Frameworks installed on the same computer. There are some problems with multiple versions of ASP.NET and .NET Framework installed on the same server, with servers promoted from Windows 2000 to Windows 2003 and from member servers to domain controllers. (Also, a possible problem occurs when customers inappropriately upgraded beta versions of Windows 2003 to the release version). For each operating system, we hope the latest versions of .NET Framework and ASP.NET are used. The latest version of enabled-ASP.NET is used on Windows 2003. Device Update-2 is a standalone application and it is installed by Exchange Server 2003 Setup. They are installed using the following files:

...\i386\exchange\oma\browse\dotnetfx.exe

...\i386\exchange\oma\browse\dupdate.exe

How does it work? Windows 2000+SP3: Exchange Server 2003 Setup installs the .NET Framework version 1.1.4322.557 if ASP.NET 1.1.0000 or above has not been installed before. ASP.Net is always enabled on Windows 2000 if .NET Framework is installed there. If the version of .NET Framework installed is below 1.1.4322.557, Setup will automatically uninstall it using this series of commands:

"MsiExec.exe /X{DF420000-A5AB-407C-92FF-D1D4B709B8F9} /q" "MsiExec.exe /X{8542DFF7-5CAB-4424-AAF7-BFEB3104D8AC} /q" "MsiExec.exe /X{C9913503-1500-4454-94CD-365ADC1BB9B9} /q"

Thus, you may possibly use these commands to clean up a server you suspect has incompatible .NET Framework versions. Then, to install ASP.NET as a component:

1. Start/Control Panel/Add or Remove Programs/Add/Remove Windows Components/Application Server/ASP.NET.

2. Check the check box and install it. 3. To enable ASP.NET: 4. Run Internet Information Services (IIS) Manager; 5. Navigate to "Web Service Extensions". 6. Select "ASP.NET v1.1.4322" and click on “Allow”

Troubleshooting ASP.NET errors: There are a few typical scenarios where ASP.NET and Device Update-2 are not installed or/and configured properly. Before troubleshooting, a simple verification test on Windows 2003 is to see if the correct version is installed is to:

Page 49: Exchange Server Admin Troubleshooting

42 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

1. Run Internet Information Services (IIS) Manager; 2. Navigate to "Web Service Extensions". 3. Ensure that "ASP.NET v1.1.4322" exists, and that its extension status is set

to “Allowed” You should also ensure the existence of this directory:

%windir%\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll

Previous Cases with problems: Engineers may want to review these previous beta-build problems as possible troubleshooting methods for post-release problems:

Scenario: .NET Framework versions < 4322 (for example: 3629) + ASP.NET 1.0 (for example: 1.0.3709) -> 3678 or above.

Problem: ASP.NET, installed as a .NET component is the same (3709) after upgrade. Uninstalling and re-installing ASP.NET does not help, since the same ASP.NET 3709 is installed.

How to fix:

1. Uninstall ASP.NET 2. IInstall .NET Framework 4322 (from

...\i386\exchange\oma\browse\dotnetfx.exe) 3. Enable ASP.NET through Add/Remove Programs.

Scenario: Windows 2000+SP3 + ASP.NET upgrading to Windows 2003 build 3678

or

Member Server (both Windows 2000+SP3 and .NET) + ASP.NET promoted to a Domain Controller (bug 214215). It always happens if Jupiter/Autosetup/Topoweb is used.

Problem: If you upgrade to Windows 2003 and dcpromo.exe, all the ACLs in the %windir% folders are reset. As a result, the ACLs do become misconfigured for some ASP.NET folders.

To fix, here is the recommended path:

1. Run the aspnet_regiis.exe with the -i switch on the server: 2. Browse to "<%windir%>\Microsoft.NET\Framework\<version of the

framework>" (for example, C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322);

3. Run "aspnet_regiis.exe -i". Not recommended, but this helps:

Manually: uninstall ASP.NET and install it back, or 1) Add user "NETWORK_SERVICE" with "Full Control" to the folder "<%windir%>\Microsoft.NET\Framework\<version of the framework>\Temporary ASP.NET Files" (for example,

Page 50: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 43

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files).

Page 51: Exchange Server Admin Troubleshooting

44 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Setup Changes

Improvements to the progress log Since the setup progress log is not localized (appears only in English), it was difficult for non English-speaking customers and support engineers to examine the progress log file. Exchange Server 2003’s setup progress logs now contain "-- ID:xxxxx --" tags after entries that are localized string resources. A new logparser (with the ability to parse these string IDs) is now available.

M: drive removed In Exchange 2000 Server, the M: drive caused a lot of problems, as it was easily exposed through explorer and customers adopted the impression that mailbox and public folder permissions could be manipulated through this file system interface. Furthermore, file-based antivirus and backup software would manipulate objects in the M: drive, accidentally corrupting messages or setting the archive bits on them such that they would not be properly accessible. Therefore, Exchange Server 2003 no longer presents the Exchange Installable File System as drive letter M. Reclaiming the M: drive is possible by enabling this registry parameter, although it is strongly discouraged:

Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EXIFS\Parameters\ Parameter: DriveLetter Type: String Value: M

The M: drive should only be exposed for gaining access to non-MAPI file data (such as editing an .asp that executes out of the store). You should not share out portions of the M: drive for SMB user access (instead, you should use Web Folders).

Enabling the Installable File System drive will lead to prompts for Note

Page 52: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 45

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

reboots during future upgrades.

Setup creates configuration Connection Agreement with fully qualified domain names When setup installs the first Exchange 2003 server into a pure Exchange 5.5 site, the Site Replication Service (SRS) is always added to the Exchange 2003 server. Each SRS requires a configuration Connection Agreement to replicate configuration naming context data between Active Directory and Exchange Server 5.5. This replication data comprises of new or changed server properties, connectors, address generator settings, etc. Creation of the Configuration connection agreement by Exchange 2000 setup would populate the endpoints - msexchserver1networkname and msexchserver2networkname (domain controller and Exchange 5.5 directory service, respectively) - to NetBIOS server names. In Exchange Server 2003, setup will configure the configuration connection agreement (located on the ‘Connections’ property sheet) to fully-qualified domain names.

How this affects Microsoft Product Support Services: When troubleshooting why server settings, connectors, or site addressing properties are not replicating between Exchange Server 5.5 and Active Directory, we will need to check if DNS name resolution is non-problematic. Although deployment tools prevents this before setup is run, customers may misconfigure their DNS settings post-setup in a way such that an endpoint is not resolvable through DNS. If this is the case, the configuration connection agreeements will not be able to synchronize the endpoints. To correct, either enter the NetBIOS names of server endpoints or ensure DNS has an updated host record for the Exchange 5.5 or domain controller servers.

Launch.exe replaced by setup.exe at CD root Tthe splash screen that autoruns upon inserting a product CD, was able to be launched manually by executing launch.exe in Exchange 2000. In Exchange 2003, launch.exe has been dropped so that setup.exe at the root will spawn the splash screen.. However, the root setup.exe should not be confused with the real setup wizard, located in \setup\i386\setup.exe. If the root setup.exe is run with any switches, then rather than spawning the splash screen, the root setup.exe will automatically redirect the switches to the real setup.

Files will be recopied upon reinstall In Exchange 2000 and earlier builds of Exchange Server 2003, the reinstall process would always skip copying files from source media if an installed file has the same or greater file version. Due to how there could be some corruption on the target file, Exchange 2003 changes the process by ensuring source files are re-copied and that installed files are overwritten if their timestamps are not greater than their source. A minor disadvantage in copying more files, is the amount of time it takes to reinstall a server increases.

Files renamed during setup Files on the CD are not guaranteed to have the same filename that will appear on an installed server. Since all files on the CD need unique names (which is done when the Build Team builds the drop), setup will rename them back to their original filenames during the install. For example, language-specific Outlook Mobile Access files are renamed during setup because, on the CD, these files reside in different language directories. Once setup installs them, they rename those DLLs to a non-language-specific name.

Page 53: Exchange Server Admin Troubleshooting

46 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

If a file cannot be read from the installation media, or if you ever need to manually replace an installed file that has already been installed, consult the following list for special files when searching from new media; you may need to manually copy the file to disk and then rename it to emulate the setup process.

List of files renamed during setup: Exwfmsg.dll will be renamed during the installation. For example, the source file

Setup\i386\exchange\res\<Code_Page>\exwfmsg.dll.<Code_Page>

will lose its language-codepage extension when the file-copy phase of setup renames it to Exwfmsg.dll

The following Outlook Web Access Controls files will be renamed in the CD:

Setup\i386\exchange\exchweb\<Version_WMTEMPLATEST>\Controls\blank.htm.0 Setup\i386\exchange\exchweb\<Version_WMTEMPLATEST>\Controls\dlg_ANR.js.0 Setup\i386\exchange\exchweb\<Version_WMTEMPLATEST>\Controls\dlg_GAL.css.0 Setup\i386\exchange\exchweb\<Version_WMTEMPLATEST>\Controls\dlg_GAL.js.0 Setup\i386\exchange\exchweb\<Version_WMTEMPLATEST>\Controls\dls_Recurrence.js.0

The following Outlook Web Access Themes files will be renamed on the CD:

Setup\i386\exchange\exchweb\themes\0\*.0 Setup\i386\exchange\exchweb\themes\1\*.1 Setup\i386\exchange\exchweb\themes\2\*.2 Setup\i386\exchange\exchweb\themes\3\*.3 Setup\i386\exchange\exchweb\themes\4\*.4

The Help files of WebClient will be renamed in the drop:

Setup\i386\exchange\exchweb\HELP\<Client_Lang>\ie3\*.ie3.<Client_Lang> Setup\i386\exchange\exchweb\HELP\<Client_Lang>\ie3\basics\*.ie3.basics.<Client_Lang> Setup\i386\exchange\exchweb\HELP\<Client_Lang>\ie3\gif\*.ie3.gif.<Client_Lang> Setup\i386\exchange\exchweb\HELP\<Client_Lang>\ie5\*.ie5.<Client_Lang> Setup\i386\exchange\exchweb\HELP\<Client_Lang>\ie5\basics\*.ie5.basics.<Client_Lang>

The following files from Setup\i386\Exchange\conndata\dxanotes folder will be renamed:

Troubleshooting Tip

Page 54: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 47

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Amap.tlb.0 Mapmex.tlb.0

All the Outlook Mobile Access resources files will be renamed in the CD:

Setup\i386\Exchange\Oma\Browse\bin\<Culture_Lang>\*.<Culture_Lang>

The CDO.dll in Setup\i386\Exchange\Oma\Browse\bin will be renamed to CDO.dll.0.

Logoff.asp has been renamed in Setup\i386\exchange\exchweb\bin\<Client_Lang> folder to logoff.asp.<Client_Lang>

Logon.asp has been renamed in Setup\i386\exchange\exchweb\bin\auth\<Client_Lang> folder to logon.asp.<Client_Lang>

ChooseDC switch The Exchange 2003 SETUP.EXE program has a new switch; /choosedc. This allows you to specify which domain controller should be using during the installation process for reading and writing directory information. Its command-line syntax is:

Setup.exe /choosedc name_of_DC

This switch may be used in conjunction with other switches, such as /domainprep. However, the user-specified domain controller must be in the same domain as the installing server (ScPRQ_ChosenDCMustBeInSameDomain is the prerequisite check which enforces same-domain domain controller).

One good use of this switch is when you are installing multiple Exchange servers simultaneously into the same domain (e.g. test lab or training room) and have multiple domain controllers; you can hone all installations to use the same domain controller. By doing this, when SETUP adds the computer account to the 'Exchange Domain Servers' group, the change will only take place on a single domain controller and replicate out. Without this switch, setup behaves like in Exchange 2000 Server, in that it talks to multiple domain controllers and then you get replication clashes - the net result being that some Exchange servers fail to start their services after installation.

If you need to confirm that a customer typed the /chooseDC switch properly, search the progress log for this string (without the servername) to verify:

User has specified a DC; m_strDC = "CDCS00"

Otherwise, if /chooseDC switch contains bad syntax, such as a colon (for example setup.exe /choosedc:CDCS00) or if the switch isn’t used at all, this entry is written to the progress log:

Tip

Page 55: Exchange Server Admin Troubleshooting

48 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

No user-specified DC; setup has chosen m_strDC = "CDCS00"

If the /chooseDC switch is used, but does not have a server name after the switch, then this window appears:

Service startup state retained through reinstalls or upgrades If an Exchange 2000 service, such as MSExchangeMTA, is disabled before the upgrade to Exchange 2003, the upgrade process will leave the Exchange 2003 MTA disabled. This is a major change from Exchange 2000, as reinstalls and upgrades would reset their component services to a default state (typically automatic).

Changes to IIS6 after upgrading to Windows 2003 On Windows 2000, Exchange 2003 services run in the same IIS application pool as any other services. So just like Exchange 2000, this is potentially less stable because any problems in any one IIS ISAPI application can cause problems with all of InetInfo – including Exchange services. So in Windows 2003, IIS6 apps run in their own dedicated application pools, though not by default when upgraded from IIS5. Therefore, immediately after an operating system upgrade, the Exchange metabase update service run by the system attendant creates two dedicated app pools – one for DAV, Web forms, exadmin; and one for spellcheck for the web client. In addition to creating the metabase keys for application protection, Exchange benefits from all of the security improvements included in the worker process isolation mode. However the rest of IIS6 still runs in IIS5Compat mode until the administrator switches to worker-process isolation mode.

Changes to Maximum Hop Count during install (225648) The maximum hop count on SMTP virtual server instances has changed from 15 to 30. For new server installations, a value of 30 is always set. For upgrades and reinstalls, the existing value is checked; if it is 15, the value is automatically changed to 30. If the existing value is something other than 15, no change is applied.

Page 56: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 49

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Security improvements to setup:

Relaxed Permissions on cluster service accounts Previously in Exchange 2000, anybody who wanted to install a cluster needed to ensure that the cluster service account was granted Full Exchange Administrator rights at the organizational level. The permissions have been relaxed in Exchange 2003, so the Windows 2003 Cluster Admin account does not need to have any rights on the Exchange org level. The cluster service account just needs to be a local admin on each node of the cluster. Though Windows 2000 cluster service accounts still needs permissions to Active Directory, but they are not needed on the org level unless the EVS is the first server in the org.

Windows 2000 Cluster Service Account: Local Administrator on each Node in the cluster

Exchange Full Administrator on org object if other Exchange 2000 clusters with same cluster service account remain in org

Windows 2003 Cluster Service Account Local Administrator on each Node

No permissions required on org

If you are upgrading, you can remove the Exchange Full Admin right from the cluster service account once you have upgraded all your servers to Exchange 2003.

Exchange Domain Servers group no longer granted Receive-As right during DomainPrep In Exchange 2000 Server, a rogue Exchange Admin could access mailboxes in remote domains when using the Computer account (LocalSystem account

Page 57: Exchange Server Admin Troubleshooting

50 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

context). The workaround was to use the “EDSLOCK.VBS” script to lockdown permissions. In Exchange Server 2003, the servers are locked-down by default by removing the computer account’s ability to read mail from servers other than its own. In addition to the domainprep change, setup now does these things during forestprep and first server install:

EXCHANGE 2003 ForestPrep: Enumerates all existing "Exchange Domain Servers" groups and all admin groups; set a "Deny-Receive-As" ACE for each Exchange Domain Servers group on each admin group's "Servers" container [this happens every time ForestPrep is run].

Enumerates all server objects, searches the ACL of each one for any "Deny-an Exchange Domain Servers group-Receive-As" ACEs (which were set by the EDSLOCK script) and removes them. This happens only the first time EXCHANGE 2003 ForestPrep is run. The heuristics bit on the Microsoft Exchange object is set to “6” to represent this change. (In Exchange 2000, it was only set to “2” after forestprep).

EXCHANGE 2003 server install: If installing the first Exchange 2003 server in a new domain, setup enumerate all admin groups, and sets a "Deny-Receive-As" ACE for the new Exchange Domain Server group on each admin group's "Servers" container.

For every install/reinstall/upgrade, setup searches the ACL of the server object for any "Deny-an Exchange Domain Servers group-Receive-As" ACEs and removes them.

When creating a new Exchange 2003 admin group (via setup or Exchange System Manager), setup enumerates all existing Exchange Domain Server groups, and sets a "Deny-Receive-As" ACE for each Exchange Domain Server group on the new admin group's "Servers" container.

Authenticated Users removed from local (and terminal) login Previously in Exchange 2000 Server, no changes were made to the local security settings of member servers by setup. This meant that a normal user could potentially wreak havoc with server settings for files. Although this security setting already prevents domain users from logging onto domain controllers, Exchange Server 2003 takes security steps for member servers by improving the setup to remove "BUILTIN\Users" from the "Log on locally" policy (called "Allow log on locally" on 2003 servers.) BUILTIN\Users contains "Authenticated Users" and "Domain Users" by default, so setup removes those users' ability to log on to the machine. Administrators, power users, and backup operators are still allowed to log on. This change applies to install, upgrade, and reinstall modes.

Exchange Server 2003 drops support for FAT partitions As part of hardening the product for security, setup no longer allows the installation path to reside on a FAT partition. Setup checks for the partition of the installation directory path on the component selection screen, as well as the following locations:

The System partition

Page 58: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 51

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

The partition where Exchange binaries are held The partition(s) containing transaction log files The partition(s) containing database files Any partition(s) containing other Exchange files Exchange Management component only (a.k.a. Exchange System Manager-

only) installations. If the partition is non-NTFS, the prerequisite check halts installation. Although this is very reasonable for securing servers, installing Exchange System Manager on workstations may cause minor problems due to the perception that workstations need not be as secure. There is a way to workaround this prerequisite check by mounting a FAT partition to a directory on an NTFS partition. However, the recommended solution is to convert the FAT file system, using the convert /fs:NTFS command.

Message limits reset on installs When the very first Exchange Server 2003 Server is installed into an org, the Sending Message Size and Receiving Message Size will be set to 10,240 KB (10 MB) if the value is not currently set. This also means that on upgrades from Exchange 2000 Server, reinstalls of Exchange Server 2003, or Exchange 2003 service pack upgrades, that the global message size restriction will be set to 10 MB if it isn’t already set. If the message size restriction is already set to some value, then that value will be preserved. Additionally, on every Exchange Server 2003 server installation or upgrade, the Maximum Item Size for Public Folder postings will be set to 10 MB if the value is not already set, and preserved if it is.

NNTP/POP3/IMAP4 services disabled by default As most Exchange 2000 customers did not use the Network News Transfer Protocol (NNTP), Post Office Protocol version 3 (POP3), and Internet Message Access Protocol version 4rev1 (IMAP4) services, Exchange 2003 setup disables them by default in order to prevent any possible protocol attacks. Additionally, on each server installation, upgrade, or reinstallation, the default NNTP Virtual Server Instance is reset so that anonymous auth is disabled. On uninstalls of Exchange Server 2003, NT_Auth will be disabled. Extending this security push to clusters, EVS creation will no longer create the POP3/IMAP4 resources. So for the latter case, customers not only need to enable these services on each node; they must also manually create the POP3 and IMAP4 resources for each virtual server. These services/resources will not be affected if in-place upgraded from Exchange 2000 Server.

Non-admins are no longer able to create top-level public folders Due to a relaxed default permissions set, any user in an Exchange 2000 Server organization could potentially litter the public folder hierarchy with tons of top-level folders. Because this was a common administrative problem, Exchange Server 2003 “secures” the public folders from users: The permission to ‘Allow create top level public folder’ has been removed from ‘Everyone’ and ‘Anonymous Logon’ ACEs at the organization container’s level. This ACE is checked during each instance of setup /forestprep.

Message Tracking Logs share is more secure In Exchange 2000, the servername.log share, which contained the message tracking log files, wasn’t secure. The ‘Everyone’ group had access to read the

Page 59: Exchange Server Admin Troubleshooting

52 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

tracking logs, so attackers could read username and servername information throughout the Exchange organization. In Exchange 2003, the tracking logs share is locked down so that ‘Administrators’ built-in group is the only default ACE.

Page 60: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 53

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Troubleshooting Exchange Server 2003 setup failures:

Where to start: There are many different ways setup can go wrong, as there are many distributed processes occurring on the network environment to complete the task. Although the deployment tools avoid many of the directory and DNS mis-configuration problems of past Exchange 2000 Server experiences, problems can still occur. Therefore, you may find that traditional Exchange 2000 Server troubleshooting methods to still apply. Much of this begins with reading from the application event log to determine whether the setup completed, or if the problem logged by the setup program is consistent with the customer’s problem description. A minor new feature in Exchange Server 2003 is the improvement on informational events from the source MSExchangeSetup, and you can expect to see the following at the end of each successful setup session:

Event Category: Microsoft Exchange Setup Event ID: 1001 Description: Exchange Server setup (build 6885) completed successfully.

You should examine the event, error, and setup.log files located in the \Program Files\Microsoft Integration\exchsrvr\logs folder. The setup.log file will tell you which components were selected to be installed, as they are not easy to glean from the progress log (discussed below).

The Exchange 2003 setup engine is not much different from the Exchange 2000 setup process, which was designed to tolerate minor errors. If a minor error occurs during installation, you can correct the cause of the error and continue. In fact, Setup attempts to run as many checks as possible before the actual installation takes place. When you are presented with the component selection, you can determine whether errors occurred during the initial checks. If you see four dashes (----) adjacent to a component instead of Install, it usually means that an error message will occur if you try to perform an action (Install, Remove, and so forth) on that component. The error messages at this stage provide useful information about what the error might be. Setup may be unable

Page 61: Exchange Server Admin Troubleshooting

54 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

find installed prerequisite services, such as NNTP and SMTP, or the installer doesn’t have the permissions to run setup. Thus, when selecting the install action, you will be warned of the problem.

If the initial Setup checks do not uncover an error, and you proceed to the later stages of setup, a catastrophic failure occurs. Usually, you are notified in which component or action the error occurred, and then given a hexadecimal error number; for example, 0xC0070430. If you encounter such an error, try clicking Retry because a transient error may have occurred on either the local computer or network. For example, the error code earlier in this paragraph indicates that Setup attempted to install a service that already exists. You might get this error if you are reinstalling the server after a failed attempt. If you are not proceeding in Setup after you click Retry, use the Knowledge Base to see if the error is known.

Your next step is to look at the progress log in the root directory of your system partition. Exchange 2003 setup will always generate or append to the Exchange Server Setup Progress.log file. In cases where you are joining the first Exchange 2003 server to an Exchange 5.5 site, the SRS and configuration connection agreement must be created. Any replication errors between Active Directory and the SRS would appear in the Active Directory Connector.log file.

The progress log contains extremely detailed lists of all functions called and the results of the Setup process. Because you need the source code to understand the function names, not everything in the progress log is recognizable. However, by viewing the contents of a log file, you can discover reasons why Setup failed.

Progress logs are concatenated. This means that all Setup attempts are recorded in one long file, so it is best to go to the end of the file and work backwards. Just plant the cursor at the last byte of the progress log, and search upwards for errors. Some useful keywords to search on are the word “failed,” and snippets of text/error codes from popup dialogs during the setup session, such as “0xC0070430” from the previous example.

Note that setup errors can be either soft or hard, and both kinds of errors appear in the logs. Soft errors are ignored by the Setup process, and you will not see a visual indication of them in the user interface. Here is a prime example of a soft error:

[09:49:41] ScGetClusterSvcDir (l:\admin\src\libs\exsetup\exmisc.cxx:2339) Error code 0XC0070424 (1060): The specified service does not exist as an installed service.

Setup is attempting to access the shared cluster directory. If your computer is not in a cluster, you would expect to see this error. After the soft errors, you see a statement in the logs that indicates that these errors were ignored:

[09:49:41] === IGNORING PREVIOUS ERRORS === CFileManager::ScAutoDetectDirectoryLocations (l:\admin\src\udog\setupbase\tools\filemgr.cxx:569) The operation has completed successfully.

Interestingly, you see the file name and path to the source code in these errors. One of the most interesting sections of the progress log is the following:

Page 62: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 55

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[10:56:58] Setup configuration information: -- ID:62227 -- [10:56:58] This is a(n) Enterprise version of Microsoft Exchange Server -- ID:62232 -- [10:56:58] This is an evaluation copy of Microsoft Exchange Server; it expires in 360 days -- ID:62233 -- [10:56:58] InstallSourceDir = d:\setup\i386\exchange -- ID:62228 -- [10:56:58] InstallDestDir = C:\Program Files\Exchsrvr -- ID:62228 -- [10:56:58] InetSrvDir = C:\WINNT\System32\inetsrv -- ID:62228 -- [10:56:58] System32Dir = C:\WINNT\System32 -- ID:62228 -- [10:56:58] LocalServer = TILAB-DC02 -- ID:62228 -- [10:56:58] SchemaMasterDC = TILAB-GC01 -- ID:62228 -- [10:56:58] DC = TILAB-DC02 -- ID:62228 -- [10:56:58] Domain = tilab.gsx -- ID:62228 -- [10:56:58] DomainDN = /dc=gsx/dc=tilab -- ID:62228 -- [10:56:58] NetBIOSDomain = TILAB -- ID:62228 -- [10:56:58] NT5Site = Default-First-Site-Name -- ID:62228 -- [10:56:58] Org = Microsoft -- ID:62228 -- [10:56:58] LegacyOrg = Microsoft -- ID:62228 -- [10:56:58] AdminGroup = GSXSite1 -- ID:62228 -- [10:56:58] LegacyAdminGroup = GSXSite1 -- ID:62228 -- [10:56:58] AdminGroupContainingRoutingGroup = GSXSite1 -- ID:62228 -- [10:56:58] RoutingGroup = GSXSite1 -- ID:62228 -- [10:56:58] 55ServiceAccountLogin = Uninitialized -- ID:62229 -- [10:56:58] PTAdministratorAccount = TILAB\exservice -- ID:62228 -- [10:56:58] This is not a clustered machine -- ID:62231 –

The most important information included in this excerpt concerns the domain controller from which the server reads Active Directory. You can also see the schema master here, so if you receive an error saying that Setup cannot contact the schema master, you can find the computer name of the schema master and try to contact it manually.

Short names are used frequently in the logs. It helps if you understand some of them, such as the following:

Whether the installation process is successful or unsuccessful, the last entry in the log indicates that the Setup process is being removed from memory, and looks something like the following:

[16:03:46] CComBOIFacesFactory::QueryInterface (K:\admin\src\udog\BO\bofactory.cxx:52) Error code 0X80004002 (16386): No interface.

The log files may provide more information than you need. Fortunately, there is a tool called LogParser (\\exutils\exes\logparser\2003) that reads the progress logs and presents them to you in a format that is easier to read. This does not

Page 63: Exchange Server Admin Troubleshooting

56 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

mean that LogParser translates errors, but it does show you the individual installation attempts and categorizes the errors.

New to Exchange 2003 is this new entry that appears in all progress logs. Logparser sees it as a “hard error,” but it may be safely ignored:

GetServerAppletalkAddress (f:\df6803\admin\src\udog\excommon\ptudutil.cxx:1917) Error code 0X00273F (10047): An address incompatible with the requested protocol was used.

In summary, troubleshooting the progress log file can be straightforward, even though you do not know what any functions mean: Simply gather as much information relating to which setup options were selected for which components (i.e. typical, custom, change, remove, etc.), and use logparser to locate the date and time of the problem setup session. When you locate the failure in the progress log, refer to the Knowledge Base; otherwise, examine the calls that are made at or around the error. At times, you may find it necessary to troubleshoot at a lower layer, such as obtaining a network trace between the installing server and a domain controller, to determine why setup is failing.

Page 64: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 57

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

General Log Flow

In a full length setup log instance (a log instance from a successful setup), the general, high-level flow of logged information is as follows (colors stand for phases of setup: Initialization (plum/burgundy), PreSetup (orange), Setup (blue), PostSetup (red)

Initialization

Beginning of log Setup initialization

o Gathering of domain information o Checking domain schema version o Gathering Exchange org information o Checking permissions

Dump of component selection o Checking for prerequisites

Configuration information o Install source and destination directories o Local server, schema master server, domain controller server o Domain information, o Exchange org information o Service and Administrator Account information

PreSetup

Setting installation action on selected components OrgPrep actions (ForestPrep, DomainPrep), as appropriate Stopping of services Creation of file copy queues File copy

SetUp

Installation of components/subcomponents o Registry entries

Delivery Tip Student workbooks will not see color. Instructors may want to display the electronic file information in color to illustrate.

Page 65: Exchange Server Admin Troubleshooting

58 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

o Creation of directory objects o Metabase entries

Post-Setup

Post-Setup o Start services

How a Log Instance Begins: The first line of every log instance looks like this:

[04:12:39] ************** Beginning Setup run **************

The second line of every log entry is also the same:

[15:33:20] Starting Exchange 6885 setup on Windows 5.2.3765. at 15:33:20 02/24/2003

Notice the Exchange version information, the operating system version information, and the time/date information.

Strategy Tip: Because each log instance is appended to each previous instance in the log, it can be difficult to find the Setup log instance you’re interested in. To quickly scan through the instances in the log, cut and paste a portion of the first line, without the time stamp (************** Beginning Setup run **************) into a Search window. As you encounter each instance in the log, quickly scan the second line of the instance to find Exchange version information, as well as the time/date stamp of that particular setup. Of course, you can also continue searching until you find the last entry of the first-line text, which will be the start of the log instance for the most recently run Setup (up to and including any in-progress setup.exe or update.exe).

How a Log Instance Ends: The Setup log always has as its last entry (except when viewed in-progress, or when setup ended prematurely):

[13:36:40] CComBOIFacesFactory::QueryInterface (k:\admin\src\udog\bo\bofactory.cxx:54) Error code 0X80004002 (16386): No interface.

The error code in this case is expected.

Terminology The progress log will be scattered with component names or their nicknames, as well as notes from prior versions of Exchange. Use the following table to decipher what a progress log is trying to check:

55 = Exchange Server 5.5 Osmium = Exchange Server 5.5 Oz = Exchange Server 5.5

Page 66: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 59

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Pt or Platinum = Exchange 2000 Server PtOz = Mixed Exchange 5.5 and Exchange 2000 site or organization Udog = Exchange Setup Underdog = Exchange Setup Cartman = BackOffice Setup MSIExec = Windows installer service

For example, CAtomPtOz::ScLaunchADCToSynchPtWithOzTopology may be seen from a customer’s log. You can figure-out that setup is at a phase where it’s probably instructing the ADC to replicate the configuration connection agreement, thereby replicating Active Directory and the Exchange 5.5 naming context (via the SRS).

Configuration Information One of the first places to look for an idea of how Exchange Setup interpreted your domain, org, and accounts is in the configuration information section of the log. The configuration information looks like this:

Page 67: Exchange Server Admin Troubleshooting

60 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[15:45:12] Setup configuration information: -- ID:62227 -- [15:45:12] This is a(n) Enterprise version of Microsoft Exchange Server -- ID:62232 -- [15:45:12] This is not an evaluation copy of Microsoft Exchange Server; it will not expire -- ID:62234 -- [15:45:12] InstallSourceDir = c:\ti6885.0\setup\i386\exchange -- ID:62228 -- [15:45:12] InstallDestDir = C:\Program Files\Exchsrvr -- ID:62228 -- [15:45:12] InetSrvDir = C:\WINDOWS\system32\inetsrv -- ID:62228 -- [15:45:12] System32Dir = C:\WINDOWS\system32 -- ID:62228 -- [15:45:12] LocalServer = DARKWINGDUCK -- ID:62228 -- [15:45:12] SchemaMasterDC = DARKWINGDUCK -- ID:62228 -- [15:45:12] DC = DARKWINGDUCK -- ID:62228 -- [15:45:12] Domain = darkforest.internal -- ID:62228 -- [15:45:12] DomainDN = /dc=internal/dc=darkforest -- ID:62228 -- [15:45:12] NetBIOSDomain = DARKFOREST -- ID:62228 -- [15:45:12] NT5Site = Default-First-Site-Name -- ID:62228 -- [15:45:12] Org = Microsoft -- ID:62228 -- [15:45:12] LegacyOrg = Microsoft -- ID:62228 -- [15:45:12] AdminGroup = First Administrative Group -- ID:62228 -- [15:45:12] LegacyAdminGroup = First Administrative Group -- ID:62228 -- [15:45:12] AdminGroupContainingRoutingGroup = First Administrative Group -- ID:62228 -- [15:45:12] RoutingGroup = First Routing Group -- ID:62228 -- [15:45:12] 55ServiceAccountLogin = Uninitialized -- ID:62229 -- [15:45:12] PTAdministratorAccount = DARKFOREST\Administrator -- ID:62228 -- [15:45:12] This is not a clustered machine -- ID:62231 --

With the configuration information, a user can tell which is the Schema operations master setup is trying to contact, which domain controller setup has contacted, what domain it thinks it is a part of, and often what account has been granted installation permissions (PTAdministratorAccount).

How Failures Get Logged There are several ways for Exchange Setup to “fail”. Many of these have been anticipated, and checks are made to ensure that the conditions leading to such failures are not present. These checks constitute Setup prerequisites. Prerequisite “failures” should happen early – usually when a user attempts to set an illegal installation action on the Component Picker page. In these cases the user will be presented with the prerequisite failure message through the UI. However, in the case of unattended setups, the UI is never presented, so the user must go to the logs to determine the cause of failure. Multiple prerequisite messages may be logged together.

Page 68: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 61

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

A prerequisite failure will occur in the initialization phase of setup, and looks like this:

[04:12:39] Prerequisites for Microsoft Exchange Messaging and Collaboration Services failed: Multiple components cannot be assigned the requested action(s) because: - Setup is unable to access the Windows 2000 Active Directory - Failed to look up the Windows 2000 site to which this computer belongs. Please verify that your computer is configured with correct site information and is in a Windows 2000 domain where the domain controller is reachable.

The component "Microsoft Exchange Messaging and Collaboration Services" cannot be assigned the action "Install" because: - The NNTP component of Microsoft Internet Information Services (IIS) is not installed

If a user has successfully launched setup, satisfying all the Exchange setup prerequisites, there are several kinds of setup errors that are logged to the progress log, some of which are expected. Whenever setup attempts to perform an action, and that action fails, an error is returned. In many cases we expect failure, so you will see many errors logged like this:

[16:05:11] CService::ScQueryServiceConfig (f:\df6885\admin\src\libs\exsetup\service.cxx:539) Error code 0XC0070424 (1060): The specified service does not exist as an installed service. [16:05:11] Service = 'MSExchangeSRS' CServiceManager::ScGetServiceInfo (f:\df6885\admin\src\udog\setupbase\tools\svcmgr.cxx:415) Error code 0XC0070424 (1060): The specified service does not exist as an installed service. [16:05:11] Service = 'MSExchangeSRS' CServiceManager::ScStartService (f:\df6885\admin\src\udog\setupbase\tools\svcmgr.cxx:481) Error code 0XC0070424 (1060): The specified service does not exist as an installed service. [16:05:11] Failed to start the SRS on server DARKWINGDUCK [16:05:11] CAtomSRS::ScRemoveAllDRCsAssociatedWithLocalServer (f:\df6885\admin\src\udog\exsetdata\components\server\a_srs.cxx:2443) Error code 0XC0070424 (1060): The specified service does not exist as an installed service. [16:05:11] Leaving CAtomSRS::ScRemoveAllDRCsAssociatedWithLocalServer [16:05:11] === IGNORING PREVIOUS ERRORS === ScRemoveAllDRCsAssociatedWithLocalServer called from CAtomSRS::ScRemoveDSObjects (f:\df6885\admin\src\udog\exsetdata\components\server\a_srs.cxx:1967) The operation has completed successfully.

Page 69: Exchange Server Admin Troubleshooting

62 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

In these cases, setup may be trying to query the status of a service it has not yet created, or starting a service that is not installed, or trying to clean up directory objects that do not exist.

However, in the case of a fatal string of errors, you might see something more like this (notice the component error message, in bold):

Page 70: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 63

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[11:13:59] The command "x:\Exchange Server 2003\rtm\6623.0\server\rtl\usa\setup\i386\exchange\adc" -console -RO -CA "cn=Config CA_FirstAG_EXSETUPLAB57,cn=Active Directory Connections,cn=Microsoft Exchange,cn=Services,cn=Configuration,dc=alextestdom22,dc=extest,dc=microsoft,dc=com" -dc EXSETUPLAB64 -log 2 "D:\Active Directory Connector.Log" failed, returning error code -2147467259 (8An unknown error has occurred.). ScCreateProcess (k:\admin\src\libs\exsetup\hiddenw1.cxx:1816) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] CAtomPtOz::ScLaunchADCToSynchPtWithOzTopology (k:\admin\src\udog\exsetdata\components\server\a_ptoz.cxx:462) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] While launching the ADC for synchronization, Setup encountered an error:' Error: 'An internal component has failed.' (k:\admin\src\udog\exsetdata\components\server\a_ptoz.cxx:1045) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] CAtomPtOz::ScAddDSObjects(), while calling ScLaunchADCToSynchPtWithOzTopology (k:\admin\src\udog\exsetdata\components\server\a_ptoz.cxx:1046) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] CAtomPtOz::ScAddDSObjects (k:\admin\src\udog\exsetdata\components\server\a_ptoz.cxx:271) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] CBaseAtom::ScAdd (k:\admin\src\udog\setupbase\basecomp\baseatom.cxx:885) Error code 0XC103798A (31114): An internal component has failed [11:13:59] Service = '' CBaseServiceAtom::ScAdd (k:\admin\src\udog\setupbase\basecomp\basesvcatom.cxx:203) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] CAtomPtOz::ScAdd (k:\admin\src\udog\exsetdata\components\server\a_ptoz.cxx:204) Error code 0XC103798A (31114): An internal component has failed. [11:13:59] mode = 'Install' (61953) CAtomPtOz::ScSetup (k:\admin\src\udog\exsetdata\components\server\a_ptoz.cxx:2176) Error code 0XC103798A (31114): An internal component has failed.

Page 71: Exchange Server Admin Troubleshooting

64 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[11:13:59] >>>>>>>>>> Setup encountered a fatal error during Microsoft Exchange Forest Preparation Install component task. CBaseComponent::ScSetup (k:\admin\src\udog\exsetdata\components\forprep\compforprep.cxx:461) Error code 0XC103798A (31114): An internal component has failed.

This is the kind of error that would result in a failed installation.

Strategy Tip for any Error Messages: If you have an instance of setup running, and are being presented with an error message, you can open up the setup log while setup is still running and skip right to the end of the log, which should have as its last entry the error which generated the UI error message. Highlight the last log entry, and several lines above it (look for the closest Entering CAtom… line above the error entry. It will contain the name of the function being called by setup at the time of the failure, which is very useful for tracking down the section of code in which the error occurred), and cut and paste that section into a separate text file. By grabbing the error text in this way, you will make it easier to find in the log later. If you instead continue with the setup, allowing setup to complete, the error message can be difficult to find, “hidden” as it is in the body of the log file.

Strategy for “Retry” dialog boxes: If your setup is hung with a retry/cancel pair of buttons, you are often at a recoverable state if the problem is a transient network error (as mentioned in the Troubleshooting Exchange 2003 failures section). A bad entry in a DNS cache (i.e. negative DNS caching), would be an example of a transient error that can go away if not retried in ten minutes. However, if resolving the root cause requires user interaction, and if the errors in the progress log are vague, there are some utilities you can use to assist in assisting setup to recover:

Use network monitor to capture packets between the installing server, any Exchange 5.5 servers in the site, the ADC server, and the domain controllers and global catalogs chosen by setup (gleaned from the progress log’s configuration information section). Once the capture filter is set, start the capture and immediately hit retry. The server will usually make the same calls (hopefully across the wire) and you can later examine the capture to determine where the problem resides. To keep the number of frames small so that you need not examine extraneous traffic, immediately stop the capture as soon as the “retry/cancel” dialog returns.

If the error message relates to Active Directory Service Interfaces (ADSI), or if the error code in the progress log is accompanied by an LDAP protocol error similar to those listed in 218185 - Microsoft LDAP Error Codes, the directory service is probably from a domain controller. In the case where the problem is reproducible, running setup again using the /chooseDC switch is used to narrow-down the domain controller choices, thereby making it easier to decipher the netmon capture.

If the error code or the surrounding progress log entries mention ‘DAPI’, then you can focus on the Exchange 5.5 directory service. The root cause could just be a simple permissions setting on a naming context. However, in most cases, the root cause is with IIS, WMI, or some other operating system-level

Page 72: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 65

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

component. Use the clues from the progress log to initiate general troubleshooting in those areas, and contact a platforms specialist in case KB-hunting results in nothing fruitful.

Strategy for common “Exchange 2000” errors: Often, error messages that you search upon will produce search results matching Exchange 2000 errors. Do not discount the errors, as Exchange 2000 and Exchange 2003 setup engines are essentially the same. For example, if you receive a 0xC103798A error during Exchange 2003 setup, and setup is having problems registering a DLL, you may find Exchange 2000 KB article Q245029. Do attempt the workaround in that article, even if the documented DLL is different. It is likely that there is still an Exchange 5.5 exchmem.dll present on the server.

0xC103798A is an EXTREMELY generic error, and so a Knowledge Base resolution should never be used after diagnosing that error code alone. This error means that an internal component has failed, and thus you must ALWAYS dig within the progress log when troubleshooting this. Customers will typically only use the error code before searching in KB, and so their initial problem reports may mislead you by quoting from the KB. Thus, you should always be diligent in obtaining the progress log when troubleshooting.

0X80072030 is another common error code, and it typically means that an object could not be found – either a file on the local disk, a local registry entry, but more typically in the Active Directory. If it is the first two, use filemon or regmon (from www.sysinternals.com) to monitor the setup program. If it is the latter, use network monitor to determine what object the setup program is attempting to use. The packets in the network monitor trace can be viewed by timestamp (Display | Options | Time of Day), and you can correlate these with the timestamps on the errors in the setup progress log.

Strategy for clusters: Unlike single-server installations, where the files are copied and registered and directory service objects are added in a single session, cluster installs are divided into phases. The first phase is the file-copy and registration phase of the binaries onto each node. Here, the setup engine logs to the progress log as usual. However, setup does not create objects in Active Directory. Although setup will declare success for installing on each node, the installation is only partially complete. The second phase involves manipulating the cluster administrator program (cluadmin.exe) to create the necessary resources. At this stage, the setup engine is not running, and instead, cluadmin.exe performs the background processing. Surprisingly, the creation of the Exchange System Attendant resource will write to the setup progress log, logging another session beginning with “Beginning setup run.” So when troubleshooting problems with the System Attendant creation or initial startup, check in 2 places:

The Exchange Server Setup Progress.Log The cluster.log file, located in %windir%\cluster on each node. The

cluster.log file shows important cluster properties such as whether or not a parameter is being set, based upon an application’s needs:

Progress log sometimes misleading: The progress log will usually tell you what actions the user selected on the “Component selection” screen. However, you cannot always trust what the

Page 73: Exchange Server Admin Troubleshooting

66 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

progress log says the installer has selected. This is because the progress log will pre-process through some of the installation types even before the user has a chance to pick an action from the component picker tree. For example, in a forest containing a single domain controller that already has Exchange 2000 Server SP2 installed, the progress log for Exchange 2003 setup will say:

[09:18:51] Prerequisites for Microsoft Exchange Messaging and Collaboration Services failed: The component "Microsoft Exchange Messaging and Collaboration Services" cannot be assigned the action "Upgrade" because: - The local domain configuration is not up-to-date. You must run setup with the "/DomainPrep" switch within this domain. If you have already done this with the current version of Setup, then you must wait for replication to complete. Consult your documentation for details. - Server Z2 must be a Microsoft Exchange 2000 server with Microsoft Exchange Service Pack 3, or higher, installed.

The above text may mislead you to believe that the installer chose the “upgrade” action, but this is what the progress log contains, even before the user has a chance to choose an action on the component selection screen. Another strategy is to search the progress log for the string “is set to action” for the components’ selected action in the component picker tree. However, these too may show system-selected component actions. You may be inclined at this point to look at the setup.log in \program files\microsoft integration\Microsoft Exchange\logs, but that log only lists the summary of chosen components post-installation.

To accurately determine what action the customer selected during installation, look for a series of looping entries. (These looping entries correspond to user clicks onto the component picker screen itself, but not an action that the user has selected.) At the end of those loops, you may find an entry similar to the following:

[09:39:25] Using cached result for domain "/dc=com/dc=contoso" Following the “cached result” entry, the progress log will show the actual user-selected action. In most cases, it will look exactly like the pre-processed entries from above. However, in some cases, the installer may have chosen another action, such as “remove.”

Troubleshooting setup example: Problem description: Setup shows a popup with “Setup failed while installing sub-component Miscellaneous Atom with error code 0xC1037989 (please consult the installation logs for a detailed description). You may cancel the installation or try the failed step again.” Setup has already exited, and the customer has already attempted to uninstall/reinstall WMI per Q318731. However, the next setup attempt results in the same error.

Troubleshooting steps: If the knowledge base does not produce any hits, you would search for 0xC1037989 from the end of “Exchange server setup progress.log” located at the root of the system partition. Here is the relevant text within the progress log file, when logparser is used to view problems at level 0:

Page 74: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 67

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[18:34:08] CInsParser::ScProcessLine (K:\admin\src\libs\exsetup\hiddenw1.cxx:1226) Error code 0XC1037989 (31113): An internal component is not responding. [18:34:08] Processing file 'z:\setup\i386\exchange\Misc.ins', at or near line 10 (CreateProcess:C:\WINNT\System32\WBEM;C:\WINNT\System32\WBEM\mofcomp.exe "C:\WINNT\System32\WBEM\exwmi.mof";600000) CInsParser::ScProcessLine (K:\admin\src\libs\exsetup\hiddenw1.cxx:486) Error code 0XC1037989 (31113): An internal component is not responding. [18:34:08] Registry file name: 'z:\setup\i386\exchange\Misc.ins' CRegistryManager::ScProcessFile (K:\admin\src\udog\setupbase\tools\regmgr.cxx:95) Error code 0XC1037989 (31113): An internal component is not responding. [18:34:08] Filename = '%sourcedir%\Misc' CBaseAtom::ScRefreshRegistryKeys (K:\admin\src\udog\setupbase\basecomp\baseatom.cxx:1217) Error code 0XC1037989 (31113): An internal component is not responding. [18:34:08] CBaseAtom::ScReinstall (K:\admin\src\udog\setupbase\basecomp\baseatom.cxx:1015) Error code 0XC1037989 (31113): An internal component is not responding. [18:34:08] Service = '' CBaseServiceAtom::ScReinstall (K:\admin\src\udog\setupbase\basecomp\basesvcatom.cxx:231) Error code 0XC1037989 (31113): An internal component is not responding. [18:34:08] mode = 'Reinstall' (61955) CBaseAtom::ScSetup (K:\admin\src\udog\setupbase\basecomp\baseatom.cxx:775) Error code 0XC1037989 (31113): An internal component is not responding.

Any of these functions could be causing the problem, and the main clue may likely have appeared when setup runs the misc.ins script. Since setup has already exited, you may often find that manually running some of the commands that setup attempted will reveal more clues: When running ‘mofcomp.exe "C:\WINNT\System32\WBEM\exwmi.mof” from the command prompt revealed that it was successful after 25 minutes (1,500,000 milliseconds), this meant that the 600,000 millisecond sleep parameter wasn’t enough. At this point, one would copy all of the setup files to the hard drive, modify line ten of the misc.ins script file by bumping-up the sleep time, and then rerun setup. In this case, setup proceeded past this point and completed successfully.

Page 75: Exchange Server Admin Troubleshooting

68 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Lab 1.2: Logparser and examination of progress logs

Objective: This lab will get the student familiarized with logparser and significant sections of the progress log. Answers are in Appendix A, but the learning experience will be ruined if answers viewed prematurely.

1. Power on any virtual machine (preferably SOLO since we will be using it in the next lab).

2. Mount the “Admin_Labfiles.ISO” CD image into the virtual machine. 3. Open the file d:\module1_setupquestions\question1.log using notepad. 4. Can you tell what option(s) were set during component selection?

5. Open the Exchange Server Setup Progress log from

d:\module1_setupquestions\question2.log. Can notepad easily show you how many times setup was run?

6. Navigate to the D: drive and install the new version of the logparser utility. 7. Launch logparser and reopen the question2.log. How many times was setup

run? (Hint: The number of sessions is on the upper-left panel of logparser) 8. When was Exchange initially installed? Which version was it? 9. What version was the most recent install? Was it successful? 10. On the last setup session, is the user installing into the forest root domain?

Page 76: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 69

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

11. Try to find out the scenario (i.e. pure Exchange 2003, mixed Exchange 2003 with Exchange 5.5, Exchange 2003 and Exchange 2000, or all server versions (5.5/2000/2003) installed) (Hint, if logparser only has configuration information checked, you might find useful data.)

12. Open the Exchange Server setup progress log from d:\module1_setupquestions\question3.log

13. Was setup successful? Did the services start? 14. If the customer says that his Exchange services are inoperable, and he sent

you progress3.log, can you explain why it is not operable? Do you think the stores mounted?

15. Open the Exchange Server setup progress log from

d:\module1_setupquestions\question4.log 16. Was the /chooseDC switch used? 17. How many administrative groups exist in the Exchange organization? Do

we have proper permissions to read them?

Other questions: 18. Marker checks are enforced in which of the following scenarios? (Check all

that apply) a) Setup of a new Exchange 2003 server in a site where Exchange 5.5 and

Exchange 2000 servers exist b) Setup of a new Exchange 2003 server in a site where Exchange 2003

and Exchange 5.5 servers exist c) Setup of new Exchange 2003 server in a site where Exchange 5.5 exists d) Setup of new Exchange 2000 server in a site where Exchange Server

5.5 and Exchange 2000 Server exist e) Setup of new Exchange 2000 server in a site where only one Exchange

Server 2003 server already exists. 19. T/F: There are a fixed number of atoms running during setup.

Page 77: Exchange Server Admin Troubleshooting

70 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Lab 1.3: Applying troubleshooting concepts

Objectives: In the next three exercises, students will

practice troubleshooting using the above procedures. recognize that even the simplest of problems are ambiguous.

Exercise 1: In this exercise, students will troubleshoot an upgrade from Exchange 2000 SP3 to Exchange Server 2003 build 6851. There is a specific problem to this pre-release build that does not exist in the released/shipped build. However, we can practice troubleshooting using the procedures discussed in the troubleshooting lesson.

Lab setup: Power-on “Solo.”

Username: Standalone\Administrator

Password: password

“Solo” is a Windows 2000 SP3 domain controller running Exchange 2000 Server SP3. It is a standalone server with no complicated components (no SRS or installed connectors). Exchange Server 2003 build 6851 forestprep and domainprep have already been executed without problems.

1) Use Logparser to examine the existing progress log file on the C: drive. Can you determine if forestprep and domainprep have already been executed? Were there any problems with those installs?

2) Mount the Exchange 2003 beta build (6851) .ISO image onto the virtual CD ROM drive, and start the server upgrade process. Setup will fail catastrophically during the setup phase. DO NOT choose the CANCEL button!

3) For each time you choose “retry” confirm that you see “Retrying failed operation.” DO NOT choose the CANCEL button!

4) Make a copy of the setup progress.log file, and you may run logparser against that copy. (The reason we choose to make a copy is because the logparser cannot open the file that is already locked by setup. Similarly, if logparser opens a previous progress log, a new setup instance cannot append to the progress log because is locked. Then, setup will resort to creating a new progress log with a “2” suffix.) DO NOT choose the CANCEL button! As in the upgrade scenario, pressing cancel will result in a partial/broken installation.

5) Proceed to troubleshoot only using logparser against the progress log file.

Instructor notes: The mssearch service has been disabled. In this simple exercise, students are to simply review the progress log file while they get the “retry/cancel” option. They should open the progress log, view the last setup session, and look for these errors:

Page 78: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 71

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[16:58:04] Entering CAtomMDB::ScInstallCreateSearchApplication

[16:58:04] Creating Microsoft Search application

[16:58:04] Creating search admin component

[16:58:04] Getting the applications interface

[16:58:04] CAtomMDB::ScInstallCreateSearchApplication (f:\df6803\admin\src\udog\exsetdata\components\server\a_mdb.cxx:1975)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CAtomMDB::ScInstallCreateSearchApplication

[16:58:04] Entering CAtomMDB::ScPauseSearchFullPopulation

[16:58:04] Entering CAtomMDB::ScGetBuildCatalogsInterface

[16:58:04] Creating search admin component

[16:58:04] Getting the applications interface

[16:58:04] CAtomMDB::ScGetBuildCatalogsInterface (f:\df6803\admin\src\udog\exsetdata\components\server\a_mdb.cxx:2253)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CAtomMDB::ScGetBuildCatalogsInterface

[16:58:04] CAtomMDB::ScPauseSearchFullPopulation (f:\df6803\admin\src\udog\exsetdata\components\server\a_mdb.cxx:2352)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CAtomMDB::ScPauseSearchFullPopulation

[16:58:04] CAtomMDB::ScRefreshMDBDSObjects (f:\df6803\admin\src\udog\exsetdata\components\server\a_mdb.cxx:835)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CAtomMDB::ScRefreshMDBDSObjects

[16:58:04] CAtomMDB::ScRefreshDSObjects (f:\df6803\admin\src\udog\exsetdata\components\server\a_mdb.cxx:627)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CAtomMDB::ScRefreshDSObjects

Page 79: Exchange Server Admin Troubleshooting

72 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

[16:58:04] CBaseAtom::ScReinstall (f:\df6803\admin\src\udog\setupbase\basecomp\baseatom.cxx:1138)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Service = 'MSExchangeIS' CBaseServiceAtom::ScReinstall (f:\df6803\admin\src\udog\setupbase\basecomp\basesvcatom.cxx:247)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CBaseServiceAtom(Information Store Service)::ScReinstall

[16:58:04] CBaseServiceAtom::ScUpgradeFrom2000 (f:\df6803\admin\src\udog\setupbase\basecomp\basesvcatom.cxx:418)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

[16:58:04] Leaving CBaseServiceAtom(Information Store Service)::ScUpgradeFrom2000

[16:58:04] mode = 'Upgrade' (61968) CBaseAtom::ScSetup (f:\df6803\admin\src\udog\setupbase\basecomp\baseatom.cxx:841)

Error code 0X80070422 (1058): The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

The strategy is to look a few lines above the initial error for the calling function. They will see that setup tried to create the MSSearch service. Hopefully, the students will understand that this Exchange 2000 service was disabled, and it must be re-enabled. Once started, they hit the retry button, and the upgrade to Exchange 2003 proceeds.

When you are finished with your troubleshooting and have proceeded through the upgrade past the point of failure, power off the virtual machine and discard changes on the undo drive.

Exercise 2: Turn on the virtual machine located in the “SetupExercise2” folder.

Servername: Z2

Username: MS\Administrator

Password: Password1

Page 80: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 73

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

This virtual machine is similar to the virtual machine in the first exercise. Practice upgrading and troubleshooting the errors you encounter. You may use the release bits (Ti6844.4) to perform the upgrade.

In this exercise, the disk is near capacity, but not full enough for setup to fail the prerequisite check. The GUI portion of setup should fail without any indication of a true reason. Thus, students have an opportunity to investigate using the progress log. When you are finished with this lab, power off Z2 and discard any changes to the undo drive.

Exercise 3: Upgrading a cluster The purpose of this exercise is to observe the steps required for a rolling upgrade on a cluster. There is nothing “broken” in this configuration.

Power on the cluster nodes 1 and 2 located in the “c:\vms\flats\module 1 - setup*” folder (or it may be called something similar). Please be sure not to start the cluster virtual machines from module #5 (Clustering Lab).

Setup: Each cluster node is also a domain controller. Note: This configuration is meant to optimize lab equipment; it is not a recommended configuration in reality, as cluster nodes should not reside on domain controllers.

1. What build of Exchange Server 2003 is installed? 2. Open cluster administrator 3. Right-click on the EXVS1 group. What new option do you see? 4. Right-click on the system attendant resource. Observe that the same option

is selectable. 5. In the “net name test” cluster group, create a new System Attendant

resource. 6. Open the setup progress log using logparser (located on the desktop).

Although you never ran setup.exe, do you see any changes? 7. Mount the Ti6944.4EntEval.ISO image to the virtual machine. 8. Upgrade each node via rolling upgrade (refer to getting started guide). But

instead of choosing “upgrade” on the component selection screen, choose “reinstall.”

Page 81: Exchange Server Admin Troubleshooting

74 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Appendix A: Answers 1.2.4: "The component Microsoft Exchange is set to "Reinstall"

1.2.5: Not really. If we wanted to find out, we'd need to search for the string "Beginning Setup" or patterns of asterisks ("*****") and count the lines they occur.

1.2.6: Nine times

1.2.8: Setup was run on 1/13/2003. This was Exchange 2000 Server release build (4417)

1.2.9: Build 6895. No, it was not successful because this entry does not exist at the end:

CComBOIFacesFactory::QueryInterface (f:\df6895\admin\src\udog\bo\bofactory.cxx:54) Error code 0X80004002 (16386): No interface. Instead, it was probably killed in the middle of a dialog box.

1.2.10: No, this is where we can make use of our "m_str" searches to find server role information. All in the same general area, we find:

DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "MLABNET" DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "mlabnet.com" DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "mlabroot.com"

m_strRootDomain = "mlabroot.com"

In this case, the user is installing to some other domain in the forest (mlabnet.com) that appears to be a root of its own tree. The forest root domain is mlabroot.com.

1.2.11: Not likely to have Exchange 5.5 in the environment, because 55serviceaccountlogin is uninitialized. Furthermore, there are no hrdirprereq* strings anywhere in the log. This is likely a Pure Exchange 2003 or Exchange 2000/2003 admin group. However, this cursory inspection doesn't rule-out the possibility that there might be an Exchange Server 5.5 server in some other admin group in the org.

1.2.13: Yes, setup was successful from seeing this string:

!!!!!!!!!!Setup completed successfully! And a few lines above that, we see many services starting.

1.2.14: The server is not operational because this was a cluster node installation. And as such, no Active Directory objects were created representing the server or its stores. So users will not be able to use it until a virtual server

Page 82: Exchange Server Admin Troubleshooting

Module 1: Setup Changes 75

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

has been created, which in turn creates the IS resource, in which case the stores can mount. In its present state, no stores are mounted.

1.2.16: No, setup chose its own suitable domain controller for setup:

No user-specified DC; setup has chosen m_strDC = "CRPDALDCS00"

1.2.17: Four administrative groups exist, and we do have perms to read:

[08:28:00] Enumerating all admin groups in the org [08:28:00] Found 4 admin groups [08:28:00] Checking permissions on the admin group: /dc=com/dc=testofamerica/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=Bank of America/cn=Administrative Groups/cn=ASIA Administrative Group [08:28:00] We have permission ExchAG_Read [08:28:00] We have permission ExchAG_Write [08:28:00] We have permission ExchAG_SetPerms [08:28:00] Checking permissions on the admin group: /dc=com/dc=testofamerica/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=Bank of America/cn=Administrative Groups/cn=EMEA Administrative Group [08:28:00] We have permission ExchAG_Read [08:28:00] We have permission ExchAG_Write [08:28:00] We have permission ExchAG_SetPerms [08:28:00] Checking permissions on the admin group: /dc=com/dc=testofamerica/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=Bank of America/cn=Administrative Groups/cn=North American Administrative Group [08:28:00] We have permission ExchAG_Read [08:28:00] We have permission ExchAG_Write [08:28:01] We have permission ExchAG_SetPerms [08:28:01] Checking permissions on the admin group: /dc=com/dc=testofamerica/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=Bank of America/cn=Administrative Groups/cn=Routing Adminisrative Group [08:28:01] We have permission ExchAG_Read [08:28:01] We have permission ExchAG_Write [08:28:01] We have permission ExchAG_SetPerms [08:28:01] Final set of permissions: 0XF0C0E0E0

1.2.18: A and C. (B is not an answer because the first Exchange 2003 server install had already performed its marker checks)

1.2.19: False. The number of atoms running varies, depending on the scenarios detected, and which options the installer chooses from the component picker.

Page 83: Exchange Server Admin Troubleshooting

76 Module 1: Setup Changes

Last Saved: 7/24/2003 1:55 AM Last Printed: 7/24/2003 12:55 PM

Acknowledgments Microsoft Employee Vincent Yim

Max Vaysburg, Ross TenEyck, Alexander MacLeod, Gwen Zierdt, Bryan Atwood

KB article 823145 XADM: Exchange 2003 Server Setup and Installation Top Support

Issues

Page 84: Exchange Server Admin Troubleshooting

Contents

New Administrative features in System Manager and Active Directory ............1 New Exchange System Manager Features ..........................................................2 New Diagnostics .................................................................................................7 Public Folder Store and Public Folder Tree ......................................................15 Exchange HTTP Virtual Server ........................................................................20 Mailbox .............................................................................................................26

Module 2: New Administrative Features

Page 85: Exchange Server Admin Troubleshooting

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 86: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 1

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

New Administrative features in System Manager and Active Directory

After completing this course, students will be able to:

Identify and locate the Administrative new features and properties in Exchange Server 2003.

Page 87: Exchange Server Admin Troubleshooting

2 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

New Exchange System Manager Features

Internet Mail Wizard From Exchange 2003’s System Manager console, right-click on your Organization and choose "Internet Mail Wizard" option to set up the server for Internet Mail.

Page 88: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 3

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Real-Time Black Lists; Inbound SMTP Recipient Filtering for Individual Users; Inbound SMTP Recipient Filtering for Users in the Directory Right-click on Message Delivery in Exchange System Manager, under Global Settings and pull up Properties.

Notice three new tabs here:

Sender Filtering Connection Filtering Recipient Filtering

Page 89: Exchange Server Admin Troubleshooting

4 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Page 90: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 5

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Mobile Services Under Global Settings, right-click on Mobile Services and pull up Properties:

Note that right-clicking on Mobile Services also allows you to create a new Mobile Carrier:

Page 91: Exchange Server Admin Troubleshooting

6 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

New Details and Address templates New templates are: Bulgarian, Catalan, Estonian, Latvian, Lithuanian and Serbian.

In Exchange System Manager, under Recipients, click on Details Templates and Address Templates to see new language templates.

Ability to add SMTP addresses to mixed Administrative Group policy View Properties of any mixed Administrative Group (site) policy in Exchange System Manager (those policies will always have the highest priority and a filter based on legacyExchangeDN). On E-mail Addresses tab, the New button is available and additional SMTP addresses can be added:

Exchange 2003 Standard Server can be made a Front End server Pull up Properties of the Exchange 2003 standard server in Exchange System Manager. On the General tab, notice the “This is a front-end server” checkbox:

Page 92: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 7

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

New Diagnostics

Servers container - there are more details for servers Under the Administrative Group, select the Servers container and view additional information about servers in the right pane:

Servers container -

Page 93: Exchange Server Admin Troubleshooting

8 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Mailboxes object - more details on Logons Under the Mailbox Store object in Exchange System Manager, click on Logons. In the right pane, you will see more detailed information about accounts that have logged on to mailboxes. Additional fields can be viewed by going to View > Add/Remove Columns:

Page 94: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 9

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Server properties - can change the tracking log file path Right-click and pull up properties of Exchange 2003 in Exchange System Manager. The General tab shows the path to message tracking logs:

Page 95: Exchange Server Admin Troubleshooting

10 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Additional Diagnostics Logging available on server object Open the Properties of Exchange 2003 server object in Exchange System Manager, and select the Diagnostics Logging tab

Page 96: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 11

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Public Folder referrals tab on server properties Open the Properties of Exchange 2003 server object in Exchange System Manager select the Public Folder Referrals tab:

Page 97: Exchange Server Admin Troubleshooting

12 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Queue Viewer entirely replaced Under Server in Exchange System Manager, click on Queues.

New Queue Types are exposed, creating a much cleaner view of all queues created on the server. You can disable outbound mail and search for messages from here, as well as change the refresh rate for queue viewer:

Page 98: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 13

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Creation of Recovery Storage Group in Exchange System Manager Right-click on the Exchange 2003 server object in Exchange System Manager and choose New, Recovery Storage Group:

Page 99: Exchange Server Admin Troubleshooting

14 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Database tab of the store shows time and type of last backup To view Properties of a Mailbox or Public folder stored in Exchange System Manager (under the Storage Group), go to the Database tab:

Page 100: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 15

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Public Folder Store and Public Folder Tree

Send hierarchy to other servers Right-click on the Public Folder store or Public Folder tree itself and choose the Send Hierarchy option. Choose the source and destination servers that the hierarchy should be sent to and specify the number of days of hierarchy that should be sent.

Page 101: Exchange Server Admin Troubleshooting

16 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Ability to send content of Public Folder to other servers on demand In Exchange System Manager, expand the Public Folder tree and right-click on any public folder. Choose the Send Contents option under the All Tasks menu:

Page 102: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 17

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Improved Age Limits explanation for public folders On the Properties of a public folder under Public Folder Store, Public Folders, the object’s Limits tab clarifies when the age limit is used:

Page 103: Exchange Server Admin Troubleshooting

18 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Make a public folder member of groups On the Properties of a public folder under Public Folder Store, Public Folders, you can add a public folder as a member of a group.

Page 104: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 19

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Public Folders – new tabs show different folder properties View the Properties of folders under the Public Folder tree. There are new tabs in the right pane: Details, Content, Find, Status and Replication tabs.

Page 105: Exchange Server Admin Troubleshooting

20 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Exchange HTTP Virtual Server

In Exchange System Manager, under the Server object, expand the Protocols container and then HTTP and select Exchange Virtual Server. Microsoft-Server-ActiveSync and Outlook Mobile Access are new virtual directories.

Page 106: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 21

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Ability to configure Forms-based authentication In Exchange System Manager, pull up the properties of the Exchange Virtual Server object itself. On the Settings tab, you can enable Forms-based authentication and – if that is enabled – the compression mode:

Page 107: Exchange Server Admin Troubleshooting

22 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

New General tab for Exchange Virtual Directory under HTTP virtual server and new Authentication method In Exchange System Manager, under the Server object, expand the Protocols container and then HTTP and select Exchange Virtual Server. Pull up the properties of Exchange Virtual Directory.

On the Access tab, click the Authentication button:

Page 108: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 23

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Page 109: Exchange Server Admin Troubleshooting

24 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Relocation of SMTP Queue Folder Under Server Protocols, expand SMTP and pull up the properties of SMTP Virtual Server. Go to the Messages tab.

Page 110: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 25

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Relocation of X.400 Message Queue Folder

Under Server Protocols, pull up X.400 Properties.

Page 111: Exchange Server Admin Troubleshooting

26 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Mailbox

Mailbox Recovery Center in Exchange System Manager

Expand Tools in Exchange System Manager. Mailbox stores can be added by right-clicking on Mailbox Recovery Center.

Page 112: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 27

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Exchange tasks on mailboxes from Exchange System Manager Click on Mailboxes object under the mailbox store. Select one or multiple mailboxes in the right pane, right-click and run Exchange Tasks.

Page 113: Exchange Server Admin Troubleshooting

28 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Completely rewritten Move Mailbox wizard Run Exchange tasks on mailboxes either from Exchange System Manager or Active Directory Users and Computers and choose Move Mailbox.

Some of new features include multithreaded move, scheduling of mailbox move, and reporting.

Improved Exchange Features tab in user’s properties Pull up the properties of a user that has a mailbox on Exchange Server 2003 using Active Directory Users and Computers. Select the Exchange Features tab.

Page 114: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 29

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Configure Exchange Features task Run Exchange Task Wizard on mailboxes either from Exchange System Manager or Active Directory Users and Computers and choose Configure Exchange Features.

Ability to create the InterOrgPerson object in Active Directory In Active Directory Users and Computers, right-click on an organizational unit (OU) and choose New, and then InterOrgPerson:

Page 115: Exchange Server Admin Troubleshooting

30 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Creation of Query-based Distribution groups in Active Directory In Active Directory Users and Computers, right-click on an OU and choose New, and then Query-based Distribution Group:

Page 116: Exchange Server Admin Troubleshooting

Module 2: New Administrative Features 31

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

Distribution Groups can be “locked down” for Authenticated users only From the Properties of the Distribution Group in Active Directory Users and Computers select the Exchange General tab:

Page 117: Exchange Server Admin Troubleshooting

32 Module 2: New Administrative Features

Last Saved: 7/24/2003 2:09 AM Last Printed: 7/24/2003 5:52 PM

All Exchange tabs are visible by default In Active Directory Users and Computers, bring up the Properties of a user that has Exchange mailbox. Note that all Exchange tabs are visible even though Advanced Mode is not turned on for the Exchange management snap-in (selecting View and then Advanced Features is not needed):

Page 118: Exchange Server Admin Troubleshooting

1

Contents

Introduction ......................................................................3 Dependencies ..................................................................4 General Overview.............................................................4 Troubleshooting..............................................................10 Tools ..............................................................................17 Labs ...............................................................................19 Acknowledgments ..........................................................22

Module 3: Move Mailbox Wizard

Page 119: Exchange Server Admin Troubleshooting

2

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 120: Exchange Server Admin Troubleshooting

3

Introduction In Exchange Server 2003 the Move Mailbox Wizard has been updated to be multithreaded and have the ability to recover if it encounters corrupt mailbox entries.

By default it will attempt to move the whole mailbox in one attempt. If it fails, it will attempt to move the mailbox item by item. You can set a maximum number of corrupt items, but if it exceeds this number the mailbox move will fail.

By default, Move Mailbox uses four threads, which means it will attempt to move four mailboxes at a time.

Limitations As of the Exchange 2003 release, there is no ability to move mailboxes between Administrative groups when you are in mixed mode environments.

Being in Native Mode means that you have only Exchange 2000 or Exchange 2003 servers only in your org, and you have selected to switch your org to Native mode in Exchange System Manager.

The limitation of not being to move mailboxes between Administrative groups is because of the distinguished names (also known as DNs) and other attributes placed on the mailboxes and mailbox items within the mixed mode admin groups. These attributes involve a large cost of adding and changing when moving between administrative groups within the mixed mode environment.

There is no ability to have the destination store of a moving user to reside in a Recovery Storage Group.

Moving mailboxes is allowed within the same storage groups with no limitations. This is to say that any store that exists within a storage group can be the destination of the Move User request. Within the same storage group directly implies that the Mailbox Stores reside on the same server.

Moving mailboxes outside of the current Mailbox Store’s current Storage Group is allowed within the same server. If the storage groups reside on different servers, then it depends on whether the servers exist in different Admin Groups or not.

You cannot move the System Mailbox, System Attendant Mailbox or SMTP Mailbox. When you right-click Exchange Tasks, you will not see the Move Mailbox option for these special mailboxes.

Likewise, if you attempt to move System Mailboxes with user mailboxes, the detailed report will show errors for the system mailboxes; they will not be moved.

Native Mode gives you more flexibility

System Mailbox has no option to move it.

Page 121: Exchange Server Admin Troubleshooting

4

Dependencies

Client Windows 2003 Admin Tools

Exchange 2003 System Manager

Server Exchange 2003 System Manager

General Overview It is possible to move a mailbox between versions of Exchange that are visible through System Manager.

So it is possible to move mailboxes between Exchange 5.5 / Exchange 2000 / Exchange 2003.

Starting the Wizard You can start the Move Mailbox Wizard in a number of ways:

In Active Directory Users and Computers, navigate to a user and then select one or more user objects. Then, right-click and select Exchange Tasks.

In Exchange System Manager, navigate to a server object, then expand until you get to a mailbox store. Expand Mailboxes and then select one or more user objects. Then, right-click and select Exchange Tasks.

In Active Directory Users and Computers, use the Find dialog. Then, right-click and select Exchange Tasks.

Step by Step Walk Through This is the Welcome screen.

Click Next.

Page 122: Exchange Server Admin Troubleshooting

5

You will see this dialog, as long as the user(s) have a mailbox on a server.

Click Next.

Select the destination Server and Mailbox Store that you want to move the mailboxes to.

Depending on the mode set (native/mixed), all of the mailbox stores in the org/site are available.

Click Next.

This dialog box allows you to select how Move Mailbox handles corrupted messages.

You can either just abort moving the mailbox, or skip corrupted items and generate a failure report.

You can select maximum number of corrupted items to skip before you file the move of the mailbox and generate a failure report. Exchange provides the ability to handle up to 100 bad messages. The UI will restrict the number of bad messages skipped to be 100. This way, no move will occur when there are major store corruptions issues with a mailbox store or with a particular mailbox.

Click Next.

or

Page 123: Exchange Server Admin Troubleshooting

6

You can schedule the start date and time of when the move will start.

You can also specify when to terminate the move. If only eighty out of one hundred mailboxes are finished moving by the ending time, then the administrator can schedule the remaining twenty another day.

Click Next.

By default, it will run four threads and thereby do four tasks at once.

The move starts by connecting to the destination server.

Page 124: Exchange Server Admin Troubleshooting

7

This Event gets logged in the application log when the move starts.

Then it opens the source mailbox.

Exchange prepares the mailbox to be moved.

Page 125: Exchange Server Admin Troubleshooting

8

Then it opens the destination mailbox.

Exchange moves the messages.

Once a thread is finished, Exchange starts on the next mailbox.

Page 126: Exchange Server Admin Troubleshooting

9

This Event gets logged in the application log when the move is successful.

When the moves are completed, you see a summary of the move.

This indicates the number of successful and failed mailbox moves, as well as other summary information, such as time of move.

You can choose to view a detailed report.

Click Finish to close the wizard.

Reports If you selected to view the detailed report, you will see a similar XML output of the summary report.

The XML log file is generated on the disk with the errors pertaining to each failed mailbox move operation.

The log file contains mailboxes that failed with message ID, date of message and subject in the log.

The XML report resides in the “My Documents\Exchange Task Wizard Logs\ “ directory.

Note that the XML report may not be viewable on a machine that does not have the XML parser installed. (The XML parser is automatically installed whenever you install any Exchange 2003 binaries on a machine)

Page 127: Exchange Server Admin Troubleshooting

10

Troubleshooting

Under the bonnet Summary

The mechanics of how the Move Mailbox operation works is that a connection is made to the source and destination servers, the folder hierarchy is created on the destination server, and then a ‘fast MAPI transfer’ process is used to move the date from the source to the destination server. Then, if any corrupted items are encountered, the operation defaults to a non-fast MAPI transfer.

In the "fast" mode we copy the folder "Mailbox – User1" from the source server to the destination server. If this fails, we just know that the whole operation failed and not necessarily which item it failed on.

In the "non-fast" mode we copy "Message 1", "Message 2", "Message 3", "Message ...", from the Inbox, then copy "Message 1", "Message 2", "Message 3", "Message ...", from the Sent Items, etc.

Detail

The way in which we actually perform the move mail box is strictly MAPI.

1. We perform a MAPI logon to the first message store where we are moving the messages from. 2. A pointer to the Message Store is retrieved. 3. A pointer to the destination Message Store is retrieved.

4. The Mapi CopyTo() function is called on the top of the information store first to see if we can do a one

copy operation, from the source message store to the new destination message store. By default the lcid (localization) of the destination mailbox will match that source. When the admin

sets PR_IN_TRANSIT to FALSE on the destination mailbox store, we set the ptagIsLocalized property on the mailbox table. This prevents the mailbox from becoming relocalized.

5. This can be overwritten by a registry value "Localize On First Logon" which is a DWORD value on ParametersSystem,and if it is set > 0, we will revert to the old/current mailbox.

Page 128: Exchange Server Admin Troubleshooting

11

Note: Related KB is 810326 XADM: Folder Names May Be Changed to Hebrew in Migration from Exchange http://support.microsoft.com/?id=810326

6. If there is a failure in the Message Store copy, we make a copy message by message through the Message Store calling the proper CopyTo() function from each message containing a failure. As long as it does not exceed the failure limitation, we will continue the operation message by message until the full contents of the Message Store are copied or the limit of “bad messages” is reached.

Note: Each time we call the MAPI/extended MAPI functions to copy/open the messages, we are employing the store functions that call out to their Virus scanning APIs. When the user has Virus scanning ability on the server, these abilities will be employed.

As the number of users who are chosen to have their mailbox moved increases, Exchange’s performance will decrease, depending on the size of the mailbox as well as the destination of the move and network bandwidth.

If the move happens between the same server, the scalability will be a lot higher than between two slow link servers, since many of the moves will complete faster on the same server.

The majority of the scalability features of Move Mailbox depend on external factors, disk speed, network speed, and complexity of the mailbox. The actual copy of messages itself is spawned across a maximum of five threads and will scale accordingly to hardware and other external factors mentioned previously.

The monitoring of a Move Mailbox feature comes through the admin UI and the call backs received from the copyTo MAPI command that come into maildsmx.dll. A status bar indicates the current progress.

Event Log Entries You can set different diagnostic logging, but it makes no difference to the output in the application log.

Event ID 1006:

You get this when Move Mail starts to move a mailbox.

Page 129: Exchange Server Admin Troubleshooting

12

Event ID 1007:

You get this when a move is successfully completed.

Event ID 1008:

A move failure due to unknown causes. This could be related to a global catalog problem, or offline store. Always look for accompanying events, such as 91xx, and perform knowledge base searches to determine cause and resolution.

Event ID 1008:

Mailbox already exists.

Page 130: Exchange Server Admin Troubleshooting

13

Event ID 1009:

You get this if you cancel the move OR if the move is terminated by selecting the “Cancel tasks that are still running after:” in the Task Schedule.

Event ID 1023:

Severity=Warning

SymbolicName = evtMoveMailboxSetInTransitFailed

Language=English

Unable to set a property on the message store on '%1'. Result: %2

Event ID 1024:

Event ID 9162:

Severity=Error

SymbolicName = evtMoveMailboxContextInitFailed

Language=English

Failed to initialize the Move Mailbox command.

%nError: %1

Event ID 9163:

Severity=Error

SymbolicName = evtMoveMailboxGet55Session

Language=English

Failed to connect to the Exchange 5.5 directory on server '%1'.

Page 131: Exchange Server Admin Troubleshooting

14

%nError: %2

Event ID 9164:

Severity=Error

SymbolicName = evtMoveMailboxFailedToOpenObject

Language=English

Failed to open object '%1' on server '%2'.

%nError: %3

Event ID 9165:

Severity=Error

SymbolicName = evtMoveMailboxUpdate55Directory

Language=English

Failed to update the Exchange 5.5 directory on server '%1'.

%nhomeMDB: %2

%nhomeMTA: %3

%n%nError: %4

Event ID 9166:

Event ID 9167:

Event ID 9168:

Page 132: Exchange Server Admin Troubleshooting

15

Severity=Error

SymbolicName = evtMoveMailboxOpenStore

Language=English

Failed to open mailbox '%1' in mailbox store '%3' on server '%2'.

%nError: %4

Event ID 9169:

Event ID 9170:

Severity=Error

SymbolicName = evtMoveMailboxUpdateW2kDirectory

Language=English

Failed to update mailbox on Active Directory server '%1'.

%nhomeMDB: %2

%nhomeMTA: %3

%nmsExchHomeServer: %4

%n%nError: %5

Event ID 9171:

Severity=Error

SymbolicName = evtMoveMailboxLoadMapi

Language=English

Failed to initialize MAPI32.DLL.

%nError: %1

Event ID 9172:

Severity=Error

SymbolicName = evtMoveMailboxCopyTo

Language=English

Failed to copy messages to the destination mailbox store.

%nError: %1

Page 133: Exchange Server Admin Troubleshooting

16

Event ID 9173:

Severity=Warning

SymbolicName = evtMoveRollbackNT

Language=English

Failed to restore mailbox on Active Directory server '%1'.

%nhomeMDB: %2

%nhomeMTA: %3

%nmsExchHomeServer: %4

%n%nError: %5

Event ID 9174:

Severity=Warning

SymbolicName = evtMoveRollback55

Language=English

Failed to restore mailbox on Exchange 5.5 server '%1'.

%nhomeMDB: %2

%nhomeMTA: %3

%n%nError: %4

Event ID 9175:

Registry Tweaks All of these can be found under:

HKEY_CURRENT_USER\Software\Microsoft\Exchange\MSExchangeAdmin\Exchange Task Wizard\ShowWelcomePage is a dword with a value of 1 to show the Welcome page and 0 not to show it.

HKEY_CURRENT_USER\Software\Microsoft\Exchange\MSExchangeAdmin\Exchange Task Wizard\MaxBadItems is a dword. By default it is 3, and this updates to the value you set in the GUI.

Scripting Move Mailbox is scriptable. This is documented on Microsoft Developer Network (MSDN) and uses CDOEXM (Collaboration Data Objects for Exchange Management).

Page 134: Exchange Server Admin Troubleshooting

17

Tools To help resolve issues in the mailbox move process, Development is working on getting the documentation into the SDK of the schema of the files, which are XML output from the full Exchange Task Wizard.

Move Mailbox Wizard Report: XML Style Sheet Out-of-the-box, it may be difficult to find the information you need in the XML report that is generated:

Fortunately, you can customize the XML reports to your own liking; all you have to do is apply an extensible style sheet to the files that are output by the task wizard. See the attached file for an example style sheet:

MoveMailboxReport.xslt (8 KB)

To apply the style sheet, open one of the XML reports under the "..\Exchange Task Wizard Logs" folder with Notepad, and insert the following statement as the second line in the file (i.e. immediately after the <xml> top line):

<?xml-stylesheet type="text/xsl" href="C:/MoveMailboxReport.xslt"?>

Save your report and open it using Internet Explorer. You should now see the transformation.

Instructor note: .xslt file is available on corpnet at \\learn05\courseware\messaging\Exchange_2003_Admin_TS\Instructor_Guide

Page 135: Exchange Server Admin Troubleshooting

18

Transformed Output

Page 136: Exchange Server Admin Troubleshooting

19

Labs

Lab 3.1: Move Mailbox - Exchange Server 5.5 to Exchange Server 2003 Objectives

After completing this lab, you will be able to:

Move multiple mailboxes from an Exchange Server 5.5 server to an Exchange Server 2003 server.

Observe how new "Exchange Tasks" now operates upon multiple objects.

Servers:

Exchange 2000 Server domain controller/global catalog (to be turned-on first): Z2

Exchange Server 5.5 member: z0

Exchange Server 2003 member: Z3

Estimated time to complete this lab: 10 minutes

Goal: In this exercise, you will move multiple mailboxes from the Exchange 5.5 server to the Exchange 2003 server.

Tasks Detailed Steps

1. Move Exchange Server 5.5 Mailboxes

a. Open Active Directory Users and Computers snap-in. b. Click on View, then Choose Columns, and select Exchange

Mailbox Store and move it to the Displayed Columns field. Click OK.

c. Select multiple Exchange 5.5 users by either using Shift + Click or CTRL + Click from the Users container. (You may need to pre-create them if they do not already exist)

d. Right-click the users and click Exchange Tasks. e. The Welcome to the Exchange Tasks Wizard appears. Click

Next. f. In the Select a Task to Perform box, click Move Mailbox and

click Next. g. On the Move Mailbox screen, verify the Current Mailbox

Store is the Exchange 5.5 server name and the Server name (which is the intended destination) is Z3. Select the Mailbox Store and the Storage Group that the mailbox should be moved to on the Exchange 2003 store.

h. Click Next. i. At the Completing the Exchange Task Wizard, click Finish. j. Repeat the steps to move the remaining users named Move

and Swing.

2. Verify Mailboxes have moved and are accessible to the Outlook web client.

a. Using Outlook Web Access, logon to the moved mailboxes, which reside on the Exchange 2003 server.

Page 137: Exchange Server Admin Troubleshooting

20

Lab 3.2: Move Mailbox - Exchange 2000 Server to Exchange Server 2003 Objectives

After completing this lab, you will be able to:

Move mailboxes from an Exchange 2000 server to an Exchange 2003 server using Exchange System Manager.

Observe move mailbox ability using Exchange System Manager.

Estimated time to complete this lab: 10 minutes

Goal

In this exercise, you will move multiple mailboxes from the Exchange 2000 server to the Exchange 2003 server, using Exchange System Manager.

Tasks Detailed Steps

1. Move Exchange 2000 Mailboxes using Exchange System Manager

a. Open Exchange System Manager on the Exchange 2003 server (Z3).

b. Navigate to Mailboxes on your Exchange 2000 Mailbox Store (Z2).

c. Select one or more of the Exchange 2000 mailboxes. (If none appear, you must pre-create them on the Exchange 2000 store, and send a mail item into their mailboxes)

d. Right-click the user(s) and click Exchange Tasks. e. The Welcome to the Exchange Tasks Wizard appears. Click

Next. f. In the Select a Task to Perform box, click Move Mailbox and

click Next. g. On the Move Mailbox screen, verify the Current Mailbox

Store is the Exchange 2000 server name and the Server name (which is the destination) is Z3. Select the Mailbox Store and the Storage Group that the mailbox should be moved to on the Exchange 2003 server.

h. Click Next. i. At the Completing the Exchange Task Wizard, click Finish.

2. Verify Mailboxes have moved and are accessible to the Outlook web client.

a. Using Outlook Web Access, logon to the moved mailboxes, which reside on the Exchange 2003 server.

Page 138: Exchange Server Admin Troubleshooting

21

Lab 3.3: Move Mailbox - Exchange Server 2003 to Exchange Server 2003 Objectives

After completing this lab, you will be able to:

Move mailboxes from an Exchange 2003 server to an Exchange 2003 server and then view the XML output.

Apply a style sheet to the XML report generated by Move Mailbox wizard.

Estimated time to complete this lab: 15 minutes

Goal

In this exercise, you will move multiple mailboxes from the Exchange 2000 server to the Exchange 2003 server, using Exchange System Manager.

Tasks Detailed Steps

1. Move Exchange 2003 Mailboxes

a. Create a second mailbox store on Z3, which is your Exchange 2003 server.

b. Select one of the Exchange 2003 users currently residing on the first mailbox store.

c. Right-click the user and click Exchange Tasks. d. The Welcome to the Exchange Tasks Wizard appears. Click

Next. e. In the Select a Task to Perform box, click Move Mailbox and

click Next. f. On the Move Mailbox screen, verify the Current Mailbox

Store is Z3/First Storage Group/Mailbox Store and the Server name is also Z3. Select the new mailbox store you created in step 1a. Click Next.

g. At the Completing the Exchange Task Wizard, click Finish. h. Notice the location of the XML file generated at the end of the

Move Mailbox. i. Open the file with Notepad.

2. Viewing Move Mailbox XML file with a “parser”

a. Copy the XML file generated from the Move Mailbox Wizard to a temporary folder. (It is located underneath your “My Documents” folder by default.)

b. Copy the style sheet MoveMailboxReport.xslt from the admin_labfiles .ISO image to the same temporary directory as the XML file.

c. Open the XML file generated by Move Mailbox and under the first line add:

d. <?xml-stylesheet type="text/xsl" href="MoveMailboxReport.xslt"?>

e. Save Changes to the XML file. f. Double-click the XML file you modified in step 2e.

Page 139: Exchange Server Admin Troubleshooting

22

Acknowledgments

Microsoft Employee Paul Flaherty

Fabio Pintos, James Baker, Shawn McGrath, Thomas Andrychowski, Vincent Yim, Steve Schiemann

KB article 822892 Move Mailbox Improvements in Exchange 2003 (http://support.microsoft.com/?id=822892)

264413 Error Connecting to Destination Server During Move Mailbox (http://support.microsoft.com/?id=264413)

Page 140: Exchange Server Admin Troubleshooting

Contents

Introduction.............................................................................................................1 What is the Recovery Storage Group? ....................................................................3 Creating a Recovery Storage Group .......................................................................7 Adding a Mailbox Store to the Recovery Storage Group .....................................10 Active Directory Attributes...................................................................................17 Lab 4.1: Create a Recovery Storage Group and review the Active Directory attributes................................................................................................................20 Overriding the Recovery Storage Group...............................................................25 Restoring the data..................................................................................................27 Recovering Exchange 2000 Server Mailbox Stores..............................................33 Exmerge ................................................................................................................35 Known issues with Exmerge.................................................................................48 Exercise 2: Recovery Storage Group Scenario 1 ..................................................50 Exercise 3: Recovery Storage Group Scenario 2 ..................................................54 Details for Exercise 3: Disaster recovery after a Store Crash using the Recovery Storage Group .......................................................................................................55 Acknowledgments.................................................................................................58

Module 4: Troubleshooting the Recovery Storage Group

Page 141: Exchange Server Admin Troubleshooting

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 142: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 1

Introduction

In this session we will look at the Recovery Storage Group in Exchange Server 2003.

The Recovery Storage Group is a new type of Storage Group in Exchange Server 2003 which is designed to facilitate the recovery of Mailbox Store data without the need for an alternate Active Directory forest for recovery.

In Exchange 2000 Server, we required the use of an alternate Active Directory forest/Exchange org to recover data without affecting the production environment.

The Recovery Storage Group is designed to simplify the data recovery process and lower the TCO for customers by eliminating the need for hardware for an alternate recovery forest.

The Recovery Storage Group allows us to restore production Mailbox Stores to live servers. Once the restore has been performed, we can use Exmerge to merge the data back into the live stores.

We will look at the process of recovering data and also the sort of issues we can run into and how to solve them.

Page 143: Exchange Server Admin Troubleshooting

2 Module 4: Troubleshooting the Recovery Storage Group

Mailbox Recovery in Exchange 2000 Server

Exchange 2000 Server required the use of a separate Active Directory forest in order to restore databases and recover mail items without affecting the production system in any way.

A typical recovery scenario would be:

A server crashes and a mailbox store cannot be mounted. The transaction logs are still available. After unsuccessfully attempting to get the store to mount, the administrator would take copies of any log files and database files from the time of the crash and then mount blank databases so that the users could log in and send/receive mail.

A separate Active Directory forest would then have to be set up and a new Exchange organization would have to be created in this forest. The following information must remain consistent with the production environment:

Exchange Organization Name Administrative Group Name Storage Group Name Logical Database Name LegacyExchangeDN

The administrator must also ensure that the Operating System and Exchange have the same service packs and hot fixes as the production machines.

When the administrator has done this, he could start the restore of the databases to this recovery server.

Once the restore was complete and any log files have been replayed, the database would be at a consistent state. The administrator would then use a utility like Exmerge to export the data to .pst files and then import it into the live database.

The option of switching the recovered databases from the alternate forest with the blank databases in production is also available. This will reduce the amount of time required to Exmerge data back to the production server.

This is obviously a fairly long process and in larger environments, it meant that customers would have to maintain a separate Active Directory forest for this purpose.

The recovery steps for Exchange 2000 Server are explained in 813337.KB.EN-US

A Microsoft Support Webcast goes into great detail on this subject:

Support Webcast: Microsoft Exchange 2000 Alternate Server Data Recovery http://support.microsoft.com/default.aspx?scid=kb;en-us;811063&gssnb=1

Page 144: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 3

What is the Recovery Storage Group?

The Recovery Storage Group is a new type of Storage Group that can be created on servers running Exchange Server 2003 in order to facilitate Mailbox data recovery.

An existing Mailbox Store can be added to a Recovery Storage Group and then recovered while the original database is still online.

The Mailbox store is not physically added to the Recovery Storage Group. A placeholder object is created in the Recovery Storage Group that is used as a reference back to the original Mailbox Store. We will look at this in more detail later.

Exmerge is then used to merge the recovered information over to the production Mailbox Store.

A Recovery Storage Group can be created on both Standard and Enterprise editions of Exchange Server 2003.

Recovery Storage Groups will work irrespective of the Operating System that Exchange Server 2003 is installed on.

Restrictions and criteria Recovery Storage Group has the following restrictions/criteria:

One Recovery Storage Group per server can be created Five Mailbox Stores can be added to the Recovery Storage Group Only mailbox stores from the same Admin Group can be added to the

Recovery Storage Group Once a mailbox store has been added, only mailbox stores from the same

original Storage Group can be added to the Recovery Storage Group An Exchange 2000 Server mailbox store can also be added to an Recovery

Storage Group but it must be a minimum of SP3

Note

Page 145: Exchange Server Admin Troubleshooting

4 Module 4: Troubleshooting the Recovery Storage Group

In order to add a Mailbox Store to the Recovery Storage Group it must exist in the Active Directory

All restores will default to Mailbox Stores that are in the Recovery Storage Group, not to the live database. (Note: The backup client must be pointed to the server hosting the Recovery Storage Group)

There is a registry key to override this behaviour (“Recovery SG Override”) (See Section 7)

Mailbox Stores in the Recovery Storage Group do not mount on start-up/failover.

By default the “This database can be overwritten by a restore” is checked on recovery databases

Public Folder Stores cannot be added to a Recovery Storage Group New Mailbox Stores cannot be created in a Recovery Storage Group When the Mailbox Store is mounted, all mailboxes remain disconnected and

cannot be reconnected No new mailboxes can be created on a recovery database MAPI is the only protocol that is supported When the Mailbox Store in the Recovery Storage Group is mounted, you

will receive a warning (prompt) The log file prefix for the Recovery Storage Group is Rnn On an Active/Active cluster (2 Node) we can create one Recovery Storage

Group per cluster On an Active/Passive cluster we can create one Recovery Storage Group per

Exchange Virtual Server (EVS) Mailbox Stores from standalone servers can be added to an Recovery

Storage Group on a clusters and vice versa provided they are in the same Administrative Group

No online maintenance will be run on recovery databases System and Mailbox policies are not applied to recovery databases Recovery Storage Groups cannot be renamed Recovery Databases cannot be backed up In order to successfully Exmerge out the data for a particular user, their

mailbox must exist on the same mailbox store as of the time of backup

To recover Public Folder databases, the alternate forest method is still required.

Note

Page 146: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 5

Recovery Storage Group uses There are two key scenarios where the Recovery Storage Group is useful. The first occurs when we need to recover a certain number of mail items without affecting the production environment. This may be an important piece of mail for a CEO or a mailbox that was removed due to an administrative error.

The second scenario occurs when we run into a critical problem with the production mailbox stores. A customer’s Mailbox Store or a particular log file may become corrupted, which results in a store that will not mount. A Service License Agreement may dictate that after a certain number of hours users must be able to send/receive mail. In this situation the Administrator would probably move out all databases and log files and then mount blank databases. The recovery of mail data could then begin offline.

The two most common scenarios can be broken down as follows:

Recover Mail items for a particular user A user contacts the IT helpdesk and needs a piece of data recovered that they deleted from their mailbox. The item is not available in the dumpster. The Administrator will then use the following steps to recover the mail item using a Recovery Storage Group:

Create a Recovery Storage Group on any server in the Admin Group. Add the mailbox store to the Recovery Storage Group.

Restore the data to the Recovery Storage Group server and mount the Mailbox Store.

Use Exmerge to export the data and then to import it directly into the user’s mailbox.

Dismount and delete Mailbox Store in the Recovery Storage Group. Delete the Recovery Storage Group.

Mount blank Mailbox Stores due to corrupt store The steps to recover in this manner are as follows:

Mailbox Store becomes corrupted and cannot be mounted. Move out all log files (assuming no other databases are mounted in the storage group) and the corrupted database files.

Mount blank databases. Users can log in and send/receive mail. Create a Recovery Storage Group on any server in the same Admin Group. Restore a backup of the Mailbox Store to the Recovery Storage Group

server. If you wish to replay any log files you must place them in the Transaction Log Path of the Recovery Storage Group.

When you initiate a hard recovery of the mailbox store, it will start replaying log files from the backup set (usually from a path that is specified at the time of restore) and then continue replaying log files from the Transaction Log Path.

When the mailbox store has been recovered, we are ready to switch it back to live storage group.

Dismount the Mailbox Store in the Recovery Storage Group and the Mailbox Store in the live Storage Group. Switch the databases with each other and then remount them again.

Page 147: Exchange Server Admin Troubleshooting

6 Module 4: Troubleshooting the Recovery Storage Group

We then use Exmerge to export/import the new data from the clean databases we created.

When the Exmerge is finished, dismount and delete the mailbox store in the Recovery Storage Group. Delete the Recovery Storage Group.

The reason for switching the databases with each other is so that we Exmerge the minimum amount of data. The blank mailbox stores will hold a relatively small amount of data compared to the live mailbox store so it makes sense to only Exmerge this data. This is, of course, a matter of preference and there is nothing to stop you Exmerging over the larger amount of data to the new databases.

Remember that when Exmerging large amount of data, lots of transaction logs are generated so there must be sufficient disk space on the server.

Page 148: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 7

Creating a Recovery Storage Group

Using Exchange Server 2003 System Manager we now have a new choice when creating Storage Groups on an Exchange 2003 server. As well as normal Storage Groups, we can now create Recovery Storage Groups.

The steps to create a Recovery Storage Group are as follows:

Right-click on the chosen server object in Exchange System Manager and choose New – Recovery Storage Group.

Page 149: Exchange Server Admin Troubleshooting

8 Module 4: Troubleshooting the Recovery Storage Group

The Recovery Storage Group Properties sheet will now be displayed. Here we can specify a name for the Recovery Storage Group. It is a good idea to use a name that clearly identifies it as a Recovery Storage Group as the icon in Exchange System Manager for a Recovery Storage Group and a normal Storage Group are the same.

The Recovery Storage Group must have a unique name on the server. It cannot have the same name as a normal Storage Group that is already in use on the server. The name of the Recovery Storage Group does not have to match the name of the Storage Group that hosted the Mailbox Store that we are about to restore.

Specify the Transaction log location and the System Path Location using the browse buttons. This location cannot be changed afterwards.

The Transaction log location and System path location should NOT be set to the same as an existing Storage Group on the Exchange 2003 server. To try and simplify the whole process, use the same paths for the transaction logs, system path and database files.

As almost no data is written to Mailbox Stores mounted in a Recovery Storage Group, there will be very few transaction logs generated. Therefore, there is no real performance gain by splitting the transaction logs and database files onto separate drives (This is contrary to normal best practices regarding normal Storage Groups in Exchange 2000 Server and Exchange Server 2003)

Page 150: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 9

When the Recovery Storage Group has been created, we can check its default settings by taking the properties of the newly created Recovery Storage Group.

Note that the Transaction log location and System Path Location cannot be changed. If you must change these locations, then you can simply delete the Recovery Storage Group, create a new one, and then specify the correct paths.

The Log File Prefix is set to R00 by default. This cannot be changed.

Circular Logging cannot be enabled on a Recovery Storage Group.

A Recovery Storage Group cannot be renamed. If you must use a different name for your Recovery Storage Group then you will have to delete it and recreate it.

Page 151: Exchange Server Admin Troubleshooting

10 Module 4: Troubleshooting the Recovery Storage Group

Adding a Mailbox Store to the Recovery Storage Group

Once the Recovery Storage Group has been created, Mailbox Stores that need to be recovered can be added to it.

In Exchange System Manager 2003, right-click on the Recovery Storage Group and choose Add Database to Recover. Notice that there is no option to create new databases in a Recovery Storage Group.

In a mixed Exchange 2000/2003 environment, you will need to be using Exchange System Manager 2003 in order to create and manage a Recovery Storage Group.

Page 152: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 11

Page 153: Exchange Server Admin Troubleshooting

12 Module 4: Troubleshooting the Recovery Storage Group

The Select database to recover dialog box will then be presented. Here we will be presented with a list of eligible Mailbox Stores. The criteria for eligible Mailbox Stores are as follows:

The Mailbox Store must exist in Active Directory. The Mailbox Store must be hosted on a server within the same

Administrative Group. The Mailbox Store must be from a server that is running a minimum of

Exchange 2000 Server SP3 and a maximum version no higher than the server where the Recovery Storage Group resides.

If a Mailbox Store already exists in the Recovery Storage Group, then only Mailbox Stores from the same original Storage Group will be presented.

Notice that the version of Exchange is quoted and the fact that databases from later versions of Exchange and versions earlier than Exchange 2000 Server SP3 will not be shown.

Choose the desired Mailbox Store and click OK.

Page 154: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 13

If we attempt to add a Mailbox Store to the Recovery Storage Group when there are no more eligible Mailbox Stores, then the Select database to recover will NOT be shown and we will see the following error:

This error basically means that we have previously added a Mailbox Store to the Recovery Storage Group and there are no more Mailbox Stores available from the same original Storage Group.

Check the contents of the Recovery Storage Group to see if you have a Mailbox Store there and remove it if necessary.

If there is no Mailbox Store in the Recovery Storage Group, have a look at the Recovery Storage Group object using LDP or ADSIEdit. If there are any Mailbox Stores listed there, delete them and try again.

Failing that, delete the Recovery Storage Group and try to add the Mailbox Store again.

Note

Page 155: Exchange Server Admin Troubleshooting

14 Module 4: Troubleshooting the Recovery Storage Group

The Mailbox Store Properties sheet will now be shown.

On the General tab notice the default settings. We must now give the Mailbox Store a name. This will always default to the display name of the Mailbox Store in the Active Directory. This can, however, be changed to whatever you like.

Notice which settings are disabled by default.

Page 156: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 15

On the Database tab we can see the default settings.

Notice the Exchange database and Exchange streaming database default settings. These database names are generated using the display name on the General tab. In some cases this will not match the databases names of the live Mailbox Store in production.

For example the default Mailbox Store on an Exchange 2003 server has the following properties:

Database Name: Mailbox Store (SERVERNAME) Exchange Database File: Priv.edb Streaming Database File: Pub.edb

If you were to add this Mailbox Store to a Recovery Storage Group, the default settings would create a Mailbox Store with the database names “Mailbox Store (SERVERNAME).edb” and “Mailbox Store (SERVERNAME).stm”

These database names do NOT have to match those of the live database in order for the restore to be successful. This matching is handled by an attribute called MsExchOrigMDB. We will look more closely at this attribute in the Active Directory attributes section later.

The Maintenance Interval cannot be set for a Mailbox Store in the Recovery Storage Group.

Mailbox Stores in the Recovery Storage Group will not mount at startup (or after failover in a cluster). This cannot be changed. By default “This database

Page 157: Exchange Server Admin Troubleshooting

16 Module 4: Troubleshooting the Recovery Storage Group

can be overwritten by a restore” is checked. It must remain checked for a successful restore.

Click OK when you are done.

Page 158: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 17

Active Directory Attributes

Now that we have created a Recovery Storage Group and added a Mailbox Store to it for recovery, we can take a closer look at the inner workings of the Recovery Storage Group and how the various parts fit together.

There are several new Active Directory attributes to identify the various components that make up the Recovery Storage Group.

MsExchMaxRestoreStorageGroups This is an attribute of the Information Store object on each Exchange 2003 server.

It controls the maximum number of Recovery Storage Groups that can be created on an Exchange 2003 server. It is an integer value and is set to 1 by default. If you try to create a second Recovery Storage Group you will receive the following error:

Page 159: Exchange Server Admin Troubleshooting

18 Module 4: Troubleshooting the Recovery Storage Group

This MsExchMaxRestoreStorageGroups attribute can be viewed using a tool like LDP or ADSIEdit.

The image below shows the new user interface (UI) for the Windows 2003 version of ADSIEdit.

This value can be seen by drilling down to the following location:

CN=InformationStore,CN=EXCHANGESERVER,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=ORG,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com

Note

Page 160: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 19

Be aware that this value can also be changed using one of these tools.

If this value were changed to 2, for example, then we would be able to create two Recovery Storage Groups on the server. However, when we try to perform a restore to this server, the restore would fail and the following Event ID would be generated in the application log:

If you are seeing more than one Recovery Storage Group on an Exchange 2003 server then this attribute has almost certainly been changed manually. Any extra Recovery Storage Groups should be deleted and this value should be set back to 1.

Note

Page 161: Exchange Server Admin Troubleshooting

20 Module 4: Troubleshooting the Recovery Storage Group

Lab 4.1: Create a Recovery Storage Group and review the Active Directory attributes

Setup SERVERS:

Z2 has Exchange Server 2000 SP3/domain controller

Z3 has Exchange Server 2003

Username: Administrator

Password: Password1

Objective: To understand the Active Directory attributes that make up the Recovery Storage Group.

Exercise 1 1. Log into Z3 using the credentials above.

2. Start Exchange System Manager.

3. Drill down to ORG - Administrative Groups - Hubsite – Z3- First Storage Group. Notice that we have only the default Mailbox Store and Public Folder Store on this server.

4. Right-click on the Z3 server object and choose New - Recovery Storage Group.

5. The Recovery Storage Group properties page now appears.

6. Leave the name as the default setting and also leave the Transaction Log Location Path and System Path Location as the default settings. Notice that the Enable Circular logging option is grayed out.

7. Click OK. We now have a Recovery Storage Group on Z3. Notice that you cannot change the name or location of the Recovery Storage Group after it has been created.

8. Right-click on the Recovery Storage Group and choose Add Database to Recover.

9. Notice that all Mailbox Stores in the Administrative Group are now displayed. Notice that the X500 Distinguished name of each of the Mailbox Stores are also displayed.

10. Choose the Mailbox Store (Z3) and then click OK. The properties page for the Mailbox Store to be added will now appear. Accept the default settings and click OK.

11. We now have Mailbox Store(Z3) in the Recovery Storage Group

12. Start ADSIEdit by clicking on Start, then Run. Type Adsiedit.msc in the field and click OK.

13. Drill down to the following location:

Page 162: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 21

Configuration - CN=Configuration - CN=Services - CN=Microsoft Exchange - CN=ORG - CN=Administrative Groups - CN=Hubsite - CN=Servers - CN=Z3 - CN=Information Store

14. Right-click on CN=Information Store and select Properties.

15. Locate the attribute MsExchMaxRestoreStorageGroups and double-click on it. Its value is 1. This attribute is used to determine the maximum number of Recovery Storage Groups that can be created on an Exchange 2003 server. It should not be changed from 1.

16. Drill down to the following location: Configuration - CN=Configuration - CN=Services - CN=Microsoft Exchange - CN=ORG - CN=Administrative Groups - CN=Hubsite - CN=Servers - CN=Z3 - CN=Information Store - CN=Recovery Storage Group

17. Right-click on CN=Recovery Storage Group and select Properties.

18. Locate the attribute MsExchRestore. Its value should be TRUE. This is used to identify the Storage Group as a Recovery Storage Group. If you were to check the attribute on an ordinary Storage Group, its value should be <NOT SET>.

19. In the right-hand pane, locate the Mailbox Store CN=Mailbox Store(Z3). Right-click on it and select Properties.

20. Locate the attribute MsExchRestore. Its value should be TRUE. This is used to identify the Mailbox Store as a recovery store that is part of an Recovery Storage Group.

21. Locate the attribute msExchOrigMDB. Its value should be set to the Distinguished Name (DN) of the original Mailbox Store (This was visible in Step 9). This attribute is used to link the recovery mailbox store back to the original Mailbox Store. When performing an online restore to the Recovery Storage Group, this DN must match the DN of the mailbox store on the media.

22. Close ADSIEdit.

23. In Exchange System Manager right-click on the Mailbox Store in the Recovery Storage Group and choose Delete. Click Yes and then OK.

24. Right-click on the Recovery Storage Group and choose Delete. Click Yes.

Page 163: Exchange Server Admin Troubleshooting

22 Module 4: Troubleshooting the Recovery Storage Group

MsExchRestore (on Storage Group) This is an attribute of all Storage Groups in Exchange Server 2003. The attribute is a Boolean value and possible values are TRUE, FALSE and Not Set.

The attribute is used to distinguish a Recovery Storage Group from a normal Storage Group. On a Recovery Storage Group the attribute is set to TRUE and on a normal Storage Group its value is Not Set. It can be seen by using either LDP or ADSIEdit.

If this value is changed to FALSE or NOT SET then the Recovery Storage Group will effectively be converted to a normal Storage Group. It is not recommended to change the values of Storage Groups in this manner.

Page 164: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 23

MsExchRestore (on Mailbox Store) This is a Boolean value attribute of all Mailbox Store objects in Exchange Server 2003. Possible values are TRUE, FALSE or Not Set.

The attribute is used to distinguish a Mailbox Store that is hosted in a Recovery Storage Group from a Mailbox Store that is hosted in a normal Storage Group. When a Mailbox Store object is created in a Recovery Storage Group this attribute is set to TRUE. The value is NOT SET on a Mailbox Store that is hosted in a normal Storage Group.

If this attribute is changed in any way on a recovery Mailbox Store, then the Recovery Storage Group may function unpredictably.

This is not an attribute of a Public Store.

Note

Page 165: Exchange Server Admin Troubleshooting

24 Module 4: Troubleshooting the Recovery Storage Group

MsExchOrigMDB This is a string value attribute of all Mailbox Store objects in Exchange Server 2003. A Mailbox Store that is hosted in a normal Storage Group will have the attribute set to <not set>. When a Mailbox Store is added to a Recovery Storage Group this attribute is set to the Distinguished Name of the original Mailbox Store.

This attribute is used to link the new Mailbox Store Active Directory object in the Recovery Storage Group back to its original Mailbox Store.

In order for an online restore to be successful, the MsExchOrigMDB attribute must match the distinguished name of the Mailbox Store in the backup set.

If no Mailbox Store is found in the Recovery Storage Group with a matching MsExchOrigMDB attribute then the restore will fail with a 9634 error.

Page 166: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 25

Overriding the Recovery Storage Group If there is no Recovery Storage Group present on an Exchange Server 2003 server, then restore behaviour is exactly the same as in Exchange 2000 Server.

If there is a Recovery Storage Group present on the server then all restores will default to mailbox stores that exist in the Recovery Storage Group. If a Recovery Storage Group is present on the server and you attempt to restore a mailbox store that is not part of the Recovery Storage Group you will receive the error shown below:

Page 167: Exchange Server Admin Troubleshooting

26 Module 4: Troubleshooting the Recovery Storage Group

This behavior can be overridden by setting the following registry key:

HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\Recovery SG Override DWORD value = 1

If this registry key is set to 1 then the Recovery Storage Group is ignored and restores will be targeted at the mailbox stores in the ordinary storage groups on the server.

The Information Store service does not have to be restarted for this registry setting to take effect. The Information Store will recognize its existence immediately.

This registry key would typically be used if an Administrator wishes to perform an online restore of one of his production databases without disturbing the databases in the Recovery Storage Group.

Note

Page 168: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 27

Restoring the data

Once the Recovery Storage Group has been created and the mailbox store added to it, the restore process can begin.

When starting the backup client software it is important to remember that the client software must be pointed to the Exchange 2003 server that contains the Recovery Storage Group with the placeholder Mailbox Store object in it.

Using the Recovery Storage Group we can perform an online restore of the Mailbox Store using an online backup set and a backup client like NTBACKUP or an offline restore using existing .edb and .stm files.

Restoring an online backup set to a Recovery Storage Group The steps to restore data to a Recovery Storage Group using an online backup set are as follows:

1. Create a new Recovery Storage Group on any server in the same Administrative Group.

2. Add the desired Mailbox Store(s) to the Recovery Storage Group. 3. Before restoring from backup, dismount and delete any Mailbox Stores

currently mounted in the Recovery Storage Group. Remove any transaction log files (*.LOG) and checkpoint files (*.CHK) from the Recovery Storage Group to prevent them interfering with recovery.

4. Restore a full backup set to the Recovery Storage Group server. If you will not be restoring additional differential or incremental backups or adding additional log files for replay, you can set hard recovery to run automatically. In NTBACKUP this is done by checking the Last Backup Set checkbox.

5. If desired, restore incremental or differential transaction log backups. Set the Last Backup Set as appropriate.

6. If desired, add additional transaction log files to be replayed to the Recovery Storage Group transaction log folder. Verify that the sequence and

Page 169: Exchange Server Admin Troubleshooting

28 Module 4: Troubleshooting the Recovery Storage Group

signatures of these log files match those in the backup set. Use the ESEUTIL /ml command to determine the signature of the log files.

7. Verify that hard recovery has completed successfully by running the ESEUTIL /mh command against the .edb file. The header should contain a line called State: Clean Shutdown. If Dirty Shutdown is displayed, then hard recovery has not been initiated. To initiate a hard recovery, use a command prompt and go to the folder containing the restore.env file and issue the following command: ESEUTIL /cc

8. Verify that This database can be overwritten by a restore is checked and then mount the Mailbox Store.

Page 170: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 29

When the Mailbox Store is mounted, all mailboxes will appear as disconnected

Note that there is no Full Text Indexing node on Mailbox Stores that are mounted in a Recovery Storage Group.

The Mailboxes cannot be reconnected to Active Directory user objects.

Page 171: Exchange Server Admin Troubleshooting

30 Module 4: Troubleshooting the Recovery Storage Group

In the picture below we see the files that are restored to the server.

Notice that we have an E00.log and E00.chk file in this folder. This is the log file prefix of the original Mailbox Store.

The log files from the E00 series are first replayed into the database. Then the database is detached from the E00 log sequence and is then attached to the Recovery Storage Group R00 log file sequence. The E00.log files and E00.chk files are not removed from the folder.

You do not need to remove these files for the Recovery Storage Group to function correctly.

Page 172: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 31

Restoring offline file copies to a Recovery Storage Group The steps to restore data using offline copies of an .edb and .stm file are as follows:

1. Verify that all the database files to be restored are in a Clean Shutdown state by using the ESEUTIL /mh command. Remember that each .edb database must be accompanied by its matching .stm streaming database file.

2. If the database files are not in Clean Shutdown state, it will be necessary to do soft recovery of each database. This means you will have to use ESEUTIL to replay at least one transaction log file into the database before it can be mounted.

3. If required log files are unavailable the only way to bring the database to a useable state is to use the repair functionality in ESEUTIL (/P start-up switch). Before repairing a database make sure to take offline copies of the files. Then use ISINTEG to fix bad references in the database caused by forcing the database to Consistency without applying all transactions. If you have to repair a database, there will be some data loss. This data loss can be catastrophic, although more frequently the loss is minimal.

4. Copy the .edb and .stm files into the correct location defined in the Recovery Storage Group and make sure that the file names match those defined on the Database tab of the Mailbox Store in the Recovery Storage Group.

5. Dismount any databases currently running in the Recovery Storage Group and remove all transaction log files (*.LOG) and checkpoint files (*.CHK)

6. If you do not intend to replay transaction logs, you may skip this step. Copy necessary transaction logs to the same location as the database files. Truncate the last five characters of the log file with the highest sequence number. For example, if the log is E0001234.LOG, rename it E00.LOG.

7. If you do not intend to replay transaction logs, you may skip this step. To replay transaction logs, open a command prompt to the folder where the database files have been restored. All transaction log files should be in this folder as well. Run the command ESEUTIL /R Enn /I /D to complete soft recovery. Replace Enn with the filename you used for the log file in the step above. In this example, it would be E00. After soft recovery finishes, verify that the database files are in Clean Shutdown state by running ESEUTIL /MH [database filename].EDB.

8. Verify that the checkbox is set for “This database can be overwritten by a restore” for each newly recovered database, and then mount the database.

Page 173: Exchange Server Admin Troubleshooting

32 Module 4: Troubleshooting the Recovery Storage Group

When the Mailbox Store is mounted, all mailboxes will appear as connected. This is a little misleading as they are, in fact, disconnected. This is a known issue with the Recovery Storage Group and will not be fixed. It does not affect functionality of the Recovery Storage Group in any way.

Page 174: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 33

Recovering Exchange 2000 Server Mailbox Stores

An Exchange 2000 Server Mailbox Store must be at least SP3 to be added to an Recovery Storage Group for recovery.

When adding an Exchange 2000 Server SP3 Mailbox Store to a Recovery Storage Group you will be prompted as follows:

6803 is the build of Exchange Server 2003 used in this example. The Release To Manufacturing build is 6944.

In this scenario we will be able to restore our Exchange 2000 Server SP3 Mailbox Store to the Recovery Storage Group successfully. We will then be able to mount the Mailbox Store in the Recovery Storage Group on the Exchange 2003 server. Once we have mounted this Mailbox Store a database upgrade will take place on the .edb and .stm files. This will mean that you may not be able to mount this database back on the Exchange 2000 SP3 server after this operation.

During the restore process you can specify to perform a hard recovery (Last Backup Set). This will effectively bring the database to a consistent state (Clean Shutdown). During this process the Recovery Storage Group server will perform a partial mount of the database which will initiate the database upgrade. So even if you choose not to physically mount the Mailbox Store on

Note

Page 175: Exchange Server Admin Troubleshooting

34 Module 4: Troubleshooting the Recovery Storage Group

the Recovery Storage Group server, you still may not be able to move the database files back to the Exchange 2000 server.

It is recommended to Exmerge the data back from the Recovery Storage Group server to the Exchange 2000 SP3 server.

Page 176: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 35

Exmerge

Exchange Server 2003 will include a new version of Exmerge which is used to salvage data from Mailbox Stores that are mounted in a Recovery Storage Group. Earlier versions of Exmerge cannot be used for this purpose.

Exmerge uses a new method to log onto recovery databases. It performs an “Admin logon” which basically will view the Mailbox table of the database. It will see a list of mailboxes and will then match these mailboxes to the corresponding Active Directory user objects by matching the Mailbox GUID it sees in the database to a Mailbox GUID in the Active Directory.

If it cannot match this Mailbox GUID to an Active Directory object, then it will skip the mailbox and an error will be generated in the exmerge.log file. We will look at the cause of this later.

The first step of the Exmerge process is to extract the data into .pst files. The process can be broken down as follows:

1. The Exmerge Splash Screen:

Page 177: Exchange Server Admin Troubleshooting

36 Module 4: Troubleshooting the Recovery Storage Group

2. Choose a one-step or two-step procedure (we will look at the two-step procedure).

3. Step 1: Extract the data from the recovery database.

Page 178: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 37

4. Specify the Exchange Server, Domain Controller and LDAP port.

Page 179: Exchange Server Admin Troubleshooting

38 Module 4: Troubleshooting the Recovery Storage Group

5. A list of Mailbox Stores on the specified Exchange server will be presented. Choose the Recovery Storage Group/Mailbox Store and click Next

6. A list of mailboxes on the specified store will now be presented. Select the desired mailbox(es) and then click Next.

Page 180: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 39

7. Choose the correct locale and then click Next.

8. Specify a folder on the server where the .pst files will be created. A network share can be used but this will have a performance impact on the recovery. Whenever possible, use drive space on the local server.

Page 181: Exchange Server Admin Troubleshooting

40 Module 4: Troubleshooting the Recovery Storage Group

9. Here you choose to save the various settings in Exmerge. Click Next.

10. We can review the number of successes and failures.

At this point we have successfully exported the data from the mailbox in the Recovery Storage Group out to a .pst file. The next step is to import the data back into the production Mailbox Store.

If you encounter any failures during the initial data export, the best place to start is the Exmerg.log file. This file will be located in the same folder as the Exmerge.exe file.

Page 182: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 41

Once we have the data exported to .pst files, we can then import the data into the production Mailbox Store. The Exmerge import process is as follows:

1. Set the correct permissions on the target Mailbox Store. By default, Administrator(or the account used to install Exchange), Domain Admins and Enterprise Admins have an inherited Deny and an inherited Allow on Send As and Receive As on mailbox stores. This effectively results in a Deny on the Mailbox Store. In order to successfully import data to a Mailbox Store, we either have to override these permissions or use an account that does not have a Deny. The default permissions for a Mailbox Store in Exchange Server 2003 are as follows:

The Administrator has an inherited Deny for Receive As and Send As. Domain Admins and Enterprise Admins will also have the same inherited Deny. If we were to attempt to import data using the Administrator account into a user’s mailbox on this store with these permissions set, it would fail and we would see the following on the Exmerge.log file:

[21:53:44] Error opening message store (MSEMS). Verify that the Microsoft Exchange Information Store service is running and that you have the correct permissions to log on. (0x8004011d)

Page 183: Exchange Server Admin Troubleshooting

42 Module 4: Troubleshooting the Recovery Storage Group

If you are seeing this error on the Exmerge.log file, then you need to check that the account you are using has Send As and Receive As.

Page 184: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 43

In order to override these inherited, Denies we can set explicit Allows as shown in the image below:

These explicit Allows take precedence over inherited Denies. Remember that the explicit Allows must be set for the user and all groups that the user is a member of. In this case we would have to set explicit Allows for Domain Admins and Enterprise Admins.

When the permissions have been set up correctly, we can continue with the import.

Page 185: Exchange Server Admin Troubleshooting

44 Module 4: Troubleshooting the Recovery Storage Group

2. Choose Step 2: Import the data.

3. Select the database that we are going to import the data into. Notice that Recovery Storage Groups are not visible when importing data.

Page 186: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 45

4. Choose the desired mailboxes and then click Next.

5. Choose the correct locale and click Next.

Page 187: Exchange Server Admin Troubleshooting

46 Module 4: Troubleshooting the Recovery Storage Group

6. Choose the directory that contains the .pst files and then click Next.

7. Click on the Save Settings button and then click Next.

Page 188: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 47

8. The import process completed successfully.

If you have any issues during the import process,s then Exmerge will display as follows:

In this situation, the first place to check is the Exmerge.log file. The log file will offer more detailed information on what went wrong.

The most common error here is incorrect permissions. Check the security on the target mailbox store and try the import again.

Page 189: Exchange Server Admin Troubleshooting

48 Module 4: Troubleshooting the Recovery Storage Group

Known issues with Exmerge

An issue you may run into when trying to extract data from a Recovery Storage Group using Exmerge is when Exmerge cannot match the Mailbox GUID from the database with a user object in the Active Directory. The following error will be shown during the extraction phase:

In the Exmerge log file, we will see the following:

[22:55:33] Error! Cannot identify the user with the msExchMailboxGuid \2F\7B\BD\86U\C7\5CJ\9F\20\BB\24\93\EB\10\08. The legacyExchangeDN is /O=X-FOREST/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=USERY.

There a couple of reasons we may see this error:

1. The user account has been deleted from Active Directory. If the user account is deleted, then Exmerge will obviously not be able to match the Mailbox GUID from the database to an Active Directory object. You will therefore not be able to extract information from this mailbox using Exmerge.

2. The user’s mailbox has been moved to another Mailbox Store. When a user’s mailbox is moved, it effectively creates a new mailbox in the target store, moves the data, and then deletes the original mailbox. This will result in the user getting a new Mailbox GUID. Exmerge will not be able to match the Mailbox GUID in the database to a Mailbox GUID in Active Directory.

Page 190: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 49

There are a couple of workarounds to allow us to get the data from the mailboxes in this scenario.

The first workaround is to perform a “victim restore” of the mailbox store to another Exchange 2003 server in the same Admin Group. Once the mailbox store has been successfully restored to another server, we can then reconnect the required mailboxes to Active Directory objects and then use Exmerge to extract the data.

See the following article for a more detailed look at the process:

324127 XADM: How to Restore an Information Store Database to a Server That Resides in the Same Active Directory Forest http://support.microsoft.com/?id=324127

The second workaround would be implemented if the customer has already performed the restore of the data using a Recovery Storage Group. Let us assume the customer has restored a fairly large database in order to recover some data from a mailbox that has since been deleted. The customer then discovers that he cannot Exmerge out the data due to the Mailbox GUID mismatch problem. We really do not want to tell the customer to perform a new “victim restore” as that will obviously take a fair amount of time.

We have the mailbox store in the Recovery Storage Group and it should be consistent. We can create a new Storage Group on the Exchange 2003 server and then create a Mailbox Store in this Storage Group. Make sure that the Mailbox Store that is created has the same database names as our restored Recovery Database. We can then move the database files from the Recovery Storage Group into the Storage Group, mount the mailbox store, reconnect the necessary mailboxes and Exmerge out the data.

Page 191: Exchange Server Admin Troubleshooting

50 Module 4: Troubleshooting the Recovery Storage Group

Exercise 2: Recovery Storage Group Scenario 1

1. User1 needs a piece of mail or their entire mailbox recovered.

2. Create a Recovery Storage Group on any server in the Admin Group AG1.

3. Add Mailbox Store MBX1 to Recovery Storage Group.

4. Start backup client and restore valid backup of MBX1. (The backup client will need to be pointed to the server containing the Recovery Storage Group). Check the ”Last Backup Set” as appropriate.

5. Mount MBX1 in the Recovery Storage Group. (All users appear as disconnected.)

6. Exmerge out User1’s data (from Recovery Storage Group to PST).

7. Exmerge in User1’s data (from PST to live Mailbox Store).

8. Done. Dismount and delete MBX1 in the Recovery Storage Group. Delete the Recovery Storage Group.

Objective: To recover a single mail item using a Recovery Storage Group.

Exercise 2 Detailed Steps SERVERS:

Z2 has Exchange Server 2000 SP3/domain controller (power this one completely before other virtual machines).

Z3 has Exchange Server 2003.

Username: Administrator

Password: Password1

Overview of Exercise 2

Page 192: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 51

1. Log into the Z3 server using the above credentials. 2. Start Active Directory Users and Computers on Z2. 3. Drill down to the Users container 4. Create a new user in this container. Username: exercise2. Choose a

password that meets the complexity requirements. Uncheck User must change password at next logon. Click Next.

5. Create a mailbox on ORG/Hubsite/Z2/First Storage Group/Mailbox Store(Z2). (Note that you are not selecting the default store.) Click Next and then Finish.

6. Wait for the Recipient Update Service to stamp a proxy address on the user object.

7. Start Internet Explorer and go to the following location: http://Z2/exchange/exercise2

8. Log in using the credentials MS\exercise2 and the password you chose in step 4.

9. Create a new mail item. Address it to the recipient Exercise2. In the subject line type IMPORTANT. Set the mail as high importance.

10. The mail should now be in your Inbox. 11. We are now going to make a backup of the Mailbox Store. Click on

Start, then Run. Type Ntbackup in the field and click OK. 12. Choose the Backup tab. Drill down to Microsoft Exchange Server - Z2

- Microsoft Information Store - First Storage Group and check First Storage Group.

13. Set the backup media or filename to c:\exercise2.bkf and click Start Backup.

14. The Backup Job Information page appears. Click Start Backup. 15. When the backup is complete, click Close. 16. Go back to the Exercise2 mailbox in Outlook Web Access. Right-click

on the mail IMPORTANT and choose delete. 17. Right-click the Deleted Items Folder and choose Empty Deleted Items.

Click OK to confirm. 18. Click on the Recover Deleted Items icon on the right-hand pane and

then choose Permanently Delete. Click OK to confirm. 19. Close the Delete Items dialog box. The mail has been completely

deleted now. 20. Create some new mail items addressed to Exercise2. 21. Using Exchange System Manager we are now going to create an

Recovery Storage Group in order to recover this mail item 22. Start Exchange System Manager on Z3. Create a new Recovery

Storage Group on Z3, but then add the Mailbox Store (Z2) to this Recovery Storage Group. Note that a window appears, which warns that any restored Exchange 2000 Server SP3 databases will be upgraded to Exchange Server 2003’s engine if mounted in the Recovery Storage Group. Proceed by clicking OK.

23. When the Mailbox Store has been added to the Recovery Storage Group, we can now perform the restore.

Page 193: Exchange Server Admin Troubleshooting

52 Module 4: Troubleshooting the Recovery Storage Group

24. Go back to NTBACKUP and choose the Restore tab. 25. In the left-hand pane, expand the following: File - Media Created -

Z2\Microsoft Information Store\First Storage Group 26. Select checkboxes on all databases in the First Storage Group. 27. Click on Start Restore. 28. Make sure the ‘Restore to’ is set to Z3. Set the Temporary location for

log and patch files to c:\templogs. Check the Last Backup Set checkbox. Then click OK, and then OK again. (Note: If it says “unable to create logfiles,” be sure to remove the UNC prefix before the server name, then retry.)

29. The restore will now begin. When the restore is complete, you should see that it completed with errors.

30. Review the 9635 error in the applog. It clues you in on the fact that the public folder store is not present in Active Directory, underneath the Recovery Storage Group. (This is due to the fact that only mailbox stores may be added to Recovery Storage Groups.)

31. Repeat Steps 24-28, but modify step 26 by selecting ONLY the checkboxes for ‘Log Files’ and ‘Mailbox Store(Z2)’.

32. Restore should now work. Review the report to confirm that there are no errors, and then close ntbackup.

33. Open an Explorer Window and drill down to the location you specified for your Recovery Storage Group (it will usually be c:\program files\exchsrvr\recovery storage group).

34. Notice that the files have now been restored to this location. 35. Go to Exchange System Manager and go to the properties of the

Recovery Storage Group’s mailbox store. 36. Go to the Databases tab, and select “This database can be overwritten

by a restore.” (Note: This step would not be necessary if you had not failed the partial restore and received errors in step 29.)

37. Mount the Mailbox Store in the Recovery Storage Group. Click Yes to the warning. Click OK when the store has mounted.

38. Drill down to the Mailboxes node under the Mailbox Store. (You may need to refresh the view). Notice that all mailboxes are disconnected, and that you cannot use “Exchange Tasks” upon them.

39. Copy Exmerge.exe and Exmerge.ini from the lab location (or mounted .ISO image) into the c:\program files\exchsrvr\bin folder. Ask your instructor if you cannot locate the .ISO image. Alternatively, if you have access to the internet from the virtual machine, navigate to www.microsoft.com/exchange, go to Downloads, and then Tools, and then you can download the Exchange Server 2003 version of Exmerge from there.

40. Create a folder called c:\exmerge on z3. This will be used to hold the .pst file(s).

41. In Exchange Server 2003, by default the Administrator account, Domain Admins and Enterprise Admins have an inherited Allow and Deny on Receive As and Send as for Mailbox Stores. This is an effective Deny. We need to remove this in order to be able to log in to Exercise2 mailbox and import the mail item.

Page 194: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 53

42. Use Exchange System Manager and drill down to the server Z2. Take properties on the Mailbox Store(Z2) and select the Security tab.

43. Click on the Advanced tab and uncheck the Allow Inheritable permissions checkbox.

44. Click Copy and then OK and then Yes. 45. For Administrator, Domain Admins and Enterprise Admins, make sure

that they have only Allow checked for Receive As and Send As. Click Apply and then OK

46. We are now ready to recover the mail item using Exmerge. Start Exmerge.exe from the bin folder.

47. Click Next. 48. Choose Extract and Import (One Step Procedure) and click Next. 49. For the Exchange Server name, type Z3 (this is the server with the

Recovery Storage Group) and type Z2 for the domain controller. 50. Click the Options button. 51. Accept the default settings on the Data tab. 52. Click the Message Details tab. In the Enter new message subject tab

type IMPORTANT and then click Add. Then click OK. 53. Click Next. 54. In the Destination Exchange Server, type Z2 and click Next. 55. Choose the Recovery Storage Group/Mailbox Store (Z2) and click

Next. (Note: You will not get the Recovery Storage Group option with previous versions of Exmerge.)

56. Select the ‘Exercise2’ mailbox and then click Next. 57. Click Next to accept the default locale. 58. Select the folder c:\exmerge and click Next. 59. Click Next to start the import. 60. Click Finish. 61. Go to the Exercise2 mailbox in Outlook Web Access and the mail item

IMPORTANT should be back in place. 62. We can now dismount and delete the Recovery Storage Group mailbox

store. Use Exchange System Manager to dismount the Mailbox Store in the Recovery Storage Group. Then delete the Mailbox Store.

63. Then Delete the Recovery Storage Group from Exchange System Manager

64. Delete the directory c:\program files\exchsrvr\recovery storage group. 65. We now have to reset the permissions on the live mailbox Store.

Locate the Store, select Properties and then choose the Security tab. 66. For Administrator, Domain Admins and Enterprise Admins check both

the Allow and Deny checkboxes for Receive As and Send As. 67. Then check the “Allow inheritable permissions” checkbox. 68. Click Apply. You will receive a warning that the properties will not be

visible until the next time you open it. Click OK and then OK again.

Page 195: Exchange Server Admin Troubleshooting

54 Module 4: Troubleshooting the Recovery Storage Group

Exercise 3: Recovery Storage Group Scenario 2

Objective: To simulate a store crash, and recover an entire Mailbox Store while allowing users to send/receive e-mail using a “dialtone” (temporary empty swap) database.

Overview of Scenario 2 Recovery Storage Group Restore 1. MBX store1 becomes corrupted and will not mount. Log files are intact. Other databases in the Storage Group are okay. 2. Move out corrupted databases and mount blank (dialtone) databases. Users can log in and start working immediately.

3. OST files are destroyed (since their keys were in the corrupted DB). Presently, there is workitem in Outlook 2003 to stop this from happening and redirect user to OWA instead, but these are just plans and not final.

4. Create an Recovery Storage Group on any server in the same AG and add MBX1 to it

5. Copy over required log files from original server to the Recovery Storage Group.

6. Start backup client and restore valid backup of MBX1. (The backup client will need to be pointed to the server containing the Recovery Storage Group). Check the ”Last Backup Set” as appropriate.

7. Mount MBX1 in the Recovery Storage Group. All users appear as disconnected

8. Dismount both copies of MBX1. Both databases will be set as ”Clean Shutdown”. Switch the databases with each other and remount them. Old OST keys now recovered.

9. Exmerge over the restored data to the live database

10. Exmerge the ”new” data from the Recovery Storage Group MBX1 (previous dialtone) back into the recovered MBX1. This should be relatively little data.

11.Done. Dismount and delete MBX1 from the Recovery Storage Group

Page 196: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 55

Details for Exercise 3: Disaster recovery after a Store Crash using the Recovery Storage Group

Setup SERVERS:

Z2 has Exchange 2000 SP3 Server/DC

Z3 has Exchange 2003 Server

Username: Administrator

Password: Password1

1. Log into Z3 using the credentials above and start ESM 2. Start ADUC and create a user called Ex3user. Give the user a password

that meets the complexity requirements and create a mailbox on Z3 3. Log in to the Ex3user mailbox using the url http://Z3/exchange/

Ex3user 4. Create a mail and address it to your self. In the subject put "Pre Crash".

Attach the file c:\Exchange Server Setup Progress.log and send it. 5. The mail should now be in your Inbox 6. We are now going to backup the Mailbox Store. Start NTBackup (Start

- Run - NTBACKUP) 7. Click on the backup tab. Drill down to Microsoft Exchange Server - Z3

- Microsoft Information Store - First Storage Group. Select the entire storage group and specify the file c:\precrash.bkf. (Note: If you still see “Recovery Storage Group” then you will need to restart NTBackup)

8. Click on the Start Backup button and once more to confirm. 9. When the backup has completed go back to the Ex3user mailbox in

OWA. Create a new mail to Ex3user and in the subject type "Pre Crash - Post Backup". Again attach the progress log. When this is done click on Send

10. We are now going to simulate a Store crash. Open a command prompt and type the command TLIST

11. Note the PID of the Store.exe process 12. In the command prompt type the following command: "kill PID /f"

without the quotes (where PID is the PID of the Store process). The Store process should now be killed.

13. Normally when we try to start the Information Store process a soft recovery will be initiated. We will assume that the Mailbox Store cannot be mounted. We will then mount a blank mailbox store for Ex3user to log in and work.

14. Open ESM and drill down to the First Storage Group. Take properties on the Mailbox Store and set it to "Do not mount at startup" on the database tab.

Page 197: Exchange Server Admin Troubleshooting

56 Module 4: Troubleshooting the Recovery Storage Group

15. Repeat the previous step, but except change it on the Public Folder Store. We will assume that the customer does not care for immediate public folder access, so we will not cover this as the pub may be repaired on another server, or on the same server at a later time.

16. Open an explorer and drill down to c:\program files\exchsrvr\mdbdata. Move all the files (except for pub.edb & pub.stm) to c:\temp (create the folder if necessary)

17. Start the Information Store service using the command "net start msexchangeis"

18. Go to ESM and mount the Mailbox Store. You will be warned that one of the database files is missing. Click Yes to accept this.

19. Go back to Ex3user’s mailbox and confirm that the mailbox is empty. 20. Create a new mail item, address it to Ex3user and attach the progress

log file(s) again. In the subject type "Post Crash - Blank Store" 21. At this point our user can send and receive mail. For the next steps, we

must create an Recovery Storage Group, restore from our backup set and also replay any log files that we moved in to c:\temp

22. Using ESM create an Recovery Storage Group on the server and add the Mailbox Store(Z3) to it.

23. Go to c:\temp and copy the following files into "c:\program files\exchsrvr\Recovery Storage Group": E00.log & any other log file from the E00xxxxxx.log range (Do not move Res1.log and res2.log. They are not required)

24. Start NTBACKUP and choose the restore tab 25. Drill down through the media we created in the earlier step. Select the

Mailbox Store and the Log Files only. 26. Click on Start Restore. 27. Leave the Restore To: field as default. Specify c:\templogs as the

temporary location. As we want to perform a hard recovery afterwards we should check the Last Backup Set.

28. Click OK to start the restore. Click OK again to confirm. After the restore is complete it may take a while before hard recovery completes successfully.

29. Look in the Application Log for an ESE 902 event. This signifies that the hard recovery succeeded

30. The Mailbox Store in the Recovery Storage Group is now at a consistent state.

31. We are now going to dismount the live Mailbox Store and switch the database files with each other.

32. Go to ESM and dismount the nearly-empty Mailbox Store 33. Open an explorer and drill down to c:\program files\exchsrvr\mdbdata. 34. Move priv.edb & priv.stm to a temporary location (Perhaps a new

directory named something like c:\cleanshutdown-almostempty) 35. Move the files "Mailbox Store (Z3).edb & .stm from c:\program

files\exchsrvr\Recovery Storage Group to c:\program files\exchsrvr\mdbdata (Notice that the database names are not the same by default)

Page 198: Exchange Server Admin Troubleshooting

Module 4: Troubleshooting the Recovery Storage Group 57

36. Once the files are in the mdbdata folder we need to rename them back to priv1.edb & priv1.stm

37. Use ESM and take properties on the live Mailbox Store. On the database tab check the "This database can be overwritten by a restore"

38. We can now mount the Mailbox Store 39. The Ex3user user should now be able to see the mail items prior to the

crash in his mailbox 40. The Post crash mail should not be visible 41. We now have to move the database files from the temporary location

(created in step 34) into c:\program files\exchsrvr\Recovery Storage Group.

42. Clean-up any other residual files in the Recovery Storage Group directory so that only the EDB and STM remain.

43. Rename the edb and stm to the names expected by the non-Recovery Storage Group store object in AD. (Typically Mailbox Store (Z3).edb & .stm)

44. We can now mount the Mailbox Store in the Recovery Storage Group 45. The Mailbox Store in the Recovery Storage Group is now mounted.

We will now use Exmerge to merge over the post crash data into the live store.

46. Use the same steps as outlined in Scenario 1 to use Exmerge out the data from the Recovery Storage Group to the live store.

47. When you are done, dismount and delete the Recovery Storage Group.

Page 199: Exchange Server Admin Troubleshooting

58 Module 4: Troubleshooting the Recovery Storage Group

Acknowledgments

Microsoft Employee Simon Walsh Michael Lee

Page 200: Exchange Server Admin Troubleshooting

Contents

Resource Dependencies 1

Cluster Service Account Permissions 5

MsExchange_NodeState 9

DNS registration/Kerberos 12

AntiAffinityClassNames 16

Mount Point Drives 22

Creating an Exchange Virtual Server 33

Upgrading an Exchange Virtual Server to Exchange 2003 56

Removing an Exchange Virtual Server 64

Lab 5.1 : Clustering 88

Module 5: Clustering

Page 201: Exchange Server Admin Troubleshooting

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 202: Exchange Server Admin Troubleshooting

Module 5: Clustering 1

Resource Dependencies

In an Exchange 2000 cluster, we need to create a new Cluster Group to house the Exchange Virtual Server. In order to successfully create a System Attendant Resource, we must first have a physical disk resource, an IP address, and a Network Name in that group.

When we create the System Attendant resource, the other Exchange resources will be automatically created. During the creation process, a dependency tree will be created. The dependency tree is shown below.

Page 203: Exchange Server Admin Troubleshooting

2 Module 5: Clustering

The Information Store resource has five dependencies: SMTP, HTTP, POP, IMAP and Microsoft Search service. The message transfer agent (MTA) and Routing Engine resources are directly dependant on the System Attendant. In the event of a failover, all resources that have a dependency must go offline before the resource that it is dependant on them can attempt to go offline.

In the scenario above the SMTP, HTTP, IMAP4, POP3 and Microsoft Search service must successfully go offline (or fail) before the Information Store resource can attempt to go offline. The MTA and Routing Engine resources can attempt to go offline immediately, as they do not have any resources that are dependant on them.

Traditionally in Exchange 2000 clusters, the SMTP and the Information Store resources took the longest amount of time to go offline/come online. This could be attributed to large SMTP queues or mounting/dismounting large databases. This obviously will lead to longer failover times as the Information Store resource has to wait for the SMTP resource to go offline before it can attempt to go offline/come online.

Exchange 2000 Resource Dependency Tree

Page 204: Exchange Server Admin Troubleshooting

Module 5: Clustering 3

In Exchange Server 2003, the resource-dependant tree has been altered so that all Exchange 2003 cluster resources are now directly dependant on the System Attendant resource.

Here we see that all the Exchange related resources are now directly dependant on the System Attendant. This effectively means that the SMTP (and other protocol resources) can now be brought online/go offline in parallel with the store. This makes for faster failovers of the Exchange Virtual Server.

During the creation of the Exchange Virtual Server process, the correct dependencies will be set.

The POP3 and IMAP4 resources are not created by default. If they are created manually, then you will need to set a dependency on the System Attendant (this is mandatory).

During an upgrade of an Exchange 2000 Exchange Virtual Server, the resource dependencies will be changed to the new Exchange 2003 resource dependency tree. From the “Exchange Server Setup Progress.log” file we can see these changes being made. Open the log file and search for ScUpgradeResourceDependencies. Here we will see each resource being changed.

An SMTP resource being changed from the progress log:

Resource Dependency Tree in Exchange 2003

Note

Page 205: Exchange Server Admin Troubleshooting

4 Module 5: Clustering

[08:36:54] Entering ScUpgradeResourceDependencies [08:36:54] Checking dependencies of resource 'SMTP Virtual Server Instance - (EVS-01)' [08:36:54] Entering ScChangeResourceDependency [08:36:54] About to change resource dependency for resource 'SMTP Virtual Server Instance - (EVS-01)' [08:36:54] Leaving ScChangeResourceDependency

You will see the above entries for all Exchange resources that are upgraded to Exchange 2003.

Page 206: Exchange Server Admin Troubleshooting

Module 5: Clustering 5

Cluster Service Account Permissions

Related articles/bugs:

329702.KB.EN-US In order to successfully create, delete or modify an Exchange 2000 Exchange Virtual Server, the Windows 2000 cluster service account required “Exchange Full Administrator” permissions at the organization level if it was the first Exchange Virtual Server in the org. If it was not the first Exchange Virtual Server in the org then it required Exchange Full Administrator on the Admin Group that it was being installed into.

Page 207: Exchange Server Admin Troubleshooting

6 Module 5: Clustering

The Exchange Virtual Server creation process (shown above) can be broken down as follows:

1. User DOMAIN\Administrator logs in to one of the Nodes and starts Cluster Administrator (cluadmin.exe). The process cluadmin.exe runs as the currently logged in user (DOMAIN\Administrator). The Administrator then attempts to create a new Exchange System Attendant. Excluadmin.dll will gather information from Active Directory in order to create the System Attendant (e.g. Org name and Administrative Group name etc). The user DOMAIN\Administrator needs permissions to read from the configuration partition of the Active Directory.

2. When excluadmin.dll has collected the necessary information, it will then pass the information to exres.dll. Exres.dll is the Exchange resource dll. Exres.dll runs in the Resource Monitor process, which runs in the context of the Cluster Service Account.

3. Exres.dll will then load exsetdata.dll in order to create the objects in Active Directory. Exsetdata.dll also runs in the Resource Monitor process.

4. Exsetdata.dll will then create the necessary objects in the Active Directory. As Exsetdata.dll runs in the context of the Cluster Service Account, this account will require Full Exchange Administrator permissions in order to create the objects successfully.

Permission requirements in Exchange 2000

Page 208: Exchange Server Admin Troubleshooting

Module 5: Clustering 7

In Exchange 2003 the permissions have changed in order to remove this requirement. Any person or application that runs as the Windows 2000 cluster service account essentially has the ability to destroy an Exchange 2000 organization.

The Exchange 2003 permissions requirements are as follows:

In the Exchange 2003 the Exchange Virtual Server creation process can be broken down as follows:

1. The user DOMAIN\Administrator logs in to a Node in the cluster and starts

Cluster Administrator (cluadmin.exe). This process runs in the context of DOMAIN\Administrator. The Administrator then attempts to create a new Exchange System Attendant resource. Excluadmin.dll will gather information from Active Directory in order to create the System Attendant (e.g. Org name and Administrative Group name etc). The user DOMAIN\Administrator will need to permissions to read from Active Directory for this operation to be successful.

2. When excluadmin.dll has collected the necessary information, it will then load Exsetdata.dll directly. Exsetdata.dll runs in the same process as Excluadmin.dll (DOMAIN\Administrator).

3. Exsetdata.dll will then create the objects in Active Directory. As exsetdata.dll runs in the context of DOMAIN\Administrator, it is this account that requires the Exchange Full Administrator permissions to the configuration partition of Active Directory.

Permissions requirements in Exchange 2003

Page 209: Exchange Server Admin Troubleshooting

8 Module 5: Clustering

After an Exchange 2000 Exchange Virtual Server has been successfully upgraded to Exchange 2003 the cluster service account for that cluster can be removed from the organization and/or Administrative Group objects’ permissions using the delegate control wizard. Remember that if that account is used by other Exchange 2000 clusters, then you will have to leave the permissions in place until they have been upgraded to Exchange 2003

Windows 2000 Cluster Service Account: Local Administrator on each Node in the cluster Exchange Full Administrator on org object if other Exchange 2000 clusters

remain in org

Windows 2003 Cluster Service Account Local Administrator on each Node No permissions required on org

Permissions required quick check:

Page 210: Exchange Server Admin Troubleshooting

Module 5: Clustering 9

MsExchange_NodeState

MsExchange_NodeState is a registry value that is set during the installation of the Exchange 2003 binary files on a cluster Node. It can be seen in the following screen shot:

In this screen shot we can see that Nodes 1 and 2 have the value set and Nodes 3 and 4 do not.

In Exchange 2000 we did not check if a node in the cluster had the Exchange binary files installed. This would cause problems when calculating if Active/Active or Active/Passive was allowed in the cluster.

For example, If we have a three-Node cluster but we have only installed Exchange 2000 on two of the Nodes, we should be allowed to have an

Page 211: Exchange Server Admin Troubleshooting

10 Module 5: Clustering

Active/Active or Active/Passive cluster, as only two of the Nodes can actually host an Exchange Virtual Server. Unfortunately exres.dll (The Exchange cluster resource .dll) does not see that one of the Nodes does not have the Exchange 2000 binaries and therefore should not be counted when calculating if Active/Active or Active/Passive is allowed. In this scenario if we created two Exchange Virtual Servers and then tried to bring both of them online on one Node, one of the Exchange Virtual Servers would fail to come online.

In Exchange 2003 we now write this value to the registry on each Node in order to tell us that the Node is Exchange enabled (i.e. it can be added as a possible owner of an Exchange 2003 resource).

It is also used to determine the maximum number of Exchange Virtual Servers that can be created in the cluster.

For example if we have six Nodes in the cluster but only four of them have the MSExchange_NodeState set to 1, then we can create a maximum of three Exchange Virtual Servers.

The maximum is three in this example, as we always enforce at least one passive node in cluster with greater than two Nodes.

The registry value is written to the following location:

HKLM\Cluster\Nodes\<Node_ID>\Parameters\MSExchange_NodeState REG_DWORD=1

The Node_ID value is set when the node is added as a member of the cluster. The first node in a cluster will always have a Node ID of 1.

When the value is set to 1, then the node is Exchange 2003-enabled and can be set as a possible owner of Exchange 2003 resources in the cluster.

If this value is set to 0 or does not exist, then the Node will not be allowed as a possible owner for Exchange 2003 resources.

If you attempt to add a cluster node as a possible owner of an Exchange 2003 resource you will receive the following error:

Another registry entry MSExchange_CurrentBuild is used to track the version of Exchange 2003 that is installed on the Node. For example, if we were to upgrade the binary files on Node 1 to a higher value than Node 2, then the MSExchange_CurrentBuild versions would be different on the two nodes.

Note

Page 212: Exchange Server Admin Troubleshooting

Module 5: Clustering 11

From the Exchange Server Setup Progress log we can see Setup writing these attributes:

[02:25:13] Entering CAtomClusterServer::ScSetExchangeStateOnCluster [02:25:13] Entering CAtomClusterServer::ScSetNodeProperty [02:25:13] Setting DWORD MSExchange_NodeState=1 on node 'NODE1' [02:25:13] Setting DWORD MSExchange_CurrentBuild=452526080 on node 'NODE1' [02:25:13] Leaving CAtomClusterServer::ScSetNodeProperty [02:25:13] Leaving CAtomClusterServer::ScSetExchangeStateOnCluster [02:25:13] Entering CAtomClusterServer::ScEnableNodeAsPossibleOwer [02:25:13] Leaving CAtomClusterServer::ScEnableNodeAsPossibleOwer

This can also be seen from the cluster log:

00000550.00000308::2003/04/20-00:25:13.497 INFO [DM] Setting value of MSExchange_NodeState for key Nodes\1\Parameters to 0x00000001

00000550.00000308::2003/04/20-00:25:13.497 INFO [DM] Setting value of MSExchange_CurrentBuild for key Nodes\1\Parameters to 0x1af90000

Page 213: Exchange Server Admin Troubleshooting

12 Module 5: Clustering

DNS registration/Kerberos

Related articles:

Article 235529 Windows 2000 SP3 added support for Kerberos authentication against clustered virtual servers. Prior to Windows 2000 SP3, all authentication against clustered virtual servers was NTLM or NTLM version 2 (NTLMv2). Prior to Windows 2000 SP3, a clustered virtual server did not have a corresponding Active Directory computer object. When the RequireKerberos property was set to 1 and the Network Name resource was restarted an Active Directory computer object would be created.

Support for another property RequireDNS was also added in SP3. This would mean that the clustered virtual server would have to successfully register its own host record (A-record) in DNS in order to come online.

Both of these properties are private properties of a Network Name resource and can be set to either 0 or 1. RequireKerberos and RequireDNS have defaults of 1 and 0, respectively. Setting either of these values to 1 is not supported for an Exchange 2000 Virtual Server. An Exchange 2000 Network Name resource required that these properties be set to 0.

Therefore all authentication against Exchange 2000 Virtual Servers is NTLM or NTLM version 2.

In Exchange 2003 we now enforce the use of Kerberos authentication. This is done automatically by the setup process for non-clustered servers. For clusters, these private properties are set during the creation of the virtual server. To view the properties in Windows 2000 we need to use the command line cluster.exe tool as follows:

Cluster res “my EVS Network Name” /priv

Windows 2000 SP3

Page 214: Exchange Server Admin Troubleshooting

Module 5: Clustering 13

In Windows 2000 this can only be set by using the command line tool cluster.exe. In the screenshot above, the cluster.exe command has already been used to change the RequireDNS property to a value to “1”.

Page 215: Exchange Server Admin Troubleshooting

14 Module 5: Clustering

In Windows 2003 Server these properties are changeable from the GUI of cluster administrator.

As mentioned earlier, there is no need to set these manually as the setup process will automatically set requireKerberos to 1 and requireDNS to 0. If, after starting these resources with their defaults, you change requireKerberos to 0 (i.e. uncheck it from the UI,) then the Network Name resource will fail to come online. Changing the DNS requirement does not affect how resources come online unless you are using static DNS, where all records need to be entered by-hand.

From the Exchange Server Setup Progress.log we can see Exchange Virtual Server-creation setting these values:

Page 216: Exchange Server Admin Troubleshooting

Module 5: Clustering 15

[07:23:58] Entering ScEnableKerberosAndDNSOnEVS [07:23:58] Entering ScBindToEVS [07:23:58] Binding to cluster'node02.exchange.org' [07:23:58] Binding to EVS 'EVS01' [07:23:58] Leaving ScBindToEVS [07:23:58] Searching for network name resource 'EVS01' [07:23:58] Resource 'EVS01 Network Name' owns network name 'EVS01' [07:23:58] Entering ScEnableKerberosAndDNSOnResource [07:23:58] Entering ScGetKerberosAndDNSFromNetworkNameResource [07:23:58] RequireKerberos=0 [07:23:58] RequireDNS=0 [07:23:58] Leaving ScGetKerberosAndDNSFromNetworkNameResource [07:23:59] Setting properties to enable Kerberos on resource 'EVS01 Network Name' [07:23:59] Entering ScSetKerberosAndDNSPropertiesOnResource [07:23:59] Setting property 'RequireKerberos'='1' [07:23:59] Leaving ScSetKerberosAndDNSPropertiesOnResource [07:23:59] Bringing resource 'EVS01 Network Name' online [07:24:39] Network name resource 'EVS01 Network Name' successfully enabled for Kerberos [07:24:39] Entering BringDependentResourcesOnline [07:24:39] Bringing online resources dependent on network name resource [07:24:39] Leaving BringDependentResourcesOnline [07:24:39] Leaving ScEnableKerberosAndDNSOnResource [07:24:39] Leaving ScEnableKerberosAndDNSOnEVS

The requirement for DNS registration to succeed was removed from Exchange 2003 in build 6917. This was due to the fact that the Network Name resource would fail to come online if the DNS service in use did not support dynamic updates.

If a dynamic update DNS service is in use, then the RequireDNS can be set to 1 for an Exchange 2003 Network Name resource.

If a non dynamic update DNS service is in place then it is essential to make sure that the correct A-records exist for the Exchange Virtual Server Network Names resource. If this is missing you may run into transport/routing issues.

The requireKerberos setting will now create an Active Directory object for the Exchange Virtual Server. In order for this to succeed, the Cluster Service account will need to have the “Add workstations to the Domain” right. It should be a domain account and will therefore have this right by default.

A detailed description of the Network Name resources in Windows 2003 can be obtained in article 302389.

Page 217: Exchange Server Admin Troubleshooting

16 Module 5: Clustering

AntiAffinityClassNames

AntiAffinityClassNames is a new feature of Windows 2003 clustering. It gives us the ability to assign a node as a hot spare for a particular cluster group in a cluster of three or more Nodes.

AntiAffinityClassNames is a public property of cluster groups in Windows 2003 that contains an arbitrary string of characters.

AntiAffinityClassNames is not available in Windows 2000 clusters.

If there is a failure of a resource in a Windows 2003 cluster group (and the resource is configured to affect the group), the failover manager will check the value of the AntiAffinityClassNames attribute. It will then look for a Node in the cluster that does not host a cluster group with that AntiAffinityClassNames value. If there is such a node in the cluster, then it is considered to be the best possible destination for the cluster group.

Note

Page 218: Exchange Server Admin Troubleshooting

Module 5: Clustering 17

In the image above we can see a four-Node cluster. There are three Exchange Virtual Servers and the AntiAffinityClassNames property is set to “Microsoft Exchange Virtual Server”.

If EVS01 was to fail, then failover manager would try and find a node in the cluster that did not own a cluster group with this property set to “Microsoft Exchange Virtual Server”.

In this case EVS01 would fail over to node CLU-NODE04.

This, of course, assumes that node CLU-NODE04 is set as a possible owner of the resources that make up EVS01 (which it should be by default).

Note

Page 219: Exchange Server Admin Troubleshooting

18 Module 5: Clustering

To check the value of AntiAffinityClassNames use the cluster.exe from the command prompt.

Cluster group “my Group” /prop

The value is blank by default.

To set the property on a cluster group use the following syntax:

Cluster group “my Group” /prop AntiAffinityClassNames=”Some_Text_String”

The value of AntiAffinityClassNames is not visible from the GUI. It can only be seen using the cluster.exe command line tool.

Page 220: Exchange Server Admin Troubleshooting

Module 5: Clustering 19

When one creates an Exchange 2003 virtual server on a Windows 2003 cluster this attribute will be automatically set to “Microsoft Exchange Virtual Server”. If you are seeing it set to some other string then it has probably been changed manually and should be changed back to the default setting.

Page 221: Exchange Server Admin Troubleshooting

20 Module 5: Clustering

Using Cluster Administrator, we can manually move a group to another Node in the cluster. When we have a cluster with more than two Nodes, we will have a new choice called Best Possible.

In the above image we can see that there are three Nodes in the cluster. By selecting Best Possible, then the cluster group containing the Exchange Virtual Server will be moved to a Node that does not currently host a group with the AntiAffinityClassNames set to “Microsoft Exchange Virtual Server”.

Page 222: Exchange Server Admin Troubleshooting

Module 5: Clustering 21

It is important to note that the AntiAffinityClassNames property is set on the cluster group. If we were to create a new cluster group and then move all of the Exchange 2003 cluster resources into this group then the attributes that were set on the original cluster group do not follow the resources.

The attributes that are affected in this scenario are AntiAffinityClassNames and MSExchange_VirtualServerNames.

It is unsupported to move the Exchange 2003 cluster resources between cluster groups. If you are not seeing these properties set on the Exchange 2003 cluster group then the resources may have been moved manually.

In this scenario it is necessary to recreate the attributes using the command line cluster.exe utility

Related articles:

Windows 2000 DataCenter Server:

279802 – The default behaviour when you do a move group operation on a cluster

197047 – Failover/Failback policies on Microsoft Cluster Server Windows 2003 Enterprise and DataCenter:

301114 – Proper usage of the Preferred Owner List on a cluster Server (not finished)

299631 – Failover behaviour on clusters of three or more nodes 296799 – How to configure Windows Clustering Groups for hot spare

support

Page 223: Exchange Server Admin Troubleshooting

22 Module 5: Clustering

Mount Point Drives

Mount Point Drives have been available since Windows 2000. They were, however, not supported for use in Windows 2000 clusters. This essentially limited the drive letter assignments in a Windows 2000 cluster to a maximum of 26. This number included local drive assignments on each cluster node.

For example, a cluster node may have local physical disks A: (Floppy Drive), C: (Local Disk) and D: (Local Disk). This would effectively leave 23 drives that could be assigned to the cluster. As each node in the cluster may be a possible owner of each of these 23 disk resources, this would limit the entire cluster to 23 disk resources.

As the supported number of Nodes in clusters increases, this limitation begins to seriously hamper the ability to build enterprise class solutions.

In Windows 2003 Server, support for mount point drives has been added in Windows clusters.

Support for Mount Point Drives does not exist in Windows 2000.

A mount point drive is an entire volume that is mounted inside another folder that is hosted by another volume.

Page 224: Exchange Server Admin Troubleshooting

Module 5: Clustering 23

The steps to create a mount point drive available for cluster use are as follows:

1. A new unformatted disk will be available in Disk Manager. Make sure that it is a Basic Volume. Right-click it and choose new partition.

2. Choose Primary partition and click Next.

Page 225: Exchange Server Admin Troubleshooting

24 Module 5: Clustering

3. Set the size for the partition and click Next.

4. Instead of assigning a Drive Letter to the disk, we are going to mount it inside a folder on another volume. The volume must be a disk that is visible to the cluster (i.e. not a local disk). The mount point volume will also be hosted in the same cluster group as the hosting volume.

Page 226: Exchange Server Admin Troubleshooting

Module 5: Clustering 25

5. In this scenario we are going to use Disk R: which is a disk in our cluster.

6. I have created a new folder on R:\ called Mount which will host the new volume.

Page 227: Exchange Server Admin Troubleshooting

26 Module 5: Clustering

7. Give the volume a label and then format it using NTFS. It must be formatted with NTFS.

8. Click Next to complete the New Partition Wizard.

Page 228: Exchange Server Admin Troubleshooting

Module 5: Clustering 27

9. We can now see our Mount Point Drive in Disk Manager.

10. To avoid confusion, Mount Point drives appear as physical disks rather than folders when viewed in Explorer. If you were to remove the Mount Point Drive then the folder R:\Mount would remain on R:\ as a normal folder.

Page 229: Exchange Server Admin Troubleshooting

28 Module 5: Clustering

11. Now we have to create the cluster resource for the Mount Point Drive.

The Mount Point Drive resource must be in the same cluster group as our hosting disk R:\.

Using Cluster Administrator, locate the correct Cluster Group and then create a new resource. Give the resource a name and choose Physical Disk for the resource type. Click Next.

Note

Page 230: Exchange Server Admin Troubleshooting

Module 5: Clustering 29

12. Set the possible owners for the resource. In this scenario we only have one Node.

Page 231: Exchange Server Admin Troubleshooting

30 Module 5: Clustering

13. Set a dependency on R:\. NOTE: It is essential that we do this. Cluster Administrator will not prompt you to do this. By setting a dependency on R:\ we ensure that the resource for the Mount Point Drive is taken offline when the R:\ resource is taken offline.

14. Now choose the correct Mount Point Volume that we created earlier. If you are unsure, then use Disk Manager to locate the correct disk number.

Page 232: Exchange Server Admin Troubleshooting

Module 5: Clustering 31

15. After clicking “Finish,” the Mount Point Drive resource will now appear in Cluster Administrator.

16. The Mount Point Drive properties can then be seen in the registry:

Page 233: Exchange Server Admin Troubleshooting

32 Module 5: Clustering

A few rules of thumb regarding Mount Point Drives:

1. The partition must be mounted inside a disk/volume that is available to the cluster, i.e. Shared Storage.

2. The cluster resource for the Mount Point Drive must exist in the same cluster group as the disk that is hosting it.

3. The cluster resource for the Mount Point Drive must have a dependency on the resource for the disk that is hosting it.

4. The System Attendant cluster resource must have a dependency on all disk resources including Mount Point Drive resources (assuming they are being used for Exchange purposes).

Page 234: Exchange Server Admin Troubleshooting

Module 5: Clustering 33

Creating an Exchange Virtual Server

Once the Exchange 2003 binaries have been installed on the cluster Node we can now create an Exchange Virtual Server. It is a good idea to make sure that all the required Nodes in the cluster have the Exchange 2003 binary file installed before proceeding with the Exchange Virtual Server creation.

The Exchange Virtual Server creation process is much the same as Exchange 2000.

First we need to create a cluster group to house the Exchange Virtual Server. Within this group we must have at least one physical disk resource, at least one IP address resource, and a network name resource.

The network name resource must have a dependency on all the IP address resources in the cluster group (assuming that all the IP address resources will be used for Exchange 2003 purposes)

When we have these resources in the group, we can begin the creation of the System Attendant resource. Once the System Attendant Resource has been created all the other Exchange 2003 cluster resources will be created automatically.

Right-click on the Exchange 2003 cluster group and choose New Resource and then choose Microsoft Exchange System Attendant. Give it an identifiable name and click Next

Page 235: Exchange Server Admin Troubleshooting

34 Module 5: Clustering

Page 236: Exchange Server Admin Troubleshooting

Module 5: Clustering 35

Add the Nodes that will be possible owners of the System Attendant Resource.

These Nodes will also be added as possible owners of all the other Exchange resources that are automatically created. A Node must be specified as a possible owner of a resource in order for us to failover to that Node.

If you attempt to add a Node that does not have the MSExchange_NodeState value set to 1 then you will receive the following error:

The specified Node will not be able to be a possible owner of the resource until the Exchange 2003 binary files have been installed on the Node (which in turn will add the MSExchange_NodeState value in the registry).

The MSExchange_NodeState value is used to determine which Nodes can be possible owners of an Exchange Virtual Server.

Please refer to the MSExchange_NodeState section for a more detailed description of this.

Note

Page 237: Exchange Server Admin Troubleshooting

36 Module 5: Clustering

Now we have to set the dependencies on the System Attendant Resource. You must set a dependency on the Network Name resource and all disk resources that you plan to use for Exchange 2003 related use. This includes all Mount Point Disks which will contain Exchange 2003 data.

If more than one Administrative Group exists, choose the Administrative Group that we will install the Exchange Virtual Server into.

Only mixed Administrative Groups or pure Exchange 2000/2003 Administrative Groups will be displayed. Pure Exchange 5.5 sites will not be displayed, as the first Exchange 2003 Server in a Exchange 5.5 site cannot be a cluster.

If this is not the first Exchange Virtual Server in the cluster, then this choice will be automatically set to the Admin Group of the other Exchange Virtual Server(s). All Exchange Virtual Servers in a cluster must be part of the same Admin Group.

Note

Page 238: Exchange Server Admin Troubleshooting

Module 5: Clustering 37

Page 239: Exchange Server Admin Troubleshooting

38 Module 5: Clustering

Within the chosen Administrative Group we now have to choose a Routing Group that the Exchange Virtual Server will be a member of, if more than one Routing Group exists. This can be changed afterwards if an incorrect Routing Group is chosen.

Now we have to choose a path for the Exchange Virtual Server. We must specify a path on a disk that we specified in the dependencies section earlier.

We can specify a location on a Mount Point disk.

Note

Page 240: Exchange Server Admin Troubleshooting

Module 5: Clustering 39

If the specified location is not empty, then we will receive the following error:

This is to safeguard against installing the Exchange Virtual Server into a directory that may cause possible conflicts.

From the “Exchange Server Setup Progress.log” file we can see the error:

Remove files from the specified directory, or specify a new directory that is empty.

Page 241: Exchange Server Admin Troubleshooting

40 Module 5: Clustering

We will now be presented with an overview of the choices made in the creation wizard.

If you need to change any of the choices, then you can use the Back button to navigate back and make the necessary changes.

Click Finish to start the creation process. This may take a little while as it actually creates all the other Exchange 20003 cluster resources.

When the process is complete you will receive a prompt as follows:

Page 242: Exchange Server Admin Troubleshooting

Module 5: Clustering 41

In the Cluster Administrator tool, we should now see all the Exchange 2003 cluster resources. They will be offline.

Exchange 2003 no longer creates an IMAP4 and POP3 cluster resource. This is part of a security push and also ties in with a standalone installation which also disables the IMAP4 and POP3 services by default.

The entire Exchange Virtual Server process is logged to the Setup Progress Log. If any difficulties arise as part of the creation process, the best place to start looking is in this log.

Below is a copy of the entire process:

Exchange Virtual Server Creation outlined in setup progress log

Page 243: Exchange Server Admin Troubleshooting

42 Module 5: Clustering

[07:10:39] ************** Beginning Setup run ************** [07:10:39] Starting Exchange 6944 setup on Windows 5.2.3790. at 07:10:39 07/03/2003 [07:10:39] Entering CFileManager::ScInit [07:10:39] Entering CFileManager::ScAutoDetectDirectoryLocations [07:10:39] Leaving CFileManager::ScAutoDetectDirectoryLocations [07:10:39] Leaving CFileManager::ScInit [07:10:39] Entering CRegistryManager::ScInit [07:10:39] Leaving CRegistryManager::ScInit [07:10:39] Entering CDirectoryManager::ScInit [07:10:39] Entering ScIsComputerMemberOfDomain [07:10:39] NetGetJoinInformation: Domain/workgroup = "EXCHANGE" [07:10:39] NetGetJoinInformation: NETSETUP_JOIN_STATUS = 3 [07:10:39] The computer is a member of a domain [07:10:39] Leaving ScIsComputerMemberOfDomain [07:10:39] Entering CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Getting information about the local domain [07:10:39] m_strLocalServer = "NODE02" [07:10:39] m_strLocalSite = "Default-First-Site-Name" [07:10:39] DsRoleGetPrimaryDomainInformation returned: [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::MachineRole = 3 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::Flags = 1000000 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "EXCHANGE" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "exchange.org" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "exchange.org" [07:10:39] Entering CDirectoryManager::ScCheckCommandLineForDC [07:10:39] Leaving CDirectoryManager::ScCheckCommandLineForDC [07:10:39] No user-specified DC; setup has chosen m_strDC = "DC" [07:10:39] schema master server name: DC [07:10:39] schema master domain : /dc=org/dc=exchange [07:10:39] m_strSchemaMasterDC = "DC" [07:10:39] m_strSchemaMasterDCDomainDN = "/dc=org/dc=exchange" [07:10:39] strConfigNC = "CN=Configuration,DC=exchange,DC=org" [07:10:39] m_strRootDomain = "exchange.org" [07:10:39] m_strOwnershipControlDC = "DC" [07:10:39] m_strPermissionControlDC = "DC" [07:10:39] Leaving CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Entering CDirectoryManager::ScInitializeSessions [07:10:39] Entering CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Entering ScGetSchemaVersion [07:10:39] About to create the dob for object /dc=org/dc=exchange/cn=Configuration/cn=Schema/cn=ms-Exch-Schema-Version-Pt [07:10:39] The schema version identified for the Server is 6870 [07:10:39] Leaving ScGetSchemaVersion [07:10:39] Leaving CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Entering ScGetMicrosoftExchangeCTHeuristics [07:10:39] Leaving ScGetMicrosoftExchangeCTHeuristics [07:10:39] Entering CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScDeterminePermissionLevel [07:10:39] Checking permissions in the Config NC: /dc=org/dc=exchange/cn=Configuration/cn=Services [07:10:39] We have permission ConfigNC_Read [07:10:39] We have permission ConfigNC_Write [07:10:39] We have permission ConfigNC_SetPerms

Page 244: Exchange Server Admin Troubleshooting

Module 5: Clustering 43

[07:10:39] Checking permissions on the Schema container: /dc=org/dc=exchange/cn=Configuration/cn=Schema [07:10:39] We have permission ConfigNC_UpdateSchema [07:10:39] Checking permissions in the Domain NC: /dc=org/dc=exchange [07:10:39] We have permission DomainNC_Read [07:10:39] We have permission DomainNC_Write [07:10:39] Checking to see if an Exchange org exists [07:10:39] Found the organization "EXCHANGE" [07:10:39] Checking read permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups [07:10:39] We have permission ExchOrg_Read [07:10:39] Checking write/security permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE [07:10:39] We have permission (ExchOrg_Write | ExchAG_Write) [07:10:39] We have permission (ExchOrg_SetPerms | ExchAG_SetPerms) [07:10:39] Looking for an existing server object [07:10:39] Didn't find an existing server object [07:10:39] Enumerating all admin groups in the org [07:10:39] Found 1 admin groups [07:10:39] Checking permissions on the admin group: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group [07:10:39] We have permission ExchAG_Read [07:10:39] We have permission ExchAG_Write [07:10:39] We have permission ExchAG_SetPerms [07:10:39] Final set of permissions: 0XF0C0E0E0 [07:10:39] Leaving CDirectoryManager::ScDeterminePermissionLevel [07:10:39] We have sufficient admin rights, the schema is up to date and org-level objects are present on the local DC; m_strDCToUse = "DC" [07:10:39] Sanity check: [07:10:39] m_strDCToUse = "DC" [07:10:39] m_psesToUse->m_strServerName = "DC" [07:10:39] Leaving CDirectoryManager::ScInitializeSessions [07:10:39] Leaving CDirectoryManager::ScInit [07:10:39] === IGNORING PREVIOUS ERRORS === CExchangeSetupCtx::ScDetermineExchangeObjectStateFromDS (f:\titanium\admin\src\udog\excommon\exsetctx.cxx:411) The operation has completed successfully. [07:10:39] Entering ScSetupExchangeVirtualServer [07:10:39] Performing create for EVS 'EVS01' [07:10:39] Entering CDirectoryManager::ScReInitWithDC [07:10:39] Reinitalizing the DS Manager, using the DC DC [07:10:39] Entering CDirectoryManager::ScInit [07:10:39] Entering ScIsComputerMemberOfDomain [07:10:39] NetGetJoinInformation: Domain/workgroup = "EXCHANGE" [07:10:39] NetGetJoinInformation: NETSETUP_JOIN_STATUS = 3 [07:10:39] The computer is a member of a domain [07:10:39] Leaving ScIsComputerMemberOfDomain [07:10:39] Entering CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Getting information about the local domain [07:10:39] m_strLocalServer = "NODE02" [07:10:39] m_strLocalSite = "Default-First-Site-Name" [07:10:39] DsRoleGetPrimaryDomainInformation returned: [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::MachineRole = 3 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::Flags = 1000000

Page 245: Exchange Server Admin Troubleshooting

44 Module 5: Clustering

[07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "EXCHANGE" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "exchange.org" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "exchange.org" [07:10:39] User has specified a DC; m_strDC = "DC" [07:10:39] schema master server name: DC [07:10:39] schema master domain : /dc=org/dc=exchange [07:10:39] m_strSchemaMasterDC = "DC" [07:10:39] m_strSchemaMasterDCDomainDN = "/dc=org/dc=exchange" [07:10:39] strConfigNC = "CN=Configuration,DC=exchange,DC=org" [07:10:39] m_strRootDomain = "exchange.org" [07:10:39] m_strOwnershipControlDC = "DC" [07:10:39] m_strPermissionControlDC = "DC" [07:10:39] Leaving CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Entering CDirectoryManager::ScInitializeSessions [07:10:39] Entering CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Leaving CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Entering ScGetMicrosoftExchangeCTHeuristics [07:10:39] Leaving ScGetMicrosoftExchangeCTHeuristics [07:10:39] Entering CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScDeterminePermissionLevel [07:10:39] Checking permissions in the Config NC: /dc=org/dc=exchange/cn=Configuration/cn=Services [07:10:39] We have permission ConfigNC_Read [07:10:39] We have permission ConfigNC_Write [07:10:39] We have permission ConfigNC_SetPerms [07:10:39] Checking permissions on the Schema container: /dc=org/dc=exchange/cn=Configuration/cn=Schema [07:10:39] We have permission ConfigNC_UpdateSchema [07:10:39] Checking permissions in the Domain NC: /dc=org/dc=exchange [07:10:39] We have permission DomainNC_Read [07:10:39] We have permission DomainNC_Write [07:10:39] Checking to see if an Exchange org exists [07:10:39] Found the organization "EXCHANGE" [07:10:39] Checking read permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups [07:10:39] We have permission ExchOrg_Read [07:10:39] Checking write/security permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE [07:10:39] We have permission (ExchOrg_Write | ExchAG_Write) [07:10:39] We have permission (ExchOrg_SetPerms | ExchAG_SetPerms) [07:10:39] Looking for an existing server object [07:10:39] Didn't find an existing server object [07:10:39] Enumerating all admin groups in the org [07:10:39] Found 1 admin groups [07:10:39] Checking permissions on the admin group: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group [07:10:39] We have permission ExchAG_Read [07:10:39] We have permission ExchAG_Write [07:10:40] We have permission ExchAG_SetPerms [07:10:40] Final set of permissions: 0XF0C0E0E0 [07:10:40] Leaving CDirectoryManager::ScDeterminePermissionLevel

Page 246: Exchange Server Admin Troubleshooting

Module 5: Clustering 45

[07:10:40] We have sufficient admin rights, the schema is up to date and org-level objects are present on the local DC; m_strDCToUse = "DC" [07:10:40] Sanity check: [07:10:40] m_strDCToUse = "DC" [07:10:40] m_psesToUse->m_strServerName = "DC" [07:10:40] Leaving CDirectoryManager::ScInitializeSessions [07:10:40] Leaving CDirectoryManager::ScInit [07:10:40] Leaving CDirectoryManager::ScReInitWithDC [07:10:40] Entering ScGetDomainInfoFromServer [07:10:40] Leaving ScGetDomainInfoFromServer [07:10:40] Entering ScGetExchangeDsConfigFromDs [07:10:40] Looking for Exchange server object 'EVS01' [07:10:40] Couldn't find the Exchange Server Object for this server. [07:10:40] Leaving ScGetExchangeDsConfigFromDs [07:10:40] Entering ScGetOrganization [07:10:40] Organization name='EXCHANGE' [07:10:40] Leaving ScGetOrganization [07:10:40] Entering ScGetExchangeServicesInstallPathFromRegistry [07:10:40] Exchange install path on server 'node02.exchange.org' is 'C:\Program Files\Exchsrvr' [07:10:40] Leaving ScGetExchangeServicesInstallPathFromRegistry [07:10:40] Data path is 's:\mount\exchsrvr' [07:10:40] Entering ScEnumerateExchangeServersInCluster2 [07:10:40] Entering ScEnumerateExchangeServersInCluster [07:10:40] Getting cluster resource info... [07:10:40] Adding cluster network name resource 'CLUSTER' to list [07:10:40] Adding cluster network name resource 'EVS01' to list [07:10:40] Executing LDAP query "(&(objectCategory=msExchExchangeServer)(|(networkAddress=netbios:CLUSTER)(networkAddress=netbios:EVS01)))" [07:10:40] Leaving ScEnumerateExchangeServersInCluster [07:10:40] Leaving ScEnumerateExchangeServersInCluster2 [07:10:40] Responsible MTA server name: EVS01 [07:10:40] Context data "ClusterToBind" (CStr), previously set to "localhost", being reset to "node02.exchange.org" ClusterToBind (f:\titanium\admin\src\udog\inc\exsetctx.hxx:146) The operation has completed successfully. [07:10:40] Context data "LocalServer" (CStr), previously set to "NODE02", being reset to "EVS01" LocalServer (f:\titanium\admin\src\udog\inc\exsetctx.hxx:139) The operation has completed successfully. [07:10:40] Context data "ResponsibleMTAServer" (CStr), previously set to "NODE02", being reset to "EVS01" ResponsibleMTAServer (f:\titanium\admin\src\udog\inc\exsetctx.hxx:138) The operation has completed successfully. [07:10:40] Entering CFileManager::ScSetInstallDestDir(sz) [07:10:40] Leaving CFileManager::ScSetInstallDestDir(sz) [07:10:40] Entering ScCheckLocalAndRemoteExchangeBuilds [07:10:40] Entering ScGetExchangeBuild [07:10:40] Exchange build number on server 'node02.exchange.org' is '6944.0' [07:10:40] Leaving ScGetExchangeBuild [07:10:40] Entering ScGetExchangeBuild [07:10:40] Exchange build number on server 'localhost' is '6944.0' [07:10:40] Leaving ScGetExchangeBuild [07:10:40] Leaving ScCheckLocalAndRemoteExchangeBuilds [07:10:40] Entering ScCheckStateOfDependentResources [07:10:40] Entering ScBindToEVS

Page 247: Exchange Server Admin Troubleshooting

46 Module 5: Clustering

[07:10:40] Binding to cluster'node02.exchange.org' [07:10:40] Binding to EVS 'EVS01' [07:10:40] Leaving ScBindToEVS [07:10:40] Leaving ScCheckStateOfDependentResources [07:10:40] Entering ScDetermineIfLocalDomainServerGroupHasAlreadyBeenACLedOnExchangeCT [07:10:40] Checking to see whether "EXCHANGE\Exchange Domain Servers" has READ permissions on the Microsoft Exchange container [07:10:40] Entering ScTestAceOnObject [07:10:40] Attempting to get DOB for DN "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange" [07:10:40] Attempting to read security descriptor from DOB [07:10:40] Attempting to initialize CAce object [07:10:40] Testing to see if given ACE is present [07:10:40] Test succeeded; fACLPresent = TRUE, fExtraRights = FALSE [07:10:40] The ACE tested for is present in the ACL of this object [07:10:40] Leaving ScTestAceOnObject [07:10:40] The Domain Servers group does have READ permissions on the Exchange container [07:10:40] Checking to see whether ANONYMOUS LOGON has READ permissions for MDB objects on the organization [07:10:40] Entering ScTestAceOnObject [07:10:39] ************** Beginning Setup run ************** [07:10:39] Starting Exchange 6944 setup on Windows 5.2.3790. at 07:10:39 07/03/2003 [07:10:39] Entering CFileManager::ScInit [07:10:39] Entering CFileManager::ScAutoDetectDirectoryLocations [07:10:39] Leaving CFileManager::ScAutoDetectDirectoryLocations [07:10:39] Leaving CFileManager::ScInit [07:10:39] Entering CRegistryManager::ScInit [07:10:39] Leaving CRegistryManager::ScInit [07:10:39] Entering CDirectoryManager::ScInit [07:10:39] Entering ScIsComputerMemberOfDomain [07:10:39] NetGetJoinInformation: Domain/workgroup = "EXCHANGE" [07:10:39] NetGetJoinInformation: NETSETUP_JOIN_STATUS = 3 [07:10:39] The computer is a member of a domain [07:10:39] Leaving ScIsComputerMemberOfDomain [07:10:39] Entering CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Getting information about the local domain [07:10:39] m_strLocalServer = "NODE02" [07:10:39] m_strLocalSite = "Default-First-Site-Name" [07:10:39] DsRoleGetPrimaryDomainInformation returned: [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::MachineRole = 3 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::Flags = 1000000 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "EXCHANGE" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "exchange.org" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "exchange.org" [07:10:39] Entering CDirectoryManager::ScCheckCommandLineForDC [07:10:39] Leaving CDirectoryManager::ScCheckCommandLineForDC [07:10:39] No user-specified DC; setup has chosen m_strDC = "DC" [07:10:39] schema master server name: DC [07:10:39] schema master domain : /dc=org/dc=exchange [07:10:39] m_strSchemaMasterDC = "DC" [07:10:39] m_strSchemaMasterDCDomainDN = "/dc=org/dc=exchange" [07:10:39] strConfigNC = "CN=Configuration,DC=exchange,DC=org" [07:10:39] m_strRootDomain = "exchange.org" [07:10:39] m_strOwnershipControlDC = "DC"

Page 248: Exchange Server Admin Troubleshooting

Module 5: Clustering 47

[07:10:39] m_strPermissionControlDC = "DC" [07:10:39] Leaving CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Entering CDirectoryManager::ScInitializeSessions [07:10:39] Entering CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Entering ScGetSchemaVersion [07:10:39] About to create the dob for object /dc=org/dc=exchange/cn=Configuration/cn=Schema/cn=ms-Exch-Schema-Version-Pt [07:10:39] The schema version identified for the Server is 6870 [07:10:39] Leaving ScGetSchemaVersion [07:10:39] Leaving CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Entering ScGetMicrosoftExchangeCTHeuristics [07:10:39] Leaving ScGetMicrosoftExchangeCTHeuristics [07:10:39] Entering CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScDeterminePermissionLevel [07:10:39] Checking permissions in the Config NC: /dc=org/dc=exchange/cn=Configuration/cn=Services [07:10:39] We have permission ConfigNC_Read [07:10:39] We have permission ConfigNC_Write [07:10:39] We have permission ConfigNC_SetPerms [07:10:39] Checking permissions on the Schema container: /dc=org/dc=exchange/cn=Configuration/cn=Schema [07:10:39] We have permission ConfigNC_UpdateSchema [07:10:39] Checking permissions in the Domain NC: /dc=org/dc=exchange [07:10:39] We have permission DomainNC_Read [07:10:39] We have permission DomainNC_Write [07:10:39] Checking to see if an Exchange org exists [07:10:39] Found the organization "EXCHANGE" [07:10:39] Checking read permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups [07:10:39] We have permission ExchOrg_Read [07:10:39] Checking write/security permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE [07:10:39] We have permission (ExchOrg_Write | ExchAG_Write) [07:10:39] We have permission (ExchOrg_SetPerms | ExchAG_SetPerms) [07:10:39] Looking for an existing server object [07:10:39] Didn't find an existing server object [07:10:39] Enumerating all admin groups in the org [07:10:39] Found 1 admin groups [07:10:39] Checking permissions on the admin group: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group [07:10:39] We have permission ExchAG_Read [07:10:39] We have permission ExchAG_Write [07:10:39] We have permission ExchAG_SetPerms [07:10:39] Final set of permissions: 0XF0C0E0E0 [07:10:39] Leaving CDirectoryManager::ScDeterminePermissionLevel [07:10:39] We have sufficient admin rights, the schema is up to date and org-level objects are present on the local DC; m_strDCToUse = "DC" [07:10:39] Sanity check: [07:10:39] m_strDCToUse = "DC" [07:10:39] m_psesToUse->m_strServerName = "DC" [07:10:39] Leaving CDirectoryManager::ScInitializeSessions

Page 249: Exchange Server Admin Troubleshooting

48 Module 5: Clustering

[07:10:39] Leaving CDirectoryManager::ScInit [07:10:39] === IGNORING PREVIOUS ERRORS === CExchangeSetupCtx::ScDetermineExchangeObjectStateFromDS (f:\titanium\admin\src\udog\excommon\exsetctx.cxx:411) The operation has completed successfully. [07:10:39] Entering ScSetupExchangeVirtualServer [07:10:39] Performing create for EVS 'EVS01' [07:10:39] Entering CDirectoryManager::ScReInitWithDC [07:10:39] Reinitalizing the DS Manager, using the DC DC [07:10:39] Entering CDirectoryManager::ScInit [07:10:39] Entering ScIsComputerMemberOfDomain [07:10:39] NetGetJoinInformation: Domain/workgroup = "EXCHANGE" [07:10:39] NetGetJoinInformation: NETSETUP_JOIN_STATUS = 3 [07:10:39] The computer is a member of a domain [07:10:39] Leaving ScIsComputerMemberOfDomain [07:10:39] Entering CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Getting information about the local domain [07:10:39] m_strLocalServer = "NODE02" [07:10:39] m_strLocalSite = "Default-First-Site-Name" [07:10:39] DsRoleGetPrimaryDomainInformation returned: [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::MachineRole = 3 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::Flags = 1000000 [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "EXCHANGE" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "exchange.org" [07:10:39] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "exchange.org" [07:10:39] User has specified a DC; m_strDC = "DC" [07:10:39] schema master server name: DC [07:10:39] schema master domain : /dc=org/dc=exchange [07:10:39] m_strSchemaMasterDC = "DC" [07:10:39] m_strSchemaMasterDCDomainDN = "/dc=org/dc=exchange" [07:10:39] strConfigNC = "CN=Configuration,DC=exchange,DC=org" [07:10:39] m_strRootDomain = "exchange.org" [07:10:39] m_strOwnershipControlDC = "DC" [07:10:39] m_strPermissionControlDC = "DC" [07:10:39] Leaving CDirectoryManager::ScGetLocalDomainInformation [07:10:39] Entering CDirectoryManager::ScInitializeSessions [07:10:39] Entering CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Leaving CDirectoryManager::ScSchemaIsUpToDate [07:10:39] Entering ScGetMicrosoftExchangeCTHeuristics [07:10:39] Leaving ScGetMicrosoftExchangeCTHeuristics [07:10:39] Entering CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetCountOfOrgsInDomain [07:10:39] Leaving CDirectoryManager::ScGetOrgLevelObjectStatus [07:10:39] Entering CDirectoryManager::ScDeterminePermissionLevel [07:10:39] Checking permissions in the Config NC: /dc=org/dc=exchange/cn=Configuration/cn=Services [07:10:39] We have permission ConfigNC_Read [07:10:39] We have permission ConfigNC_Write [07:10:39] We have permission ConfigNC_SetPerms [07:10:39] Checking permissions on the Schema container: /dc=org/dc=exchange/cn=Configuration/cn=Schema [07:10:39] We have permission ConfigNC_UpdateSchema [07:10:39] Checking permissions in the Domain NC: /dc=org/dc=exchange [07:10:39] We have permission DomainNC_Read [07:10:39] We have permission DomainNC_Write

Page 250: Exchange Server Admin Troubleshooting

Module 5: Clustering 49

[07:10:39] Checking to see if an Exchange org exists [07:10:39] Found the organization "EXCHANGE" [07:10:39] Checking read permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups [07:10:39] We have permission ExchOrg_Read [07:10:39] Checking write/security permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE [07:10:39] We have permission (ExchOrg_Write | ExchAG_Write) [07:10:39] We have permission (ExchOrg_SetPerms | ExchAG_SetPerms) [07:10:39] Looking for an existing server object [07:10:39] Didn't find an existing server object [07:10:39] Enumerating all admin groups in the org [07:10:39] Found 1 admin groups [07:10:39] Checking permissions on the admin group: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group [07:10:39] We have permission ExchAG_Read [07:10:39] We have permission ExchAG_Write [07:10:40] We have permission ExchAG_SetPerms [07:10:40] Final set of permissions: 0XF0C0E0E0 [07:10:40] Leaving CDirectoryManager::ScDeterminePermissionLevel [07:10:40] We have sufficient admin rights, the schema is up to date and org-level objects are present on the local DC; m_strDCToUse = "DC" [07:10:40] Sanity check: [07:10:40] m_strDCToUse = "DC" [07:10:40] m_psesToUse->m_strServerName = "DC" [07:10:40] Leaving CDirectoryManager::ScInitializeSessions [07:10:40] Leaving CDirectoryManager::ScInit [07:10:40] Leaving CDirectoryManager::ScReInitWithDC [07:10:40] Entering ScGetDomainInfoFromServer [07:10:40] Leaving ScGetDomainInfoFromServer [07:10:40] Entering ScGetExchangeDsConfigFromDs [07:10:40] Looking for Exchange server object 'EVS01' [07:10:40] Couldn't find the Exchange Server Object for this server. [07:10:40] Leaving ScGetExchangeDsConfigFromDs [07:10:40] Entering ScGetOrganization [07:10:40] Organization name='EXCHANGE' [07:10:40] Leaving ScGetOrganization [07:10:40] Entering ScGetExchangeServicesInstallPathFromRegistry [07:10:40] Exchange install path on server 'node02.exchange.org' is 'C:\Program Files\Exchsrvr' [07:10:40] Leaving ScGetExchangeServicesInstallPathFromRegistry [07:10:40] Data path is 's:\mount\exchsrvr' [07:10:40] Entering ScEnumerateExchangeServersInCluster2 [07:10:40] Entering ScEnumerateExchangeServersInCluster [07:10:40] Getting cluster resource info... [07:10:40] Adding cluster network name resource 'CLUSTER' to list [07:10:40] Adding cluster network name resource 'EVS01' to list [07:10:40] Executing LDAP query "(&(objectCategory=msExchExchangeServer)(|(networkAddress=netbios:CLUSTER)(networkAddress=netbios:EVS01)))" [07:10:40] Leaving ScEnumerateExchangeServersInCluster [07:10:40] Leaving ScEnumerateExchangeServersInCluster2 [07:10:40] Responsible MTA server name: EVS01

Page 251: Exchange Server Admin Troubleshooting

50 Module 5: Clustering

[07:10:40] Context data "ClusterToBind" (CStr), previously set to "localhost", being reset to "node02.exchange.org" ClusterToBind (f:\titanium\admin\src\udog\inc\exsetctx.hxx:146) The operation has completed successfully. [07:10:40] Context data "LocalServer" (CStr), previously set to "NODE02", being reset to "EVS01" LocalServer (f:\titanium\admin\src\udog\inc\exsetctx.hxx:139) The operation has completed successfully. [07:10:40] Context data "ResponsibleMTAServer" (CStr), previously set to "NODE02", being reset to "EVS01" ResponsibleMTAServer (f:\titanium\admin\src\udog\inc\exsetctx.hxx:138) The operation has completed successfully. [07:10:40] Entering CFileManager::ScSetInstallDestDir(sz) [07:10:40] Leaving CFileManager::ScSetInstallDestDir(sz) [07:10:40] Entering ScCheckLocalAndRemoteExchangeBuilds [07:10:40] Entering ScGetExchangeBuild [07:10:40] Exchange build number on server 'node02.exchange.org' is '6944.0' [07:10:40] Leaving ScGetExchangeBuild [07:10:40] Entering ScGetExchangeBuild [07:10:40] Exchange build number on server 'localhost' is '6944.0' [07:10:40] Leaving ScGetExchangeBuild [07:10:40] Leaving ScCheckLocalAndRemoteExchangeBuilds [07:10:40] Entering ScCheckStateOfDependentResources [07:10:40] Entering ScBindToEVS [07:10:40] Binding to cluster'node02.exchange.org' [07:10:40] Binding to EVS 'EVS01' [07:10:40] Leaving ScBindToEVS [07:10:40] Leaving ScCheckStateOfDependentResources [07:10:40] Entering ScDetermineIfLocalDomainServerGroupHasAlreadyBeenACLedOnExchangeCT [07:10:40] Checking to see whether "EXCHANGE\Exchange Domain Servers" has READ permissions on the Microsoft Exchange container [07:10:40] Entering ScTestAceOnObject [07:10:40] Attempting to get DOB for DN "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange" [07:10:40] Attempting to read security descriptor from DOB [07:10:40] Attempting to initialize CAce object [07:10:40] Testing to see if given ACE is present [07:10:40] Test succeeded; fACLPresent = TRUE, fExtraRights = FALSE [07:10:40] The ACE tested for is present in the ACL of this object [07:10:40] Leaving ScTestAceOnObject [07:10:40] The Domain Servers group does have READ permissions on the Exchange container [07:10:40] Checking to see whether ANONYMOUS LOGON has READ permissions for MDB objects on the organization [07:10:40] Entering ScTestAceOnObject [07:10:46] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=IMAP4/cn=1/CN=Sessions [07:10:46] ScCreateIMAPInstance : Completed [07:10:46] Leaving CAtomIMAP4::ScAddDSObjects [07:10:46] Entering CAtomPOP3::ScAddDSObjects [07:10:46] Creating Active Directory objects for POP3 Service [07:10:46] Entering CBaseExchangeProtocolAtom::ScCreateServerLevelProtocolCT [07:10:46] Leaving CBaseExchangeProtocolAtom::ScCreateServerLevelProtocolCT [07:10:46] Entering CBaseExchangeProtocolAtom::ScSetSASLMechanismsProtocolCT

Page 252: Exchange Server Admin Troubleshooting

Module 5: Clustering 51

[07:10:46] Leaving CBaseExchangeProtocolAtom::ScSetSASLMechanismsProtocolCT [07:10:46] Entering CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:46] Searching for default instance [07:10:46] No default instance found, generating new instance id [07:10:46] New instance id is 1 [07:10:46] Leaving CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:46] ScCreatePOPInstanace : Entered [07:10:46] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=POP3/cn=1 [07:10:46] attribute 894 = '1§CLEARTEXT' [07:10:46] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=POP3/cn=1/CN=Sessions [07:10:46] ScCreatePOPInstanace : Completed [07:10:46] Leaving CAtomPOP3::ScAddDSObjects [07:10:46] Entering CAtomDAV::ScAddDSObjects [07:10:46] Creating Active Directory objects for Base DAV protocol [07:10:46] Entering CAtomDAV::ScCreateOrRefreshDSObjects [07:10:46] Entering CBaseExchangeProtocolAtom::ScCreateServerLevelProtocolCT [07:10:46] Leaving CBaseExchangeProtocolAtom::ScCreateServerLevelProtocolCT [07:10:46] Entering CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:46] Searching for default instance [07:10:46] No default instance found, generating new instance id [07:10:46] New instance id is 100 [07:10:46] Leaving CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:46] ScCreateHTTPInstance : Entered [07:10:46] ScCreateHTTPInstance : Completed [07:10:46] ScCreateHTTPVirtualDirectoryInstanceEx : Entered [07:10:46] ScCreateHTTPVirtualDirectoryInstanceEx : Completed [07:10:46] ScCreateHTTPVirtualDirectoryInstanceEx : Entered [07:10:46] ScCreateHTTPVirtualDirectoryInstanceEx : Completed [07:10:46] ScCreateHTTPVirtualDirectoryInstanceEx : Entered [07:10:47] ScCreateHTTPVirtualDirectoryInstanceEx : Completed [07:10:47] Entering CAtomDAV::ScUpdateVirtualServers [07:10:47] Searching for HTTP virtual servers, base DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP [07:10:47] Entering CAtomDAV::ScUpdateVirtualServerInstance [07:10:47] Updating virtual server DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100 [07:10:47] Searching for HTTP virtual directories, base DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100 [07:10:47] Updating virtual directory DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100/cn=Public

Page 253: Exchange Server Admin Troubleshooting

52 Module 5: Clustering

[07:10:47] Updating virtual directory DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100/cn=Exchange [07:10:47] Updating virtual directory DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100/cn=Exadmin [07:10:47] Leaving CAtomDAV::ScUpdateVirtualServerInstance [07:10:47] Leaving CAtomDAV::ScUpdateVirtualServers [07:10:47] Leaving CAtomDAV::ScCreateOrRefreshDSObjects [07:10:47] Leaving CAtomDAV::ScAddDSObjects [07:10:47] Entering CAtomSMTP::ScAddDSObjects [07:10:47] Creating Active Directory objects for SMTP Service [07:10:47] Entering CBaseExchangeProtocolAtom::ScCreateServerLevelProtocolCT [07:10:47] Leaving CBaseExchangeProtocolAtom::ScCreateServerLevelProtocolCT [07:10:47] Entering ScLookupDefaultPrivateMDB [07:10:47] Entering ScLookupDefaultStorageGroup [07:10:47] Entering ScLookupChildObjects [07:10:47] Leaving ScLookupChildObjects [07:10:47] Leaving ScLookupDefaultStorageGroup [07:10:47] Entering ScLookupChildObjects [07:10:47] Leaving ScLookupChildObjects [07:10:47] Leaving ScLookupDefaultPrivateMDB [07:10:47] Entering ScLookupDefaultPublicMDB [07:10:47] Entering ScLookupDefaultStorageGroup [07:10:47] Entering ScLookupChildObjects [07:10:47] Leaving ScLookupChildObjects [07:10:47] Leaving ScLookupDefaultStorageGroup [07:10:47] Entering ScLookupChildObjects [07:10:47] Leaving ScLookupChildObjects [07:10:47] Leaving ScLookupDefaultPublicMDB [07:10:47] Entering CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:47] Searching for default instance [07:10:47] No default instance found, generating new instance id [07:10:47] New instance id is 1 [07:10:47] Leaving CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:47] ScCreateSmtpInstance : Entered [07:10:47] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=SMTP/cn=1 [07:10:47] attribute 1433 = '0000001e' [07:10:47] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=SMTP/cn=1/CN=RoutingSources [07:10:47] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=SMTP/cn=1/CN=Sessions [07:10:47] Creating object at /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=SMTP/cn=1/CN=Domain [07:10:47] ScCreateSmtpInstance : Completed

Page 254: Exchange Server Admin Troubleshooting

Module 5: Clustering 53

[07:10:47] Leaving CAtomSMTP::ScAddDSObjects [07:10:47] Entering CAtomBrowse::ScAddDSObjects [07:10:47] Creating Active Directory objects for Microsoft Exchange OMA Browse [07:10:47] Entering CAtomBrowse::ScCreateOrRefreshDSObjects [07:10:47] Determining correct vsi within cluster [07:10:47] Entering CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:47] Searching for default instance [07:10:47] Found default instance, id is 100 [07:10:47] Leaving CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:47] Creating browse vdir on virtual server "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100" [07:10:47] ScCreateHTTPVirtualDirectoryInstanceEx : Entered [07:10:47] ScCreateHTTPVirtualDirectoryInstanceEx : Completed [07:10:47] Setting HAS_WIRELESS_BROWSE bit for the HTTP server object [07:10:47] Entering ScSetFlagOnServerHeuristics [07:10:47] Leaving ScSetFlagOnServerHeuristics [07:10:47] Leaving CAtomBrowse::ScCreateOrRefreshDSObjects [07:10:47] Leaving CAtomBrowse::ScAddDSObjects [07:10:47] Entering CAtomOMASync::ScAddDSObjects [07:10:47] Creating Active Directory objects for Exchange ActiveSync [07:10:47] Entering CAtomOMASync::ScCreateOrRefreshDSObjects [07:10:47] Determining correct vsi within cluster [07:10:47] Entering CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:47] Searching for default instance [07:10:47] Found default instance, id is 100 [07:10:47] Leaving CBaseExchangeProtocolAtom::ScFindDefaultInstanceIdOrCreateNew [07:10:47] Creating sync vdir on virtual server "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP/cn=100" [07:10:47] Entering ::ScCreateOMASyncVirtualDirectoryInstance [07:10:47] Detected IIS6 or later. Setting up sync vdir as pooled [07:10:47] ScCreateHTTPVirtualDirectoryInstanceEx : Entered [07:10:47] ScCreateHTTPVirtualDirectoryInstanceEx : Completed [07:10:47] Leaving ::ScCreateOMASyncVirtualDirectoryInstance [07:10:47] Setting HAS_WIRELESS_SYNC bit for the HTTP server object [07:10:47] Entering ScSetFlagOnServerHeuristics [07:10:47] Leaving ScSetFlagOnServerHeuristics [07:10:47] Leaving CAtomOMASync::ScCreateOrRefreshDSObjects [07:10:47] Leaving CAtomOMASync::ScAddDSObjects [07:10:47] Entering ScCreateExchangeClusterResources [07:10:47] Entering ScServerOwnsMTA [07:10:47] Leaving ScServerOwnsMTA [07:10:47] Entering ScBindToEVS [07:10:47] Binding to cluster'node02.exchange.org' [07:10:47] Binding to EVS 'EVS01' [07:10:47] Leaving ScBindToEVS [07:10:47] Entering ScFindSystemAttendantResource [07:10:47] Searching for System Attendant resource [07:10:47] Found System Attendant resource named 'EVS01 SA' [07:10:47] Leaving ScFindSystemAttendantResource [07:10:47] Entering ScSetNetworkNameOnResource [07:10:47] Setting property 'NetworkName'='EVS01' [07:10:47] Leaving ScSetNetworkNameOnResource

Page 255: Exchange Server Admin Troubleshooting

54 Module 5: Clustering

[07:10:47] Entering ScSetVersionPropertiesOnResource [07:10:47] Setting property 'ResourceVersion'='393221' [07:10:47] Setting property 'ResourceBuild'='455081984' [07:10:47] Leaving ScSetVersionPropertiesOnResource [07:10:47] Searching for resources 'Microsoft Exchange Message Transfer Agent' [07:10:47] No 'Microsoft Exchange Message Transfer Agent' resource in the group [07:10:47] Creating resource 'Exchange Message Transfer Agent Instance (EVS01)' of type 'Microsoft Exchange Message Transfer Agent' [07:10:48] Setting dependency: 'Exchange Message Transfer Agent Instance (EVS01)' -> 'EVS01 SA' [07:10:48] Setting property 'DestPath'='s:\mount\exchsrvr\MTADATA' [07:10:48] Setting property 'NetworkName'='EVS01' [07:10:49] Searching for resources 'Microsoft Exchange Information Store' [07:10:49] No 'Microsoft Exchange Information Store' resource in the group [07:10:49] Creating resource 'Exchange Information Store Instance (EVS01)' of type 'Microsoft Exchange Information Store' [07:10:50] Setting dependency: 'Exchange Information Store Instance (EVS01)' -> 'EVS01 SA' [07:10:50] Setting property 'DestPath'='s:\mount\exchsrvr\MDBDATA' [07:10:50] Setting property 'NetworkName'='EVS01' [07:10:50] Searching for resources 'Microsoft Exchange Routing Service' [07:10:50] No 'Microsoft Exchange Routing Service' resource in the group [07:10:50] Creating resource 'Exchange Routing Service Instance (EVS01)' of type 'Microsoft Exchange Routing Service' [07:10:50] Setting dependency: 'Exchange Routing Service Instance (EVS01)' -> 'EVS01 SA' [07:10:50] Setting property 'NetworkName'='EVS01' [07:10:50] Searching for resources 'Microsoft Search Service Instance' [07:10:50] No 'Microsoft Search Service Instance' resource in the group [07:10:50] Creating resource 'Exchange MS Search Instance (EVS01)' of type 'Microsoft Search Service Instance' [07:10:50] Setting dependency: 'Exchange MS Search Instance (EVS01)' -> 'EVS01 SA' [07:10:50] Setting property 'ApplicationName'='ExchangeServer_EVS01' [07:10:50] Setting property 'ApplicationPath'='s:\mount\exchsrvr\ExchangeServer_EVS01' [07:10:51] Searching for resources 'Microsoft Exchange SMTP Server Instance' [07:10:51] Searching for AD objects for 'SMTP' [07:10:51] No resource for instance '1' [07:10:51] Creating resource 'SMTP Virtual Server Instance 1 (EVS01)' of type 'Microsoft Exchange SMTP Server Instance' [07:10:51] Setting dependency: 'SMTP Virtual Server Instance 1 (EVS01)' -> 'EVS01 SA' [07:10:51] Setting property 'NetworkName'='EVS01' [07:10:51] Setting property 'ProtocolInstanceId'='1' [07:10:51] Searching for resources 'Microsoft Exchange DAV Server Instance' [07:10:51] Searching for AD objects for 'HTTP' [07:10:51] No resource for instance '100' [07:10:51] Creating resource 'Exchange HTTP Virtual Server Instance 100 (EVS01)' of type 'Microsoft Exchange DAV Server Instance' [07:10:51] Setting dependency: 'Exchange HTTP Virtual Server Instance 100 (EVS01)' -> 'EVS01 SA' [07:10:51] Setting property 'NetworkName'='EVS01' [07:10:51] Setting property 'ProtocolInstanceId'='100' [07:10:51] Entering ScUpdatePropertiesOnGroup [07:10:51] Processing group 'EVS01'

Page 256: Exchange Server Admin Troubleshooting

Module 5: Clustering 55

[07:10:51] Setting property 'MSExchange_VirtualServerName'='EVS01' [07:10:51] Leaving ScUpdatePropertiesOnGroup [07:10:51] Leaving ScCreateExchangeClusterResources [07:10:51] Leaving ScSetupExchangeVirtualServer

Page 257: Exchange Server Admin Troubleshooting

56 Module 5: Clustering

Upgrading an Exchange Virtual Server to Exchange 2003

The first step to upgrade an Exchange 2000 cluster is to install the Exchange 2003 binary files on the cluster Nodes. The Exchange 2003 binary files can only be installed on a passive node in the cluster. Setup will not continue if you attempt to install them on an active node.

The process of installing the binary files on a passive Node does not really differ in any way from Exchange 2000.

For this scenario we can assume an Active/(Passive Exchange 2000 cluster that will be upgraded to Exchange 2003. We can also assume that the other installation prerequisites have been met.

The steps to upgrade are as follows:

On the passive Node run setup.exe and upgrade the binary files. In this case it is called CLU-NODE02.

Page 258: Exchange Server Admin Troubleshooting

Module 5: Clustering 57

Assuming that all the installation prerequisites have been met, the installation of the files should complete. In Exchange 2000 clusters we recommended that you have a Microsoft Distributed Transaction Coordinator (MSDTC) resource in the cluster. There was no hard requirement for this. In Exchange 2003 there is, so if the Exchange 2003 setup does not detect the presence of an MSDTC resource it will fail and report an error to the user.

Once the binary files have been installed you must reboot the server. During the installation, the Exchange 2000 binaries are updated. The Exchange resource .dll file exres.dll is also updated. To be sure that we are using the correct version of exres.dll, it is advisable to reboot the server.

Page 259: Exchange Server Admin Troubleshooting

58 Module 5: Clustering

When we have rebooted the passive Node, we then have to take the Exchange Virtual Server on the active node offline and then fail it over to the passive Exchange 2003 node. If we attempt to just fail the Exchange Virtual Server over, it will not come online. This is because we have to manually initiate the upgrade process (we will look at this later).

It is advisable to now upgrade the binary files on node CLU-NODE01. If were to upgrade the Exchange Virtual Server at this point, we would not be able to fail it over to the other Node in the event of problems. This will obviously cause more downtime for the entire cluster, as we will have the Exchange Virtual Server offline while we upgrade the binaries on CLU-NODE01.

The process of installing the binary files is logged in the progress log. A sample progress log from the binary upgrade process is included at the end of this document.

Page 260: Exchange Server Admin Troubleshooting

Module 5: Clustering 59

Using Cluster Administrator, we now have to manually upgrade the Exchange Virtual Server to Exchange 2003. This will invoke the creation of the Active Directory computer object for the Exchange Virtual Server thus enabling Kerberos support. (See the section on Kerberos for more details.)

In order for the upgrade process to successfully complete, the Network Name resource and the Physical Disk resources associated with the Exchange Virtual Server must be online. If they are not, you will receive the following errors:

Page 261: Exchange Server Admin Troubleshooting

60 Module 5: Clustering

Bring all the required resources online and try again.

Page 262: Exchange Server Admin Troubleshooting

Module 5: Clustering 61

When the upgrade process has completed, we will be notified of a success.

The Exchange Virtual Server is still offline at this point, but the cluster is effectively running Exchange 2003.

Bring the Exchange 2003 resources online.

Page 263: Exchange Server Admin Troubleshooting

62 Module 5: Clustering

We will now have a computer object in the Active Directory for the Exchange Virtual Server.

During the upgrade of the Exchange Virtual Server, the RequireKerberos setting is set and can be seen by issuing the following command:

The RequireKerberos value is set to 1. This is to enable Kerberos authentication against the Exchange Virtual Server: For more details see the section on Kerberos.

The RequireKerberos value is set to 0 by default, though in the screenshot above, it has been manually changed by the administrator to 1.

Page 264: Exchange Server Admin Troubleshooting

Module 5: Clustering 63

A few customers may desire to scale-out the number of Exchange Virtual Servers in a cluster, while simultaneously upgrading their operating systems. Here are the steps to do so:

1. Start off with Exchange 2000 SP3 running on Windows 2000 SP3.

2. In-place upgrade Exchange 2000 to Exchange 2003 on node 1 (passive).

3. Fail-over and in-place upgrade Exchange 2000 to Exchange 2003 on node 2 (now passive).

4. In-place upgrade Windows 2000 to Windows 2003 on node 2 (passive).

5. Fail-over and in-place upgrade Windows 2000 to Windows 2003 on node 1 (now passive).

6. Add more nodes.

7. Create the appropriate number of new Exchange Virtual Server in Exchange 2003.

Upgrading the number of cluster nodes with Exchange 2003 and Windows 2003

Page 265: Exchange Server Admin Troubleshooting

64 Module 5: Clustering

Removing an Exchange Virtual Server

In order to remove an Exchange Virtual Server from an Exchange 2000 cluster, we simply deleted the System Attendant Resource in the Exchange Cluster Group. This would then take care of removing all the Exchange 2000 related cluster resources and also remove the Exchange Virtual Server information from Active Directory. This has been changed for Exchange 2003.

When we create an Exchange 2003 Exchange Virtual Server, a cluster private property called MSExchange_VirtualServerName is added to the cluster group which houses the Exchange Virtual Server. This attribute is used to inform the cluster that this group contains an Exchange Virtual Server. If the Exchange Virtual Server is removed from the cluster group in an unsupported way, then this attribute will remain in place to inform us that the unsupported action has taken place and also that the Active Directory objects for the Exchange Virtual Server have not been removed.

Page 266: Exchange Server Admin Troubleshooting

Module 5: Clustering 65

The attribute MSExchange_VirtualServerName can be seen by running the following command:

Cluster group myGroup /priv

The attribute is a private property of the Cluster Group that houses the Exchange Virtual Server. This property is set on the group during the creation of the Exchange Virtual Server and can be seen in the Progress Log:

[03:04:48] Entering ScUpdatePropertiesOnGroup [03:04:48] Processing group 'EVS01' [03:04:48] Setting property 'MSExchange_VirtualServerName'='EVS01' [03:04:48] Leaving ScUpdatePropertiesOnGroup

It can also be seen by analyzing the cluster log:

00000550.00000100::2003/04/20-01:04:48.785 INFO [DM] DmUpdateSetValue 00000550.00000100::2003/04/20-01:04:48.785 INFO [DM] Setting value of MSExchange_VirtualServerName for key Groups\067d3e11-1203-4501-8ce7-afe56608590a\Parameters to EVS01

This property should not be changed in any way. If this value is different from the name of the Exchange Virtual Server, then it has probably been changed manually and should be changed back to reflect the Network Name resource that represents the Exchange Virtual Server.

Page 267: Exchange Server Admin Troubleshooting

66 Module 5: Clustering

The supported methods for removing an Exchange Virtual Server are by right-clicking on either 1) the cluster group containing the Exchange Virtual Server or 2) the System Attendant Resource and choosing Remove Exchange Virtual Server.

In order to successfully remove an Exchange Virtual Server you need to have Exchange Full Administrator permissions on the Administrative Group that the Exchange Virtual Server belongs to.

If this is the last Exchange 2003 Server in the organization, then you will need to Exchange Full Administrator permissions on the organization.

The Windows 2000/2003 cluster service account does not require any permissions on the Organization or Administrative Group.

We will then receive a confirmation dialog box.

Click Yes to continue.

Note

Page 268: Exchange Server Admin Troubleshooting

Module 5: Clustering 67

In order to successfully remove an Exchange Virtual Server we must either move or delete all mailboxes that were hosted on the Exchange Virtual Server (this is the same behavior as a standalone Exchange 2003 server). If we attempt to remove the Exchange Virtual Server with mailboxes still present we will receive the following error:

Move or delete the mailboxes and then try to remove the Exchange Virtual Server again.

The System Attendant resource must be offline in order to successfully remove it. If you attempt to remove the Exchange Virtual Server while the System Attendant is still online, you will receive the following error:

Take the System Attendant resource offline and try the operation again.

Each Exchange 2003 cluster will only have one message transfer agent (MTA) resource for the entire cluster. This is by design. By default it is the first Exchange Virtual Server in the cluster that will get the MTA resource. If we are attempting to remove the group that owns the MTA resource and there are other Exchange Virtual Servers still available in the cluster, then we will receive the following error:

Page 269: Exchange Server Admin Troubleshooting

68 Module 5: Clustering

In this situation we cannot remove the Exchange Virtual Server, as it hosts the MTA resource required by all the Exchange Virtual Servers that still exist in the cluster. One option would be to move all mailboxes from another Exchange Virtual Server in the cluster to the Exchange Virtual Server that hosts the MTA resource and then remove that Exchange Virtual Server instead. There is no supported method for moving an MTA resource from an Exchange Virtual Server to another Exchange Virtual Server.

Problem removing Exchange Virtual Server hosting an MTA resource in an Active/Active or Active/Active/Active/Passive cluster

Page 270: Exchange Server Admin Troubleshooting

Module 5: Clustering 69

If all these conditions have been met, then the Exchange Virtual Server will be successfully removed. We will not receive any dialogs to say that it was successful, but all the Exchange 2003 resources should have been deleted from the Exchange Virtual Server cluster group and the computer object will be removed from Active Directory.

The removal process will be logged in the Progress Log. This log can be reviewed for reference afterwards. The entire contents of the progress log removal process can be found at the bottom of this document.

Page 271: Exchange Server Admin Troubleshooting

70 Module 5: Clustering

After the Exchange Virtual Server has been successfully removed, the MSExchange_VirtualServerName property will be removed from the cluster group:

If a customer has deleted the Exchange 2003 System Attendant resource by mistake, we can use the following procedures to rebuild the Exchange Virtual Server based on the Active Directory information.

We can take a look at the deletion process and how to recreate the Exchange Virtual Server.

Remember that this is not the supported method. We are doing this to illustrate how we recreate the Exchange Virtual Server.

Deleting the System Attendant Resource

Page 272: Exchange Server Admin Troubleshooting

Module 5: Clustering 71

An Administrator may choose to simply delete the System Attendant resource for the Exchange Virtual Server.

We will then be prompted to confirm the deletion.

Page 273: Exchange Server Admin Troubleshooting

72 Module 5: Clustering

The System Attendant resource and all resources that have a dependency on it will now be listed. Click Yes to confirm the deletion.

All the Exchange 2003 resources will now be deleted from the Exchange Virtual Server cluster group.

If we now check the MSExchange_VirtualServerName attribute for the Exchange Virtual Server cluster group. we notice that it is still in place.

Page 274: Exchange Server Admin Troubleshooting

Module 5: Clustering 73

In this scenario we will also receive an error in the application log stating that the System Attendant resource has been deleted improperly but the Exchange Virtual Server still exists.

The full text of the error is as follows:

EVS01 System Attendant: System Attendant resource has been deleted improperly. The objects in Active Directory corresponding to this Exchange Virtual Server have not been removed. If you unintentionally deleted the System Attendant resource, you can restore it and its dependent resources to their original configuration by re-creating the System Attendant resource using the same parameters as before. If you want to remove the Exchange Virtual Server corresponding to this System Attendant resource, you have to re-create the System Attendant resource with the same parameters as before and then select the option "Remove Exchange Virtual Server" in the context menu of the group or System Attendant resource using the Cluster Administrator tool. For more information, click http://www.microsoft.com/contentredirect.asp.

Page 275: Exchange Server Admin Troubleshooting

74 Module 5: Clustering

If we now look in Exchange System Manager we can see that the Exchange Virtual Server still exists.

Page 276: Exchange Server Admin Troubleshooting

Module 5: Clustering 75

In this situation we have to recreate the Exchange Virtual Server based on the original parameters in order to successfully remove it afterwards.

The steps are as follows:

In the original Exchange Virtual Server cluster group, make sure that the disks, IP addresses and the Network Name resources are all online. Then right-click in the group and choose New Resource. Choose Microsoft Exchange System Attendant.

Set all the required possible owners of the resource and click Next.

Page 277: Exchange Server Admin Troubleshooting

76 Module 5: Clustering

Page 278: Exchange Server Admin Troubleshooting

Module 5: Clustering 77

Set a dependency on all the disks and IP addresses as you had in the original Exchange Virtual Server.

The choice of Administrative Group will be presented. Notice that this cannot be changed because Cluster administrator reads only one Admin Group in Active Directory.

Page 279: Exchange Server Admin Troubleshooting

78 Module 5: Clustering

The Routing Group will now be presented.

The data drive is then presented. Notice that it is grayed out. This information is read in from Active Directory and cannot be changed.

Page 280: Exchange Server Admin Troubleshooting

Module 5: Clustering 79

We will then be presented with the summary of the creation process. Click finish to initiate the creation.

When this is done we will now have all the Exchange 2003 cluster resources back in place. The supported method of removal can now begin.

Beginning on the next page you will find an excerpt from the progress log for a successful removal of the Exchange Virtual Server:

Page 281: Exchange Server Admin Troubleshooting

80 Module 5: Clustering

[07:01:41] ************** Beginning Setup run ************** [07:01:41] Starting Exchange 6944 setup on Windows 5.2.3790. at 07:01:41 07/03/2003 [07:01:41] Entering CFileManager::ScInit [07:01:41] Entering CFileManager::ScAutoDetectDirectoryLocations [07:01:42] Leaving CFileManager::ScAutoDetectDirectoryLocations [07:01:42] Leaving CFileManager::ScInit [07:01:42] Entering CRegistryManager::ScInit [07:01:42] Leaving CRegistryManager::ScInit [07:01:42] Entering CDirectoryManager::ScInit [07:01:42] Entering ScIsComputerMemberOfDomain [07:01:42] NetGetJoinInformation: Domain/workgroup = "EXCHANGE" [07:01:42] NetGetJoinInformation: NETSETUP_JOIN_STATUS = 3 [07:01:42] The computer is a member of a domain [07:01:42] Leaving ScIsComputerMemberOfDomain [07:01:42] Entering CDirectoryManager::ScGetLocalDomainInformation [07:01:42] Getting information about the local domain [07:01:42] m_strLocalServer = "NODE02" [07:01:42] m_strLocalSite = "Default-First-Site-Name" [07:01:42] DsRoleGetPrimaryDomainInformation returned: [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::MachineRole = 3 [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::Flags = 1000000 [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "EXCHANGE" [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "exchange.org" [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "exchange.org" [07:01:42] Entering CDirectoryManager::ScCheckCommandLineForDC [07:01:42] Leaving CDirectoryManager::ScCheckCommandLineForDC [07:01:42] No user-specified DC; setup has chosen m_strDC = "DC" [07:01:42] schema master server name: DC [07:01:42] schema master domain : /dc=org/dc=exchange [07:01:42] m_strSchemaMasterDC = "DC" [07:01:42] m_strSchemaMasterDCDomainDN = "/dc=org/dc=exchange" [07:01:42] strConfigNC = "CN=Configuration,DC=exchange,DC=org" [07:01:42] m_strRootDomain = "exchange.org" [07:01:42] m_strOwnershipControlDC = "DC" [07:01:42] m_strPermissionControlDC = "DC" [07:01:42] Leaving CDirectoryManager::ScGetLocalDomainInformation [07:01:42] Entering CDirectoryManager::ScInitializeSessions [07:01:42] Entering CDirectoryManager::ScGetOrgLevelObjectStatus [07:01:42] Entering CDirectoryManager::ScSchemaIsUpToDate [07:01:42] Entering ScGetSchemaVersion [07:01:42] About to create the dob for object /dc=org/dc=exchange/cn=Configuration/cn=Schema/cn=ms-Exch-Schema-Version-Pt [07:01:42] The schema version identified for the Server is 6870 [07:01:42] Leaving ScGetSchemaVersion [07:01:42] Leaving CDirectoryManager::ScSchemaIsUpToDate [07:01:42] Entering ScGetMicrosoftExchangeCTHeuristics [07:01:42] Leaving ScGetMicrosoftExchangeCTHeuristics [07:01:42] Entering CDirectoryManager::ScGetCountOfOrgsInDomain [07:01:42] Leaving CDirectoryManager::ScGetCountOfOrgsInDomain [07:01:42] Leaving CDirectoryManager::ScGetOrgLevelObjectStatus [07:01:42] Entering CDirectoryManager::ScDeterminePermissionLevel [07:01:42] Checking permissions in the Config NC: /dc=org/dc=exchange/cn=Configuration/cn=Services [07:01:42] We have permission ConfigNC_Read [07:01:42] We have permission ConfigNC_Write [07:01:42] We have permission ConfigNC_SetPerms

Page 282: Exchange Server Admin Troubleshooting

Module 5: Clustering 81

[07:01:42] Checking permissions on the Schema container: /dc=org/dc=exchange/cn=Configuration/cn=Schema [07:01:42] We have permission ConfigNC_UpdateSchema [07:01:42] Checking permissions in the Domain NC: /dc=org/dc=exchange [07:01:42] We have permission DomainNC_Read [07:01:42] We have permission DomainNC_Write [07:01:42] Checking to see if an Exchange org exists [07:01:42] Found the organization "EXCHANGE" [07:01:42] Checking read permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups [07:01:42] We have permission ExchOrg_Read [07:01:42] Checking write/security permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE [07:01:42] We have permission (ExchOrg_Write | ExchAG_Write) [07:01:42] We have permission (ExchOrg_SetPerms | ExchAG_SetPerms) [07:01:42] Looking for an existing server object [07:01:42] Didn't find an existing server object [07:01:42] Enumerating all admin groups in the org [07:01:42] Found 1 admin groups [07:01:42] Checking permissions on the admin group: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group [07:01:42] We have permission ExchAG_Read [07:01:42] We have permission ExchAG_Write [07:01:42] We have permission ExchAG_SetPerms [07:01:42] Final set of permissions: 0XF0C0E0E0 [07:01:42] Leaving CDirectoryManager::ScDeterminePermissionLevel [07:01:42] We have sufficient admin rights, the schema is up to date and org-level objects are present on the local DC; m_strDCToUse = "DC" [07:01:42] Sanity check: [07:01:42] m_strDCToUse = "DC" [07:01:42] m_psesToUse->m_strServerName = "DC" [07:01:42] Leaving CDirectoryManager::ScInitializeSessions [07:01:42] Leaving CDirectoryManager::ScInit [07:01:42] === IGNORING PREVIOUS ERRORS === CExchangeSetupCtx::ScDetermineExchangeObjectStateFromDS (f:\titanium\admin\src\udog\excommon\exsetctx.cxx:411) The operation has completed successfully. [07:01:42] Entering ScSetupExchangeVirtualServer [07:01:42] Performing remove for EVS 'EVS01' [07:01:42] Entering CDirectoryManager::ScReInitWithDC [07:01:42] Reinitalizing the DS Manager, using the DC DC [07:01:42] Entering CDirectoryManager::ScInit [07:01:42] Entering ScIsComputerMemberOfDomain [07:01:42] NetGetJoinInformation: Domain/workgroup = "EXCHANGE" [07:01:42] NetGetJoinInformation: NETSETUP_JOIN_STATUS = 3 [07:01:42] The computer is a member of a domain [07:01:42] Leaving ScIsComputerMemberOfDomain [07:01:42] Entering CDirectoryManager::ScGetLocalDomainInformation [07:01:42] Getting information about the local domain [07:01:42] m_strLocalServer = "NODE02" [07:01:42] m_strLocalSite = "Default-First-Site-Name" [07:01:42] DsRoleGetPrimaryDomainInformation returned: [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::MachineRole = 3 [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::Flags = 1000000

Page 283: Exchange Server Admin Troubleshooting

82 Module 5: Clustering

[07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameFlat = "EXCHANGE" [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainNameDns = "exchange.org" [07:01:42] DSROLE_PRIMARY_DOMAIN_INFORMATION::DomainForestName = "exchange.org" [07:01:42] User has specified a DC; m_strDC = "DC" [07:01:42] schema master server name: DC [07:01:42] schema master domain : /dc=org/dc=exchange [07:01:42] m_strSchemaMasterDC = "DC" [07:01:42] m_strSchemaMasterDCDomainDN = "/dc=org/dc=exchange" [07:01:42] strConfigNC = "CN=Configuration,DC=exchange,DC=org" [07:01:42] m_strRootDomain = "exchange.org" [07:01:42] m_strOwnershipControlDC = "DC" [07:01:42] m_strPermissionControlDC = "DC" [07:01:42] Leaving CDirectoryManager::ScGetLocalDomainInformation [07:01:42] Entering CDirectoryManager::ScInitializeSessions [07:01:42] Entering CDirectoryManager::ScGetOrgLevelObjectStatus [07:01:42] Entering CDirectoryManager::ScSchemaIsUpToDate [07:01:42] Leaving CDirectoryManager::ScSchemaIsUpToDate [07:01:42] Entering ScGetMicrosoftExchangeCTHeuristics [07:01:42] Leaving ScGetMicrosoftExchangeCTHeuristics [07:01:42] Entering CDirectoryManager::ScGetCountOfOrgsInDomain [07:01:42] Leaving CDirectoryManager::ScGetCountOfOrgsInDomain [07:01:42] Leaving CDirectoryManager::ScGetOrgLevelObjectStatus [07:01:42] Entering CDirectoryManager::ScDeterminePermissionLevel [07:01:42] Checking permissions in the Config NC: /dc=org/dc=exchange/cn=Configuration/cn=Services [07:01:42] We have permission ConfigNC_Read [07:01:42] We have permission ConfigNC_Write [07:01:42] We have permission ConfigNC_SetPerms [07:01:42] Checking permissions on the Schema container: /dc=org/dc=exchange/cn=Configuration/cn=Schema [07:01:42] We have permission ConfigNC_UpdateSchema [07:01:42] Checking permissions in the Domain NC: /dc=org/dc=exchange [07:01:42] We have permission DomainNC_Read [07:01:42] We have permission DomainNC_Write [07:01:42] Checking to see if an Exchange org exists [07:01:42] Found the organization "EXCHANGE" [07:01:42] Checking read permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups [07:01:42] We have permission ExchOrg_Read [07:01:42] Checking write/security permissions on the org: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE [07:01:42] We have permission (ExchOrg_Write | ExchAG_Write) [07:01:42] We have permission (ExchOrg_SetPerms | ExchAG_SetPerms) [07:01:42] Looking for an existing server object [07:01:42] Didn't find an existing server object [07:01:42] Enumerating all admin groups in the org [07:01:42] Found 1 admin groups [07:01:42] Checking permissions on the admin group: /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group [07:01:42] We have permission ExchAG_Read [07:01:42] We have permission ExchAG_Write [07:01:42] We have permission ExchAG_SetPerms [07:01:42] Final set of permissions: 0XF0C0E0E0 [07:01:42] Leaving CDirectoryManager::ScDeterminePermissionLevel

Page 284: Exchange Server Admin Troubleshooting

Module 5: Clustering 83

[07:01:42] We have sufficient admin rights, the schema is up to date and org-level objects are present on the local DC; m_strDCToUse = "DC" [07:01:42] Sanity check: [07:01:42] m_strDCToUse = "DC" [07:01:42] m_psesToUse->m_strServerName = "DC" [07:01:42] Leaving CDirectoryManager::ScInitializeSessions [07:01:42] Leaving CDirectoryManager::ScInit [07:01:42] Leaving CDirectoryManager::ScReInitWithDC [07:01:42] Entering ScGetDomainInfoFromServer [07:01:42] Leaving ScGetDomainInfoFromServer [07:01:42] Entering ScGetExchangeDsConfigFromDs [07:01:42] Looking for Exchange server object 'EVS01' [07:01:42] Organization name is 'EXCHANGE' [07:01:42] Admin group name is 'First Administrative Group' [07:01:42] Admin group name of routing group is 'First Administrative Group' [07:01:42] Routing group name is 'First Routing Group' [07:01:42] Responsible MTA server name is 'EVS01' [07:01:42] Leaving ScGetExchangeDsConfigFromDs [07:01:42] Entering ScGetExchangeServicesInstallAndDataPathFromAD [07:01:42] Exchange data path on server 'EVS01' is 'S:\EXCHSRVR' [07:01:42] Exchange install path on server 'EVS01' is 'C:\Program Files\Exchsrvr' [07:01:42] Leaving ScGetExchangeServicesInstallAndDataPathFromAD [07:01:42] Entering ScEnumerateExchangeServersInCluster2 [07:01:42] Entering ScEnumerateExchangeServersInCluster [07:01:42] Getting cluster resource info... [07:01:42] Adding cluster network name resource 'CLUSTER' to list [07:01:42] Adding cluster network name resource 'EVS01' to list [07:01:42] Executing LDAP query "(&(objectCategory=msExchExchangeServer)(|(networkAddress=netbios:CLUSTER)(networkAddress=netbios:EVS01)))" [07:01:42] Found 1 server objects: [07:01:42] DN=/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01 [07:01:42] Leaving ScEnumerateExchangeServersInCluster [07:01:42] Leaving ScEnumerateExchangeServersInCluster2 [07:01:42] Responsible MTA server name: EVS01 [07:01:42] Context data "ClusterToBind" (CStr), previously set to "localhost", being reset to "node02.exchange.org" ClusterToBind (f:\titanium\admin\src\udog\inc\exsetctx.hxx:146) The operation has completed successfully. [07:01:42] Context data "LocalServer" (CStr), previously set to "NODE02", being reset to "EVS01" LocalServer (f:\titanium\admin\src\udog\inc\exsetctx.hxx:139) The operation has completed successfully. [07:01:42] Context data "ResponsibleMTAServer" (CStr), previously set to "NODE02", being reset to "EVS01" ResponsibleMTAServer (f:\titanium\admin\src\udog\inc\exsetctx.hxx:138) The operation has completed successfully. [07:01:42] Entering CFileManager::ScSetInstallDestDir(sz) [07:01:42] Leaving CFileManager::ScSetInstallDestDir(sz) [07:01:42] Entering ScCheckLocalAndRemoteExchangeBuilds [07:01:42] Entering ScGetExchangeBuild [07:01:42] Exchange build number on server 'node02.exchange.org' is '6944.0' [07:01:42] Leaving ScGetExchangeBuild [07:01:42] Entering ScGetExchangeBuild [07:01:42] Exchange build number on server 'localhost' is '6944.0'

Page 285: Exchange Server Admin Troubleshooting

84 Module 5: Clustering

[07:01:42] Leaving ScGetExchangeBuild [07:01:42] Leaving ScCheckLocalAndRemoteExchangeBuilds [07:01:42] Entering ScCheckStateOfDependentResources [07:01:42] Entering ScBindToEVS [07:01:42] Binding to cluster'node02.exchange.org' [07:01:42] Binding to EVS 'EVS01' [07:01:42] Leaving ScBindToEVS [07:01:43] Leaving ScCheckStateOfDependentResources [07:01:43] Entering ScDetermineIfLocalDomainServerGroupHasAlreadyBeenACLedOnExchangeCT [07:01:43] Checking to see whether "EXCHANGE\Exchange Domain Servers" has READ permissions on the Microsoft Exchange container [07:01:43] Entering ScTestAceOnObject [07:01:43] Attempting to get DOB for DN "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange" [07:01:43] Attempting to read security descriptor from DOB [07:01:43] Attempting to initialize CAce object [07:01:43] Testing to see if given ACE is present [07:01:43] Test succeeded; fACLPresent = TRUE, fExtraRights = FALSE [07:01:43] The ACE tested for is present in the ACL of this object [07:01:43] Leaving ScTestAceOnObject [07:01:43] The Domain Servers group does have READ permissions on the Exchange container [07:01:46] Checking to see whether ANONYMOUS LOGON has READ permissions for MDB objects on the organization [07:01:46] Entering ScTestAceOnObject [07:01:46] Attempting to get DOB for DN "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE" [07:01:46] Attempting to read security descriptor from DOB [07:01:46] Attempting to initialize CAce object [07:01:46] Testing to see if given ACE is present [07:01:46] Test succeeded; fACLPresent = TRUE, fExtraRights = FALSE [07:01:46] The ACE tested for is present in the ACL of this object [07:01:46] Leaving ScTestAceOnObject [07:01:46] ANONYMOUS LOGON does have READ permissions for MDB objects on the organization [07:01:46] Checking to see whether the Exchange Domain Servers group has been DENIED Receive-As permissions on the Servers container(s) [07:01:46] Checking the ACL on the Servers container in the admin group "First Administrative Group" [07:01:46] Entering ScTestAceOnObject [07:01:46] Attempting to get DOB for DN "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers" [07:01:46] Attempting to read security descriptor from DOB [07:01:46] Attempting to initialize CAce object [07:01:46] Testing to see if given ACE is present [07:01:46] Test succeeded; fACLPresent = TRUE, fExtraRights = FALSE [07:01:46] The ACE tested for is present in the ACL of this object [07:01:46] Leaving ScTestAceOnObject [07:01:46] The Exchange Domain Servers group has been DENIED Receive-As permissions on the Servers container(s) [07:01:46] The required permissions have already been set

Page 286: Exchange Server Admin Troubleshooting

Module 5: Clustering 85

[07:01:46] Leaving ScDetermineIfLocalDomainServerGroupHasAlreadyBeenACLedOnExchangeCT [07:01:46] Entering ScFindRoutingGroupThatContainsServer [07:01:46] Leaving ScFindRoutingGroupThatContainsServer [07:01:46] Entering ScPRQ_DoesNotContainLastMAPIMDBInMixedModeAG [07:01:46] This prerequisite has PASSED -- ID:62235 -- [07:01:46] Leaving ScPRQ_DoesNotContainLastMAPIMDBInMixedModeAG [07:01:46] Entering ScCheckStateOfSAResource [07:01:46] Entering ScBindToEVS [07:01:46] Binding to cluster'node02.exchange.org' [07:01:46] Binding to EVS 'EVS01' [07:01:46] Leaving ScBindToEVS [07:01:46] Entering ScFindSystemAttendantResource [07:01:46] Searching for System Attendant resource [07:01:46] Found System Attendant resource named 'EVS01 SA' [07:01:46] Leaving ScFindSystemAttendantResource [07:01:46] Leaving ScCheckStateOfSAResource [07:01:46] Entering ScDeleteExchangeClusterResources [07:01:46] Entering ScBindToEVS [07:01:46] Binding to cluster'node02.exchange.org' [07:01:46] Binding to EVS 'EVS01' [07:01:46] Leaving ScBindToEVS [07:01:46] Entering ScFindSystemAttendantResource [07:01:46] Searching for System Attendant resource [07:01:46] Found System Attendant resource named 'EVS01 SA' [07:01:46] Leaving ScFindSystemAttendantResource [07:01:46] Entering ScDeleteResourceRecursive [07:01:46] Opening resource named 'EVS01 SA' [07:01:46] Entering ScDeleteResourceRecursive [07:01:46] Opening resource named 'Exchange Message Transfer Agent Instance (EVS01)' [07:01:46] Deleting resource 'Exchange Message Transfer Agent Instance (EVS01)' [07:01:47] Leaving ScDeleteResourceRecursive [07:01:47] Entering ScDeleteResourceRecursive [07:01:47] Opening resource named 'Exchange Information Store Instance (EVS01)' [07:01:47] Deleting resource 'Exchange Information Store Instance (EVS01)' [07:01:47] Leaving ScDeleteResourceRecursive [07:01:47] Entering ScDeleteResourceRecursive [07:01:47] Opening resource named 'Exchange Routing Service Instance (EVS01)' [07:01:47] Deleting resource 'Exchange Routing Service Instance (EVS01)' [07:01:47] Leaving ScDeleteResourceRecursive [07:01:47] Entering ScDeleteResourceRecursive [07:01:47] Opening resource named 'Exchange MS Search Instance (EVS01)' [07:01:47] Deleting resource 'Exchange MS Search Instance (EVS01)' [07:01:48] Leaving ScDeleteResourceRecursive [07:01:48] Entering ScDeleteResourceRecursive [07:01:48] Opening resource named 'SMTP Virtual Server Instance 1 (EVS01)' [07:01:48] Deleting resource 'SMTP Virtual Server Instance 1 (EVS01)' [07:01:48] Leaving ScDeleteResourceRecursive [07:01:48] Entering ScDeleteResourceRecursive [07:01:48] Opening resource named 'Exchange HTTP Virtual Server Instance 100 (EVS01)' [07:01:48] Deleting resource 'Exchange HTTP Virtual Server Instance 100 (EVS01)' [07:01:48] Leaving ScDeleteResourceRecursive [07:01:48] Leaving ScDeleteResourceRecursive [07:01:48] Leaving ScDeleteExchangeClusterResources

Page 287: Exchange Server Admin Troubleshooting

86 Module 5: Clustering

[07:01:48] Looking for server object... [07:01:48] Found server object; looking for serial number... [07:01:48] Serial number is "Version 6.5 (Build 6944.4)" [07:01:48] Previous build is 6944, so s_fUpgradingFromPT is FALSE [07:01:48] Entering CAtomRoutingEngine::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Leaving CAtomRoutingEngine::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Entering CAtomIMAP4::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Leaving CAtomIMAP4::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Entering CAtomPOP3::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Leaving CAtomPOP3::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Entering CAtomSMTP::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Leaving CAtomSMTP::ScInitializeExchangeAtomWithCtxInfo [07:01:48] Entering CAtomOMASync::ScRemoveDSObjects [07:01:48] Removing Active Directory objects for Exchange ActiveSync [07:01:48] Clearing HAS_WIRELESS_SYNC bit for HTTP server object [07:01:48] Entering ScSetFlagOnServerHeuristics [07:01:48] Leaving ScSetFlagOnServerHeuristics [07:01:48] Creating Ldob from "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP" [07:01:48] Checking all 5 vroots for a sync vroot [07:01:48] Checking vroot 1 ("Public") [07:01:48] Heuristics on this vroot: 0x00020000 [07:01:48] NOT deleting non-sync vroot [07:01:48] Checking vroot 2 ("Exchange") [07:01:48] Heuristics on this vroot: 0x00020000 [07:01:48] NOT deleting non-sync vroot [07:01:48] Checking vroot 3 ("Exadmin") [07:01:48] Heuristics on this vroot: 0x00020000 [07:01:48] NOT deleting non-sync vroot [07:01:48] Checking vroot 4 ("OMA") [07:01:48] Heuristics on this vroot: 0x00420000 [07:01:48] NOT deleting non-sync vroot [07:01:48] Checking vroot 5 ("Microsoft-Server-ActiveSync") [07:01:48] Heuristics on this vroot: 0x00820000 [07:01:48] This is a sync vroot. Deleting it [07:01:48] Entering CDirectoryManager::ScDeleteDSObject [07:01:48] Leaving CDirectoryManager::ScDeleteDSObject [07:01:48] Finished checking vroots [07:01:48] Leaving CAtomOMASync::ScRemoveDSObjects [07:01:48] Entering CAtomBrowse::ScRemoveDSObjects [07:01:48] Clearing HAS_WIRELESS_BROWSE bit for HTTP server object [07:01:48] Entering ScSetFlagOnServerHeuristics [07:01:48] Leaving ScSetFlagOnServerHeuristics [07:01:48] Creating Ldob from "/dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=Protocols/cn=HTTP" [07:01:48] Checking all 4 vroots for a browse vroot [07:01:48] Checking vroot 1 ("Public") [07:01:48] Heuristics on this vroot: 0x00020000 [07:01:48] NOT deleting non-browse vroot [07:01:48] Checking vroot 2 ("Exchange") [07:01:48] Heuristics on this vroot: 0x00020000 [07:01:48] NOT deleting non-browse vroot

Page 288: Exchange Server Admin Troubleshooting

Module 5: Clustering 87

[07:01:48] Checking vroot 3 ("Exadmin") [07:01:48] Heuristics on this vroot: 0x00020000 [07:01:48] NOT deleting non-browse vroot [07:01:48] Checking vroot 4 ("OMA") [07:01:48] Heuristics on this vroot: 0x00420000 [07:01:48] This is a browse vroot. Deleting it [07:01:48] Entering CDirectoryManager::ScDeleteDSObject [07:01:48] Leaving CDirectoryManager::ScDeleteDSObject [07:01:48] Finished checking vroots [07:01:48] Leaving CAtomBrowse::ScRemoveDSObjects [07:01:48] Entering CAtomSMTP::ScRemoveDSObjects [07:01:48] Removing Active Directory objects for SMTP Service [07:01:48] Entering CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CAtomSMTP::ScRemoveDSObjects [07:01:49] Entering CAtomDAV::ScRemoveDSObjects [07:01:49] Removing Active Directory objects for Base DAV protocol [07:01:49] Entering CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CAtomDAV::ScRemoveDSObjects [07:01:49] Entering CAtomPOP3::ScRemoveDSObjects [07:01:49] Removing Active Directory objects for POP3 Service [07:01:49] Entering CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CAtomPOP3::ScRemoveDSObjects [07:01:49] Entering CAtomIMAP4::ScRemoveDSObjects [07:01:49] Removing Active Directory objects for IMAP4 Service [07:01:49] Entering CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CDirectoryManager::ScDeleteDSObject [07:01:49] Leaving CAtomIMAP4::ScRemoveDSObjects[07:01:49] Entering CAtomRoutingEngine::ScRemoveDSObjects [07:01:49] Removing Active Directory objects for Routing Engine [07:01:49] Entering ScAddOrRemoveServerFromRoutingGroup [07:01:49] Leaving ScAddOrRemoveServerFromRoutingGroup [07:01:49] Leaving CAtomRoutingEngine::ScRemoveDSObjects [07:01:49] Entering CAtomMDB::ScRemoveDSObjects [07:01:49] Removing Active Directory objects for Information Store Service [07:01:49] Reassigning site folder server [07:01:49] Entering CAtomMDB::ScPickNewSiteFolderServer [07:01:49] Leaving CAtomMDB::ScPickNewSiteFolderServer [07:01:49] Entering CAtomMDB::ScRemoveDeleteSearchApplication [07:01:49] Creating Microsoft Search application [07:01:49] Creating search admin component [07:01:49] Getting the applications interface [07:01:49] Removing the search application [07:01:50] Leaving CAtomMDB::ScRemoveDeleteSearchApplication [07:01:50] Removing public folder stores from public folder replica lists [07:01:50] Entering CAtomMDB::ScCleanupPublicFolderStores [07:01:50] Getting all the public stores [07:01:50] Calling ScCanDelete for /dc=org/dc=exchange/cn=Configuration/cn=Services/cn=Microsoft Exchange/cn=EXCHANGE/cn=Administrative Groups/cn=First Administrative Group/cn=Servers/cn=EVS01/cn=InformationStore/cn=First Storage Group/cn=Public Folder Store (EVS01) [07:01:50] Deleting the public store [07:01:50] Leaving CAtomMDB::ScCleanupPublicFolderStores

Page 289: Exchange Server Admin Troubleshooting

88 Module 5: Clustering

[07:01:50] Entering CAtomMDB::ScCleanupMailboxStores [07:01:50] Getting all the private stores [07:01:50] Deleting the gateway object [07:01:50] Deleting the system mailbox [07:01:50] Leaving CAtomMDB::ScCleanupMailboxStores [07:01:50] Removing information store container [07:01:50] Entering CDirectoryManager::ScDeleteDSObject [07:01:50] Leaving CDirectoryManager::ScDeleteDSObject [07:01:50] Leaving CAtomMDB::ScRemoveDSObjects [07:01:50] Entering CAtomMTA::ScRemoveDSObjects [07:01:50] Removing Active Directory objects for MTA Service [07:01:50] This is a cluster vm, go get the ResponsibleMTAServer to see if the MTA lives under this VM. [07:01:50] ResponsibleMTAServer says the MTA lives under this VM, find out if there are other vms in the cluster [07:01:50] There are NO other vms in the cluster. No work to do. [07:01:50] Leaving CAtomMTA::ScRemoveDSObjects [07:01:50] Entering CAtomSystemAttendant::ScRemoveDSObjects [07:01:50] Removing Active Directory objects for System Attendant service [07:01:50] Removing system attendant object [07:01:50] Entering CDirectoryManager::ScDeleteDSObject [07:01:50] Leaving CDirectoryManager::ScDeleteDSObject [07:01:50] Entering ScSetFlagOnServerHeuristics [07:01:50] Leaving ScSetFlagOnServerHeuristics [07:01:50] Leaving CAtomSystemAttendant::ScRemoveDSObjects [07:01:50] Entering CAtomServer::ScRemoveDSObjects [07:01:50] Removing Active Directory objects for Microsoft Exchange Server-Level Objects [07:01:50] Entering CDirectoryManager::ScDeleteDSObject [07:01:50] Leaving CDirectoryManager::ScDeleteDSObject [07:01:50] Entering ScSetFlagOnServerHeuristics [07:01:50] Leaving ScSetFlagOnServerHeuristics [07:01:50] Entering ScConditionallyDeleteServerObject [07:01:50] Leaving ScConditionallyDeleteServerObject [07:01:50] Leaving CAtomServer::ScRemoveDSObjects [07:01:50] Entering ScDeleteExchangeClusterResources [07:01:50] Entering ScBindToEVS [07:01:50] Binding to cluster'node02.exchange.org' [07:01:50] Binding to EVS 'EVS01' [07:01:50] Leaving ScBindToEVS [07:01:50] Entering ScUpdatePropertiesOnGroup [07:01:50] Processing group 'EVS01' [07:01:50] Clearing property 'MSExchange_VirtualServerName' [07:01:50] Leaving ScUpdatePropertiesOnGroup [07:01:50] Entering ScFindSystemAttendantResource [07:01:50] Searching for System Attendant resource [07:01:50] Found System Attendant resource named 'EVS01 SA' [07:01:50] Leaving ScFindSystemAttendantResource [07:01:50] Entering ScDeleteResourceRecursive [07:01:50] Opening resource named 'EVS01 SA' [07:01:50] Deleting resource 'EVS01 SA' [07:01:51] Leaving ScDeleteResourceRecursive [07:01:51] Leaving ScDeleteExchangeClusterResources [07:01:51] Leaving ScSetupExchangeVirtualServer

Page 290: Exchange Server Admin Troubleshooting

Module 5: Clustering 89

Lab 5.1 : Clustering

Exercise 1a: Install the binary files 1. Log in to NODE01 as Administrator (password)

2. Start Cluster Administrator (Click on Start, Administrative Tools, Cluster Administrator)

3. Make sure that all cluster groups are hosted on NODE02. If a cluster group is hosted on NODE01 then right click on the group and choose "Move Group"

4. Ensure that the MSDTC resource exists. If it does not, proceed to Exercise 1b.

5. Mount the Exchange 2003 ISO image and start the install process.

6. Click Next to the splash screen

7. Choose I agree to the license agreement and then click Next

8. Leave the install type as Typical and click Next (Note: If there was no MSDTC resource, you would not be able to proceed)

9. The install will now begin

10. When the install is complete restart NODE01.

11. Run through the same process on NODE02.

12. Proceed to Exercise 2.

Exercise 1b: Create MSDTC resource 1. Start Cluster Administrator and connect to the cluster called "CLUSTER"

2. Locate the Cluster Group. Right click on the group and choose "New - Resource".

3. The New Resource wizard now appears.

4. In the Name filed type "MSDTC". Set the Resource Type to Distributed Transaction Coordinator and make sure that the Cluster Group is specified as the group. Click Next.

5. Set both Nodes as possible owners and then click Next.

6. Add a dependency for the Disk Q: and the Cluster Name resources. Click Finish

7. You have now successfully created the MSDTC resource. Click OK.

8. Right click on the new MSDTC resource and choose Bring Online.

9. The resource should come online successfully.

Page 291: Exchange Server Admin Troubleshooting

90 Module 5: Clustering

Exercise 2: Create Mount Point Drive 1. Start Cluster Administrator on the node which owns the cluster group called EVS01. (Note: EVS is a common abbreviation for Exchange Virtual Server)

2. Open an Explorer window and locate the drive S:

3. Create a folder called S:\Mount.

4. Open Disk Manager on the node

5. Notice that there is one disk that we have not yet allocated. Right-click on that disk (Disk4) and click New Partition. (If you are prompted to write a signature, proceed.)

6. Click Next to the splash screen.

7. Choose primary partition and click Next.

8. Leave it at the default size and click Next.

9. Choose Mount in the following empty NTFS folder. Click browse and locate the folder S:\Mount.Click OK and then Next.

10. Click Next to format the volume.

11. Switch back to Cluster Administrator and locate the cluster group EVS01.

12. Right-click and choose New Resource.

13. Give it the name "Mount Point S:\Mount" and then choose Physical Disk and click Next.

14. Select both Nodes as possible owners and click Next.

15. Set a dependency on Disk S:\ and click Next.

16. Make sure you choose Disk4Partition1. Click Finish and then OK.

17. Bring the resource online.

Exercise 3: Create the Exchange Virtual Server 1. Start Cluster Administrator and locate the cluster group EVS01.

2. Right-click on the group EVS01 and choose New Resource.

3. Give it the name "Exchange IP Address" and choose the resource type IP Address. Click Next.

4. Set both Nodes as possible owners. Click Next.

5. We do not need to set any dependencies. Click Next.

6. Set the address 10.10.1.130 and a subnet mask of 255.255.255.0. Set the network to LAN. Click Finish & then OK.

7. Bring the resource online.

8. Right click on the EVS01 group again and choose New Resource.

9. Give the resource the name "Exchange Network Name" and choose the Network Name resource type. Click Next.

Page 292: Exchange Server Admin Troubleshooting

Module 5: Clustering 91

10. Set both Nodes as possible owners. Click Next.

11. Set a dependency on the IP address. Click Next.

12. Set the name as "EVS01". Click Finish & then OK.

13. Bring the resource online.

14. Right click on the group EVS01 and choose New Resource.

15. Give it the name "Exchange System Attendant". Set the resource type to Microsoft Exchange System Attendant and click Next.

16. Set both Nodes as possible owners. Click Next.

17. Set a dependency on Disk R:\, Disk S:\, Mount Point S:\ and Exchange Network Name. Click Next.

18. Pick the default for admin group/routing group. Click Next.

19. Set the path to S:\EXCHSRVR. Click Next.

20. Click Finish.

21. Click OK when the creation has completed.

22. Bring the Exchange resources online.

23. Note that you can change the location of the transaction logs or MTADATA files to S:\mount (Disk 4), thereby alleviating space from the true drive S: (Disk 3).

Page 293: Exchange Server Admin Troubleshooting

Contents

Overview............................................................. 1 Lesson 1: Deployment Tools .............................. 2 Lesson 2: ADC Tools........................................ 39 Lab 6.1 Exchange 2003 ADC replication featuring Deployment and ADC Tools ............. 71 Appendix A: Sample log files .......................... 86 Appendix B: Learning Measure Answers ....... 107 Acknowledgments........................................... 107

Module 6: Deployment Tools and ADC Tools

Page 294: Exchange Server Admin Troubleshooting

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 295: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 1

Overview

For this module, we will discuss the new deployment features available with Exchange Server 2003 and differentiate the deployment process from Exchange 2000.

Topic 1 Deployment Tools

Topic 2 Active Directory Connector (ADC) Tools

Topic 3 Appendix

Prerequisites 1) Experience with installing Exchange 2000 into Exchange 5.5 sites 2) Prior usage of Virtual PC or Virtual Server

Page 296: Exchange Server Admin Troubleshooting

2 Module 6: Deployment Tools and ADC Tools

Lesson 1: Deployment Tools

Basic Overview

History: Customers that installed Exchange 2000 experienced a paradigm shift in the complexity of the underlying operating system. With Windows 2000 introducing several new concepts, installers were burdened with learning the differences in how Active Directory uses Domain Name System (DNS), Lightweight Directory Access Protocol (LDAP), and a variety of new server roles for establishing suitable infrastructures for Exchange 2000. Microsoft Product Support Services learned that these infrastructures failed too often, or were never configured correctly at their inception. Although many of the support calls were caused by platform-level mishaps (such as improper DNS configurations, Active Directory permission-misconfigurations, and lack of connectivity to operations masters), more daunting challenges existed for migrations from Exchange 5.5. These Exchange 5.5 migration challenges were often

discouraging customers from deploying Exchange 2000. (For example, a customer might find it extremely difficult to roll back after a failed disaster recovery scenario following a failed in-place upgrade.)

completely ignored or skipped by customers. For example, NTDSNoMatch is supposed to be written on Exchange 5.5 objects, yet customers didn’t know of the existence of NTDSNoMatch due to delayed documentation when Exchange 2000’s retail version shipped. Additionally, many customers skipped configuration of Connection Agreements due to their complexity, or even worse…

improperly configured through guesswork. (Installers who were new to Exchange 2000 were accustomed to “Install first, configure later” methodologies, yet Exchange 2000 deviated from other server applications

Page 297: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 3

by requiring tremendous configuration changes to their current topology before installing. This approach occurred most often with small to medium-sized companies who lacked the time or manpower to research the deployment process, and would take a simple approach to running the setup process without reviewing the appropriate pre-setup steps.)

By not meeting those challenges, what resulted were problems ranging from unintended standalone orgs incompatible with Exchange 5.5 orgs, to creating thousands of duplicate Active Directory objects that were improperly replicated due to no NTDSNoMatching, to disaster recovery on Exchange 5.5 directories where customers unintentionally created (and then mistakenly deleted) mismatched accounts, to non-functional transports. These large percentages of failed or blocked deployments rapidly cost Microsoft Product Support Services a high price because there was no easy corrective path. Instead, Microsoft Product Support Services would occasionally clean up corrupted Exchange 5.5 and Active Directories, and then re-deploy Exchange 2000 for customers. Even today, where installers are armed with a great deal of knowledge that gradually became publicly available, they are still prone to encountering problems, simply because of the sheer quantity and complexity of deployment steps. Even administrators who were simply migrating, who may not be concerned with Exchange 5.5/2000 interoperability, are required to fumble through the various coexistence measures because migrations require temporary interoperability with Exchange 5.5. So a process was needed to improve customer education and ensure the structural integrity of Exchange 5.5 and Active Directory topologies, leading to improving deployment success rates.

The solution: Efforts to prevent Exchange 2000 deployment mistakes of the past have culminated into the creation of the Exchange 2003 deployment tools. A multipurpose effort, the deployment tools not only avoids the huge gap in customer education when Exchange 2003 ships; it also proactively scans the Active Directory and Exchange 5.5 infrastructures for possible problems that may prevent successful Exchange 2003 deployments. The customer education effort is achieved through a comprehensive help file/installation guide, which takes into consideration four major deployment scenarios and provides prescriptive deployment steps for each. A picture of the help file is shown in figure 1.1:

Page 298: Exchange Server Admin Troubleshooting

4 Module 6: Deployment Tools and ADC Tools

Figure 1.1: An example of the deployment tools step-by-step deployment guide, in the form of a compiled HTML help file. (pre-release version)

Although the user-education portion may appear informational at first glance, there are ActiveX controls embedded within each HTML page that, when clicked, will spawn scripts to proactively check for problems on the local system, within Exchange 5.5 directory, within Active Directory, or all of the above. Technically, the scripts call upon the deployment tools, but the collection of tools plus help file is most commonly-referred as the “Deployment Tools.”

Page 299: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 5

Tool Execution

There are three ways to call upon the Deployment Tools:

Method 1 – From the help file (most common): The Deployment Tools’ step-by-step guide is a compiled HTML help file. In other words, it is a collection of Web pages that are combined into a single file with a .CHM extension. For customers to execute the help file, they must launch the HTML application (exdeploy.hta), which then renders the Exdeploy.chm contents within its frame. The CHM/HTA file may be navigated just like a collection of Web pages within a Web site. (See the “Process flow” heading for information about HTML applications). The CHM file may be decompiled into separate HTML pages using Visual Studio, though that is outside the scope of this discussion. Although you could launch the exdeploy.CHM file, and click on “Home” to get to the starting point of the deployment steps, it is not recommended because of Internet Explorer browsing problems. Thus, it is recommended to launch exdeploy.HTA instead.

While browsing through a page that contains a script, users launch each Deployment Tool by entering-in servername information onto the Web page. When they hit “run <toolname> now”, a script takes the user input as parameters to execute the tool(s) in a separate command window, shown by the bottom portion of figure 1.2:

Page 300: Exchange Server Admin Troubleshooting

6 Module 6: Deployment Tools and ADC Tools

Figure 1.2: Tool execution through the help file spawns the exdeploy.exe command line tool with a hidden switch. Under the hood, the CHM is running “exdeploy.exe /s:racecar /gc:palindromes /t:orgprepcheck”

The command window disappears when the exdeploy tool has finished execution. However, during execution, the tools will log success and failure information into exdeploy.log file, typically located in c:\exdeploy logs. Log files are appended-to, not overwritten, when tools are run more than once. Although exdeploy.chm contains links to launch the tools, the tools themselves may not be launched without the existence of binaries (DLLs and EXEs) within the same directory as the CHM file. The deployment tools help file and binaries are located on the Exchange 2003 CD, underneath the \support\exdeploy directory.

Method 2 – From the command prompt: The error-checking tools may also be launched from the command line by running exdeploy.exe. Exdeploy.exe is an executable that can launch various deployment tools depending upon the switches used. In fact, all of the deployment tools may be launched using exdeploy.exe, without requiring the CHM file. However, none of the tools may be launched from the CHM/HTA file if the CHM/HTA exists in a directory without exdeploy.exe supporting it.

Using Method 2 to manually execute a deployment tool should only be used when troubleshooting, or when someone is already familiar with the ordering of the help file (since some tools will fail unless you have performed certain steps only mentioned in the CHM file). Here is an example of running a deployment tool from the command prompt:

D:\SUPPORT\EXDEPLOY>exdeploy /s:55server /gc:gc01 /t:adcusercheck Results of these tools will be logged to 'exdeploy.log'. Exchange Deployment Tools documentation provides information on how to solve encountered issues. Calling ADCUserCheck... ADCUserCheck completed successfully.

Page 301: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 7

Like Method 1, tools will still create/append-to logfiles located in the logging path (c:\exdeploy logs by default). Some tools will even write their own logfile, named after the tool itself. Often, installers will attempt to run the tools comprehensively (using the /c switch), so that all of the tools will be run with one command line execution. Comprehensively running the tools is not useful for the customer before setup because many of the deployment tools tests will fail when it checks for existence of Active Directory objects that only exist post-setup. However, it is useful to Microsoft Product Support Services for troubleshooting an installation that has already completed, since a low level of information gathering (i.e. list of server names, sites, list of unreplicated users) is readily available in c:\exdeploy logs.

Deployment tools may also be launched in tool groups. For instance, when you run “/t:DSScopescan” you actually launch five deployment tools contained within that group: DSConfigSum, DSObjectSum, UserCount, VerCheck, and OrgNameCheck. Tool groupings are documented within exdeploy.exe usage (simply by typing exdeploy /?) and also within the exdeploy.chm reference topics.

Method 3 – from the Exchange 2003 setup wizard: The user has no control here, as setup.exe will automatically launch a few of the deployment tools to perform some basic pre-requisite checking before continuing setup. In Exchange 2000, there were fewer checks prior to the file-copy/register phase, so when setup proceeded to the latter stages, it would often fail catastrophically. The Exchange 2003 setup program is now improved with additional pre-requisite checks, some of which look for completion of certain deployment tools before allowing itself to proceed to file-copy/registry modification phases of setup. Since setup.exe employs the use of the same tools as exdeploy.exe, we will discuss the glue DLL that is utilized by both.

Figure 1.3: Exchange 2003 Glue DLL has multiple interfaces, since it is called by exdeploy and Exchange 2003 setup.

The Exchange 2000/2003 setup program runs through prerequisite checks upon launch, and if any prerequisite checks fail, their associated errors (possible

Page 302: Exchange Server Admin Troubleshooting

8 Module 6: Deployment Tools and ADC Tools

reasons/recommended actions) are displayed as a popup on the setup wizard’s component selection screen.

[10:44:03] ********** Beginning Exchange Deployment Tools ********** [10:44:03] Starting Exchange 6851 Deployment Tools on Windows 5.0.2195 at 10:44:03 01/13/2003 [10:44:03] Entering HrDirPreReq_Initialize [10:44:03] Init called with Domain Controller tilab-dc and Exchange 5.5 server root55. Setup's language ID is 0 [10:44:03] Entering HrRegisterAXDLL [10:44:03] Leaving HrRegisterAXDLL [10:44:03] Entering HrRegisterAXDLL [10:44:03] Leaving HrRegisterAXDLL [10:44:03] Leaving HrDirPreReq_Initialize [10:44:21] Entering HrDirPreReq_ConfigInit [10:44:55] Leaving HrDirPreReq_ConfigInit [10:44:55] Entering HrDirPreReq_ObjectInit [10:45:46] Leaving HrDirPreReq_ObjectInit [10:45:46] Entering HrDirPreReq_UserInit [10:46:20] Leaving HrDirPreReq_UserInit [10:46:20] Entering HrDSConfigSum [10:46:21] Leaving HrDSConfigSum [10:46:21] Entering HrDSObjectSum [10:46:21] Leaving HrDSObjectSum [10:46:21] Entering HrUserCount [10:46:21] Leaving HrUserCount [10:46:21] Entering HrVerCheck [10:46:21] VerCheck verifies if your Exchange 5.5 servers can be upgraded to Exchange 2000. Details are logged in vercheck.log. [10:46:21] Leaving HrVerCheck [10:46:21] Entering HrRunNetdiag [10:46:21] Entering HrGetDSILog [10:46:21] Leaving HrGetDSILog [10:46:36] Entering HrMapFileName [10:46:36] Entering HrMapFileHandle [10:46:36] Leaving HrMapFileHandle [10:46:36] Leaving HrMapFileName [10:46:36] Entering HrFindPrintErrorMessage [10:46:36] Warning: Possible error string '. . . : Failed' detected in netdiag output. [10:46:36] Leaving HrFindPrintErrorMessage [10:46:36] HrRunNetdiag (f:\df6851\dsa\src\deploy\dsintegchk\netdiag.cpp:888). Error code 0X80040001(1). [10:46:36] Leaving HrRunNetdiag [10:46:36] Entering HrOrgNameCheck [10:46:37] Leaving HrOrgNameCheck [10:46:37] Entering HrDirPreReq_Terminate [10:46:37] Leaving HrDirPreReq_Terminate

The exdeploy-progress.log can be opened using logparser.exe. However, its filters for logging levels do not work, so leave the setting on maximum (log level 3). The only benefit to opening in logparser is to use logparser’s ability to

Page 303: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 9

dissect multiple runs of the exdeploy-progress.log so that you can view each run by itself. Another minor thing to note here is that a lot of the same entries you find in exdeploy-progress.log will also be logged into the setup wizard’s progress.log file. Search for HrDirPreReq anytime setup is joining an Exchange 5.5 site, and you’ll get to the deployment tools section of the Exchange Server Setup Progress.log.

On the right-hand side of figure 1.3, the glue DLL will call into the actual tools themselves. The tools are EXEs, DLLs, or even scripts. If the individual tool is a script or separate EXE (such as policytest.exe), then the glue DLL makes a call to CreateProcess.

Page 304: Exchange Server Admin Troubleshooting

10 Module 6: Deployment Tools and ADC Tools

Markers:

Before discussing the process flow, consider that several phases of the deployment tools will create markers in Active Directory. These “completion” markers are intended to ensure that customers use the deployment tools and ADC Tools. Without them in place, setup will block customers from installing the first Exchange 2003 server any organization containing Exchange 5.5 or Site Replication services. Without Exchange 2003 setup logic to detect these markers, installers would skip the proper deployment steps and tools, thereby encountering the same deployment problems that existed with Exchange 2000.

Also, one of the main differences from Exchange 2000 is that in Exchange 2003, installers will no longer be able to launch the setup wizard from setup.exe at the root of the CD without being forced into deployment tools. This single entry-point initiative for setup was deemed necessary for several reasons: 1) Development and Product Support Services want customers to review the exdeploy.chm to prevent problems, 2) the deployment tools are not very discoverable since they are in a completely separate directory from \setup\i386\setup.exe, and 3) setup will not be able to continue unless a certain condition (ADCUserCheck marker) is satisfied by the deployment tools.

The backdoor executable (\CD ROOT\setup\i386\setup.exe) may still run the setup wizard, but this path is less discoverable for unexperienced

installers.

ADCUserCheck, along with other markers, are illustrated in figure 1.4 below:

Note

Page 305: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 11

Figure 1.4: The deployment tools completion markers, viewed through ADSI Edit

As seen in the above illustration, markers are attribute values stamped in the “description” attribute of the Microsoft Exchange container in Active Directory. Each value contains three fields:

The marker name - named after the tool that stamped it. The timestamp of the marker – indicates the time (not in GMT) that the tool

was last executed. A status flag – describing if the tool’s result was a success or failure.

The marker of biggest concern is the ADCUserCheck marker, which is stamped when the user clicks the button to run Step2’s Data Collection or to verify step 4 in the ADC tools (discussed in Lesson #2). ADCUserCheck is the most important marker, since its absence will prevent setup from proceeding beyond its initialization phase. Also, notice the timestamp: 2003003282359. If that value is more than two weeks old, the installer will be warned about the need to rerun the tool. The purpose of the timestamp is to prevent the tool’s result from becoming stale, since customer environments may have changed drastically over weeks or months, and it is highly likely they have more unreplicated objects from the time they originally passed ADCUserCheck. Specifically, the purpose of rerunning the tool is that after a time threshold, customers may need to rereplicate or configure new Connection Agreements.

Page 306: Exchange Server Admin Troubleshooting

12 Module 6: Deployment Tools and ADC Tools

As an installer and for the purposes of saving time, you could manually insert the ADCUserCheck marker using ADSIEdit and skip all of the deployment tools. However, normal customers should not utilize this shortcut since you want them to utilize deptools/adctools.

Troubleshooting Tip

Page 307: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 13

Process Flow

The deployment process begins when customers insert their Exchange 2003 CD or run setup.exe from the root of the CD. Either action launches the intro/splash screen, which in previous versions of Exchange provided a direct link to setup.exe within the \setup\i386 folder. In Exchange 2003, the splash screen no longer allows Exchange setup to be directly launched. Instead, installers may only choose the deployment tools link.

Page 308: Exchange Server Admin Troubleshooting

14 Module 6: Deployment Tools and ADC Tools

Figure 1.5: The introductory splash screen, no longer allows on-demand Exchange installations.

The splash screen link to the deployment tools actually points to \support\exdeploy\exdeploy.hta, which is a simple HTML application that calls upon exdeploy.chm. Framed within the HTA, the CHM file’s content and ordering is preserved while at the same time it bypasses script and security errors that would otherwise have been apparent if the CHM file was used to launch scripts.

The CHM file’s main menu contains a resource link to download the latest version of the help file, as well as four links to the various phases of deployment.

These four menu options are shown on the left-hand side of figure 1.6 and the installer should begin with “Deploy the first Exchange 2003 server.” Clicking on this link will produce a second-tier menu for the most significant decision-making step where customers can identify in which of these four scenarios they reside.

Page 309: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 15

Deployment Scenarios

1. Coexistence with Exchange 5.5 – This option is a necessity for intra-organizational migrations.

2. Coexistence with Mixed Mode Exchange 2000 and Exchange 5.5 –This scenario covers installing Exchange 2003 as another member of a mixed-mode organization. This option also applies when there are Site Replication Servers running in the organization, even if there are no more Exchange 5.5 servers.

3. Upgrade from Exchange 2000 Native mode – This scenario’s name not only implies in-place upgrading an Exchange 2000 server to Exchange 2003; it also covers joining an Exchange 2003 server as another member of a pure Exchange 2000 organization with no running Site Replication Servers.

4. New Exchange 2003 – This scenario is the simplest of all, as no preparatory work is necessary for any existing Exchange servers.

Page 310: Exchange Server Admin Troubleshooting

16 Module 6: Deployment Tools and ADC Tools

Figure 1.6: Process flow for all of the steps covered by exdeploy.chm

Figure 1.6 illustrates the process flow, which contains scenarios identified at the top of the screen, enclosed by borders. The most full-featured scenario for installing the first Exchange 2003 server is “Coexistence with Exchange 5.5.” In the coexistence scenario, deptools examines the existing Exchange 5.5 and Active Directory infrastructure for Exchange 2003 suitability. Note that inter-organizational (cross-forest) migrations or deployments of multiple Exchange organizations are too advanced and are not discussed by exdeploy.chm. Cross-forest deployments is discussed in another training module.

Page 311: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 17

DSScopeScan

Since the “Coexistence with Exchange 5.5” scenario runs through the entire gamut of deployment tools, this discussion will concentrate on the Exchange 5.5/2003 deployment. Beginning with the DSScopeScan tool group, which runs a set of tools intended to provide administrators a quick estimate of the size of the organization to help with gauging the deployment project’s length, one may easily foresee possible deployment blockers. When the deployment administrator/consultant runs the DSScopescan tool group, the following tools execute:

DSConfigSum: Outputs server name, Windows version, and Exchange version. Notes whether the server contains a public folder store. Notes whether Key Management Service is installed. Notes whether Internet Mail Service is installed. Notes whether the server is a directory bridgehead server. Outputs any inbound, outbound, or 2-way connectors. This tool is probably more beneficial to support engineers in data gathering, as it outputs site names and Exchange server versions within those sites. This additional output is logged to DSConfigSum.log.

DSObjectSum: Notes number of public folders. Notes number of distribution lists. Notes number of distribution lists with hidden membership. Notes number of contact objects. There is no additional logfile; all output for this tool is directed to exdeploy.log. This tool is useful in determining how many groups may be affected with disappearing membership: More information is in KB article 321205.

UserCount: Notes number of users in each site. Notes total number of users in the Exchange 5.5 directory. If this number is ever zero, it generally indicates a permissions problem (you might be running deptools with the wrong credentials to the 5.5 directory) or an LDAP protocol configuration problem (search buffer might be set too low per KB article 320482). Extended output is fed to usercount.log.

VerCheck: Determines if any Exchange 5.5 server versions are not compatible with the Active Directory connector. (At least one must be Exchange 5.5 SP3). Outputs to vercheck.log.

Page 312: Exchange Server Admin Troubleshooting

18 Module 6: Deployment Tools and ADC Tools

Orgreport: Determines if an existing object, whose objectclass is msExchOrganizationContainer, exists underneath cn=Microsoft Exchange,cn=services,cn=configuration,dc=<dn of forest root> If one is found, the tool does not qualify it as an error. However, it will write the displayname of the object to exdeploy.log if it was not created by Exchange 2003 forestprep. The existence of an org object means that an Exchange 2000 installation attempt, either through a forestprep or typical setup, was performed in the past. Additionally, this signifies the possible existence of a rogue Exchange 2000 server object in Active Directory. If this is the case, rollback using the removeorg switch (Q312878).

GCVerCheck: Checks local and adjacent Windows sites for a Windows global catalog that is SP3 or greater. If none is found, then setup will not proceed. Although Exchange 2003 setup has a prerequisite check for this situation, it is convenient to scan for this prior to setup, so that administrators can plan upgrades of their domain controllers accordingly.

OrgNameCheck: Determines if the Exchange 5.5 organization or site names exceed 64 characters or contain any of the following (excluding the surrounding braces). { , = + < > # \ " ~ ! @ # $ % ^ & * ( ) _ + = { } [ ] | \ : ; " ' < , > . ? / }. Additionally, this tool determines if an Exchange 5.5 SMTP address generator (from site addressing) contains the same invalid characters that do not follow RFC 821. Exchange 2003 setup will run this tool also as a part of setup, and will not proceed unless the Exchange 5.5 site addressing is corrected, as invalid characters would cause problems with recipient policies if replicated to Active Directory. OrgNameCheck logs additional details into orgnamecheck.log.

Troubleshooting a hanging tool: Should a tool hang, you can control-break and run exdeploy /t: <name_of_hanging_tool> from the command prompt. Each execution of exdeploy will append to the exdeploy.log and exdeploy-progress.log files. For a list of tool names, run exdeploy.exe without switches.

Troubleshooting a permissions issue: You may encounter an error message warning that you may not be able to view hidden objects in the Exchange 5.5 directory. This is most often caused by lack of a trust and organization/site/config-level permissions to Exchange 5.5 if it exists in a Windows NT 4 domain.

Debugging anything else in exdeploy.exe: If a tool is reporting a problem which you know to be false (for example, exdeploy.log says that the Exchange 5.5 server cannot be connected to, even though you can use LDP.exe to bind to its LDAP port), you may enable debug logging by typing “set DebugWalksDLL=1” at the command prompt. Output would look like the following:

#*** Exdeploy began: 06/12/2003 15:39:29 ***# + Exchange 5.5 Server: rescon01:389 + Global Catalog Server: resdc01 + Tools run: DSScopeScan. TestEXConnect - Open rescon01:389 failed error=-81 - Error: Could not connect to the Exchange 5.5 server 'rescon01:389'. Tools that require an Exchange 5.5 server will not run. TestNTConnect - Open resdc01 succeeded

Page 313: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 19

In the output above, the entries “TestEXConnect” and “TestNTConnect” are the result of the additional debug logging. Enabling this environment variable also causes exdeploy.log to be produced with debug output whenever Exchange 2003 setup.exe calls upon the deployment tools glue DLL.

Page 314: Exchange Server Admin Troubleshooting

20 Module 6: Deployment Tools and ADC Tools

DCDiag/NetDiag

Following the DSScopeScan tool group, the installer is instructed to download the Windows 2000/2003 support tool, dcdiag, or alternatively, install it from the Windows server CD’s support tools. This utility comes in two operating system-specific versions, and is used to search for Active directory-related problems. This tool checks for a variety of domain controller issues, but most importantly, it checks for any operations master role holders which cannot be contacted.

Troubleshooting DCDiag: If DCDiag fails to run due to a “DsIsMangledDnW error,” check to see if the version of dcdiag.exe is compatible with the operating system. The file version for the Windows 2003 DCDiag is 5.2.xxxx, whereas the Windows 2000 version should be 5.0.xxxx.

If dcdiag reports that, for example, a schema role holder is not available, forestprep will not be able to run. In this instance, one would utilize Q324801 or Q255504 to transfer or seize the role. Forestprep will also have problems if DCDiag reports that a domain controller is not contactable, when in fact it has been removed from the forest (but its metadata remained). In that scenario, one would remove the orphaned domain or domain controller via q230306.

If there are any other errors output by DCDiag, one would run dcdiag with the verbose (/v) switch for more details regarding the possible root cause. Additionally, the /q switch suppresses non-problematic information, so you can get to the meat of the failure quickly.

Netdiag follows DCDiag, and in the same manner, it examines various aspects of the computer’s network configuration to determine if there are any errors.

Netdiag troubleshooting tip: Netdiag with the /q switch will suppress most tests marked with “Pass,” which eases readability for failures. If netdiag encounters a problem, the /fix switch may be used to resolve transient issues. For example, if there is a negative cache issue that would eventually go away in ten minutes, /fix would correct the issue sooner. However, hard configuration problems – such as a misconfigured DNS server setting – cannot be fixed.

Page 315: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 21

Running these tools would be useless if the customer does not review their output, so following the execution of netdiag, dcdiag, and the dsscopescan tool group, installers should search through the exdeploy.log, netdiag.log, and dcdiag log files for any errors. These errors, along with their corrective suggestive actions, are generally covered in the help file. However, there may be instances where the help file does not log any errors. So engineers should often ask themselves, “Does this output make sense?” For example, it is a fact that any organization contains least one site and at least one server per site. However, if the site or server count is zero, one may conclude that DSConfigSum did not perform its task properly. In this instance, one would also determine if there are more than 100 sites or 100 servers within a container. (100 is the default number for reporting zero objects because that is the default threshold for Exchange 5.5 directory service to enumerate a container via LDAP.) If there are indeed containers with more than 100 objects (not counting recipient subcontainers), one would reconfigure the Exchange 5.5 LDAP protocol’s search limit above 100.

Forestprep/Domainprep The next steps are for the user to run setup /forestprep and /domainprep. Traditionally, these switches were executed only from the command prompt. The exdeploy.chm now includes an embedded script to launch these modes from the ActiveX® control, provided that the path is populated correctly. Running the help file from a file share or a path that contains a space will most likely output an Internet Explorer popup saying “An invalid server name was entered.” For this reason, the HTA is used to run the CHM file to slightly alter the behavior.

Page 316: Exchange Server Admin Troubleshooting

22 Module 6: Deployment Tools and ADC Tools

OrgPrepCheck

Once these actions are completed, the user is prompted to run the OrgPrepCheck tool group - comprised of the following tools:

OrgCheck: verifies the Exchange extensions to the Active Directory schema, checks the existence and membership of the Exchange Domain Servers group and Exchange Enterprise servers group, and checks that a global catalog server is available in a domain in which DomainPrep has been run. There is no additional logfile.

PolCheck: This exdeploy command simply runs a Createprocess to launch policycheck.exe (inherited from the support directory of the Exchange 2000 CD). PolCheck will determine if the Exchange Enterprise Servers group has been granted the SeSecurityPrivilege (a.k.a. “Manage Auditing and Security Logs”). Effectively, this determines if the domainprep procedure has completed successfully, and ample time has been given for this right to propagate to the domain controllers of the present domain. The extended output of all domain controllers rights will be logged to exdeploy-polcheck.log, and will appear similar to the following:

Page 317: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 23

[17:43:01] #*** Policy Check began: 01/08/2003 17:43:01 ***#

[17:43:03] Entering HrMapFileHandle

[17:43:03] Leaving HrMapFileHandle

[17:43:03] PolicyTest.exe results:

This tool will check every domain controller in the local

domain to see if the "Manage auditing and security logs"

privilege granted to the "Exchange Enterprise Servers"

group by DomainPrep has replicated to that DC. If the

policy change has not yet replicated to all DCs, then

you should avoid making policy changes on any DC that

has not received those changes yet.

You must have Domain Admin rights to run this tool

successfully. If you see an error that says:

!! LsaEnumerateAccountRights returned error 5 !!

then you don't have permission to open the LSA on the

given DC.

===============================================

Local domain is "ti.vm" (TI)

Account is "TI\Exchange Enterprise Servers"

========================

DC = "TINETDC"

In site = "Default-First-Site-Name"

Right found: "SeSecurityPrivilege"

[17:43:03] Entering HrFindPrintErrorMessage

[17:43:03] Leaving HrFindPrintErrorMessage

[17:43:03] PolCheck completed successfully.

[17:43:03] #*** Policy Check finished: ***#

Install Active Directory Connector and Run ADC Tools The next step in the deployment process is for the deployment administrator/consultant to install the Exchange 2003 ADC Service, and then use the ADC tools to prepare for and then create connection agreements. The ADC Tools process is somewhat lengthy, so we will discuss its internals in more detail in Lesson 2. At this point, it is only important to note that when the installer completes the second or last step of the ADC Tools, completion markers are written to Active Directory. These markers, though hidden from the user, are read by the setup engine later in the deployment process to determine whether the proper preparatory steps have been accomplished. If the installer does not complete the second or last steps of the ADC Tools, the completion marker will not exist and setup will block installation into an Exchange 5.5 site.

Page 318: Exchange Server Admin Troubleshooting

24 Module 6: Deployment Tools and ADC Tools

Process Flow

SetupPrep Setupprep is the next tool that the exdeploy.chm runs, although it is not actually a tool group that can be run from exdeploy.exe. When the user runs SetupPrep, the CHM file silently launches exdeploy.exe with the /t: OrgNameCheck, OrgCheck, and PubFoldCheck switches. The first two tools are redundant, and do not serve any additional purpose since they were run previously. The last tool, PubFoldCheck, is significant in that it performs the same operation as the DS/IS Consistency Adjuster against the public information store. This resolves potential issues with “zombie users” causing public folder accessibility and performance problems previously seen in Exchange 2000 deployments into Exchange 5.5 sites. Article 309788 contains information about this problem. The DS/IS Consistency Adjuster options performed by pubfoldcheck are

Remove unknown user accounts from public folder permissions: PubFoldCheck removes users that are no longer valid from public folder permissions.

Inconsistencies more than 1 day: PubFoldercheck performs the above actions for only inconsistencies that are older than one day.

Although the help file may indicate the opposite, the code in PubfoldCheck will not use the DS/IS option to rehome public folders because of the possibility causing replication storms, thereby causing a support call-generator. The helpfile text, though incorrect on the release CD, will be fixed in the web-release version of the Exdeploy package.

Run Server Setup At this stage, the user needs to run setup to install Exchange 2003. Since setup detects that this is the first server being installed into Active Directory, it displays the “installation type” screen. This screen asks whether a new Exchange organization is being created, or if the installer is joining an Exchange 5.5 org. The latter option needs to be selected for the “Coexistence with Exchange 5.5” scenario. In the next screen, the user is prompted to enter the Exchange 5.5 server name. As soon as the Exchange 5.5 server name is passed to the setup engine, setup.exe will pop up a message indicating that it is

Page 319: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 25

testing “prerequisite conditions.” In the background, setup makes several calls into the deployment tools wrapper/glue DLL for pass/fail-type results against the Exchange 5.5 directory and Active Directory. The Exchange server setup progress.log file will contain entries similar to the following:

[00:09:57] Entering HrDirPreReq_Initialize [00:09:57] Init called with Domain Controller Z2 and Exchange 5.5 server z0:389. Setup's language ID is 1033. ActiveX Path is (null) [00:09:57] Entering HrRegisterAXDLL [00:09:58] Leaving HrRegisterAXDLL [00:09:58] Leaving HrDirPreReq_Initialize [00:09:58] Entering HrDirPreReq_Test [00:09:58] Entering HrDirPreReq_ConfigInit [00:10:31] Leaving HrDirPreReq_ConfigInit [00:10:31] Entering HrDirPreReq_UserInit [00:11:04] Leaving HrDirPreReq_UserInit [00:11:04] Entering HrDirPreReq_ObjectInit [00:11:52] Leaving HrDirPreReq_ObjectInit [00:11:52] Entering HrDirPreReq_CheckOrgAndSiteNames [00:11:52] Leaving HrDirPreReq_CheckOrgAndSiteNames [00:11:52] Entering HrDirPreReq_OrgCheck [00:11:52] Leaving HrDirPreReq_OrgCheck [00:11:52] [00:11:52] Entering HrDirPreReq_ADCUserCheck [00:11:52] Leaving HrDirPreReq_ADCUserCheck [00:11:52] HrDirPreReq_Test (f:\df6900\dsa\src\deploy\dsintegchk\dsintegchk.cpp:2003). Error code 0X00000001(1). [00:11:52] Entering HrDirPreReq_ADCObjectCheck [00:11:52] Leaving HrDirPreReq_ADCObjectCheck [00:11:52] HrDirPreReq_Test (f:\df6900\dsa\src\deploy\dsintegchk\dsintegchk.cpp:2062). Error code 0X00000001(1). [00:11:52] Warning: ADCObjectCheck detected that some objects were not replicated from the Exchange 5.5 directory to Active Directory. [00:11:52] Looking for Public Folder DS/IS Consistency Check status marker [00:11:52] Entering HrDirPreReq_GetSiteName [00:12:08] Leaving HrDirPreReq_GetSiteName [00:12:08] Entering HrDirPreReq_CheckMarker [00:12:08] Leaving HrDirPreReq_CheckMarker [00:12:08] PublicFolderCheck-HubSite was last run 120 minutes ago. The value returned was 0. [00:12:08] Finished looking for Public Folder DS/IS Consistency Check status marker [00:12:08] Some tests returned failure or warning, but the prereqs passed overall. Returning 1 warning string(s) [00:12:08] Message 0: Warning: ADC Tools detected that some users were not replicated from the Exchange 5.5 directory to Active Directory. [00:12:08] Leaving HrDirPreReq_Test [00:12:08] Entering HrDirPreReq_Terminate [00:12:08] Leaving HrDirPreReq_Terminate

Page 320: Exchange Server Admin Troubleshooting

26 Module 6: Deployment Tools and ADC Tools

In the above sample output, the ADCUserCheck marker exists, but not all of the objects have been replicated between Exchange 5.5 and Active Directory (determined by the trailing flag within the ADCUserCheck value). So if any completion marker (ADCUserCheck in this case) contains a trailing “1” flag at the end of the value (for example, ADCUserCheck;2003003282359;1) the “1” value signifies that a tool detected an error, so setup will only warn the user about out-of-sync directories, as shown in figure 1.7:

Figure 1.7: Setup shows the result of its calls upon the deployment tools, or the result from a deployment tool’s completion marker in Active Directory.

In the above figure, setup warns the user of unreplicated objects, but it does not prevent setup from continuing. So after being warned, the customer may continue with setup, and subsequently install Exchange 2003 and the Site Replication Service. If the flag was set to 0, there would be no warning.

However, if the ADCUserCheck marker does not exist, setup will block itself from continuing, as shown by the blocking message in figure 1.8:

Figure 1.8: Blocking message if ADC Tools have not been executed.

Simultaneously, the following text would be logged to the exdeploy-progress.log:

[08:30:28] HrDirPreReq_Test is failing. Returning 1 warning or error string(s) [08:30:28] Message 0: Error: ADC Tools were not run in your organization. [08:30:28] Leaving HrDirPreReq_Test [08:30:28] Entering HrDirPreReq_Terminate [08:30:28] Leaving HrDirPreReq_Terminate

Page 321: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 27

Pointing connection agreements to the Site Replication Service

After setup completes, reconfiguring connection agreements is mentioned as the next step in the deployment tools. Although this step is not necessary for small environments, it is good practice to point all of the recipient connection agreements for the upgraded site to the site replication service once it has been created. The benefit is realized when pure Exchange 2000/2003 admin groups are created, whose objects can only be replicated to Site Replication Services (see KB article Q291170).

Page 322: Exchange Server Admin Troubleshooting

28 Module 6: Deployment Tools and ADC Tools

Page 323: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 29

Validation Tools

Validation Tools These are a series of tools used to check on the success of the server installation. The output of these tests should give customers a better idea of what problems exist that are not normally seen by any existing GUI administration tool.

ADCConfigCheck – verifies that the config Connection Agreement (created during setup) properly replicated server objects, connector objects, store objects, etc. from the Exchange 5.5 directory to Active Directory. If there are any inconsistencies, unsynchronized system objects are enumerated in adcconfigcheck.log and exdeploy.log. A sample adcconfigcheck.log output would look like this:

Error: Configuration object 'cn=Internet Mail Connector (Z0),cn=Connections,cn=Configuration,ou=HubSite,o=Microsoft' has not replicated to Active Directory.

Troubleshooting ADCConfigCheck: If any errors are reported at this point, ensure that the config Connection Agreement is replicating. By default, its endpoints are the Site Replication Service and a global catalog, so ensure that the fully-qualified domain names (FQDNs) of these servers haven’t been tampered with on the Connection Agreement properties. Check name resolution, since the new kerberos authentication functionality demands proper DNS resolution. If you are unsure of the Configuration Connection Agreement’s ability to be replicated by the ADC service, restart the ADC service and check for events suggesting that the config Connection Agreement has been misconfigured, or events mentioning a protocol error (event 8026).

ConfigDSInteg – Ported from E2kDSInteg from the previous Exchange 2000 SP2 support tool, ConfigDSInteg checks the health of Active Directory after Exchange 2003 setup and ADC have made directory modifications. This tool generates a simple report in e2kdsinteg.log that documents anomalies or suspect

Page 324: Exchange Server Admin Troubleshooting

30 Module 6: Deployment Tools and ADC Tools

objects. E2kdsinteg does not make changes to any objects in Active Directory. Depending on the number of mail-enabled objects and configuration objects in Active Directory, it may take a substantial period of time to process the mail-enabled objects for any of the following issues:

Public mailbox database contains no home message transfer agent (MTA). Public mailbox database contains no Exchange 5.5 distinguished name. Public mailbox database does not belong to an existing top-level hierarchy. Public mailbox database does not contain an owning server. Public mailbox database contains no proxy addresses. Public mailbox database contains no primary SMTP address. Public mailbox database contains no primary X.400 address. Public mailbox database contains an ambiguous Exchange 5.5 distinguished

name. Private mailbox database contains no Exchange 5.5 distinguished name. Private mailbox database is not linked to a corresponding public mailbox

database. Private mailbox database does not have a corresponding gateway object in

Active Directory. Private mailbox database contains an ambiguous Exchange 5.5

distinguished name. MTA contains duplicate Exchange 5.5 distinguished name. MTA contains an ambiguous Exchange 5.5 distinguished name. Site Replication Service contains no proxy addresses. Site Replication Service contains no primary SMTP address. Site Replication Service contains no primary X.400 address. Site Replication Service contains no home mailbox database. Site Replication Service contains no home MTA. Site Replication Service contains no Exchange 5.5 distinguished name. Site Replication Service has duplicate Exchange 5.5 distinguished name. System attendant contains no proxy addresses. System attendant contains no primary SMTP address. System attendant contains no primary X.400 address. System attendant contains no home mailbox database. System attendant contains no home MTA. System attendant contains no Exchange 5.5 distinguished name. System attendant contains an ambiguous Exchange 5.5 distinguished name. Exchange server does not belong to a routing group. Exchange server does not have a responsible MTA server. Exchange server contains no Exchange 5.5 distinguished name. Exchange server contains an ambiguous Exchange 5.5 distinguished name. Recipient policy does not contain an associated administrative group.

Page 325: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 31

Recipient Update Service contains no globally unique identifier (GUID) to the Exchange Enterprise Servers group.

Recipient Update Service contains no security identifier (SID) to the Exchange Enterprise Servers group.

Recipient Update Service contains no GUID to the Exchange Domain Servers group.

Recipient Update Service contains no SID to the Exchange Domain Servers group.

GWART contains no relative ID (RID) operations master. Connector contains no home mailbox database. The GUID in the name of this object does not reference a valid mailbox

database. MAIL_GATEWAY Object is mailbox enabled and contains no home MTA. Connector contains no Exchange 5.5 distinguished name. Connector contains an ambiguous Exchange 5.5 distinguished name. Object is of unknown type.

RecipientDSInteg – ported from E2kDSInteg in Exchange 2000 SP2, this utility complements ConfigDSInteg in that it checks for the status of suspicious mail-enabled group, user, or contact objects that might have potential problems, such as missing proxy addresses, no homeservername, etc. Even on existing Exchange 2000 server organizations, this tool is useful in finding these hidden problems that would have otherwise gone unnoticed:

User is both mail enabled and mailbox enabled. User is mailbox enabled and contains no home mailbox database. User is mailbox enabled and contains no home message transfer agent. User is mailbox enabled and contains no home server name. User is mailbox enabled and contains no mailbox GUID. User is mailbox enabled and does not contain an account control. User contains no Exchange 5.5 distinguished name. User contains no proxy addresses. User contains no primary SMTP address. User contains no primary X.400 address. User is not hidden and does not belong to an address list. Disabled User is mailbox enabled and does not contain an associated

external account. (For a user object) Keyword "ntdsnomatch" could not be found within any

extension attribute. User contains an ambiguous Exchange 5.5 distinguished name. User contains an ambiguous mailbox GUID. User contains an ambiguous master account SID. Group contains no Exchange 5.5 distinguished name.

Page 326: Exchange Server Admin Troubleshooting

32 Module 6: Deployment Tools and ADC Tools

Group contains no proxy addresses. Group contains no primary SMTP address. Group contains no primary X.400 address. Group is not hidden and does not belong to an address list. Group contains an ambiguous Exchange 5.5 distinguished name. Contact contains no Exchange 5.5 distinguished name. Contact contains no proxy addresses. Contact contains no primary SMTP address. Contact contains no primary X.400 address. Contact is not hidden and does not belong to an address list. Contact does not contain a target address. Contact contains an ambiguous Exchange 5.5 distinguished name. Public folder contains no Exchange 5.5 distinguished name. Public folder contains no proxy addresses. Public folder contains no primary SMTP address. Public folder contains no primary X.400 address. Public folder is not hidden and does not belong to an address list. Public folder does not contain a target address. Public folder contains no home mailbox database. Public folder contains an ambiguous Exchange 5.5 distinguished name. Object is of unknown type.

PrivFoldCheck – This tool performs a DS/IS against the Exchange 5.5 private information store. Like PubFoldCheck, it removes zombie entries from access control lists (ACLs) to prevent performance issues when the mailboxes are moved to Exchange 2003. Output is appended to exdeploy.log.

Page 327: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 33

Post-installation steps

The process flow for the Exchange 5.5 coexistence scenario ends after privfoldcheck, but the installer has several additional tasks if the project plan involves migration. Per figure 1.6, exdeploy.chm provides an optional path for deploying an additional server installation. This additional path does not introduce new steps from the first server installation, so its details will not be discussed here. Afterwards, exdeploy.chm leads towards the post-installation steps, which discuss moving mailboxes from Exchange 5.5 to Exchange 2003, using pfmigrate to rehome public folders from 5.5, optimizing memory usage, using the Internet mail wizard, configuring remote procedure call (RPC) over HTTP, and subscribing to Microsoft security bulletins. Among that list, the only new admin component feature from Exchange 2000 that is not covered in any other training module is PFMigrate, which is used for migrating public folder and system folder content.

Page 328: Exchange Server Admin Troubleshooting

34 Module 6: Deployment Tools and ADC Tools

PFMigrate

Once customers have moved mailboxes from Exchange 5.5 to Exchange 2003, exdeploy.chm will instruct customers to move their public folders to the new Exchange 2003 server. Pfmigrate.wsf is used to automatically update the hierarchy by programmatically changing replica settings of all public folders that are homed on any source server (Exchange 5.5 SP3 or later).

This is a script that was developed because the administrative process for rehoming public folders through Exchange System Manager is not clearly spelled out in steps, and customers administering thousands of public folders would have a frustrating migration experience by repetitively clicking through the UI.

Nevertheless, PFMigrate is not absolutely necessary, so customers may opt not to use the script. Instead, they may propagate the public/system folder replica list within the Exchange 5.5 admin.exe UI (Q185043) or Exchange System Manager (Q288150).

Usage: PFMigrate cannot be launched from the exdeploy help file. Instead, it is used from the command prompt with the following syntax:

pfMigrate.wsf /S:<source server> /T:<target server> [/WMI:<server>] [/N:<number of public folders>] [/A] [/D] [/R] [/F<log file path>] [/NNTP] [/Y] [/SYNC] [/SF]

The tool will work by connecting to an Exchange 2003 Windows Management Instrumentation (WMI) server. (Exchange 2000 does not have the necessary WMI monitoring interfaces.) The Target and Source servers must be at least Exchange 5.5 SP3. Lastly, the tool only works with the MAPI public folder hierarchy (Anything underneath “Public Folders” object in Exchange System Manager). It does not touch application public folder trees. The command line options are explained in this list:

Page 329: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 35

S: Name of the Exchange server where folders are currently replicated. Only Folders with SOURCE on the replica list will be affected. T: Name of the Exchange server where folders will be replicated to. Only folders without TARGET on the replica list will be affected. WMI: Name of the Exchange 2003 server that will provide WMI services. If not specified, the local machine will be used. N: Number of public folders to modify. This option limits the number of public folders updated by the script. If not specified, all affected folders will be updated. This is required in Add mode but optional in Delete mode. A: Adds the TARGET server to the replica list of folders where the SOURCE server is also a replica. D: Deletes the SOURCE server from the replica list of folders where the TARGET server is also a replica. R: Run a report on the current status of the SOURCE server. NOTE: /A, /D and /R cannot be specified together. F: File where log information should be appended. If not specified the default is pfmigrate.log in the current directory. NNTP: When specified, the script will not modify any of the folders in the Internet Newsgroups hierarchy. Y: When specified the script will not prompt for confirmation when running in Delete mode. SYNC: Executes the WMI query in synchronous mode. SF: Migrate System Folders: ‘Offline Address Book’, ‘Eforms Registry’ and ‘Schedule+ Free Busy’.

For example, the following command line execution of pfmigrate will add up to 4000 public folder replicas to a server called “tinetdc” and a report is generated in the “mypfs.txt” file:

>pfmigrate.wsf /S:ozpdc /T:tinetdc /n:4000 /a /f:mypfs.txt

After running the above command twice (appending the /sf switch on the second iteration), here is what will be logged to the output file:

Page 330: Exchange Server Admin Troubleshooting

36 Module 6: Deployment Tools and ADC Tools

Microsoft Exchange Public Folder Migration Tool Mon Jan 20 17:31:21 EST 2003 Source Server: OZPDC Version 5.5 (Build 2653.23: Service Pack 4) Target Server: TINETDC Version 6.5 (Build 6803.8) WMI Server: TINETDC Add Replica Mode in progress. Analysis of 4 folders with replica on 'OZPDC' completed. 3 folders without replica on 'TINETDC'. The analysis was limited by the /n parameter. Adding 'TINETDC' to the replica list of the following 3 folders: /PF-DeletedAccountOwner/ /PF-DL-ACLd/ /e55user1-PF/ 3 folders updated successfully. Mon Jan 20 17:31:38 EST 2003 -------------------------------------------------------------------------------- Microsoft Exchange Public Folder Migration Tool Mon Jan 20 17:32:56 EST 2003 Source Server: OZPDC Version 5.5 (Build 2653.23: Service Pack 4) Target Server: TINETDC Version 6.5 (Build 6803.8) WMI Server: TINETDC Add Replica Mode in progress. Analyzing system folders under 'OFFLINE ADDRESS BOOK' Analysis of 2 folders with replica on 'OZPDC' completed. Analyzing system folders under 'EFORMS REGISTRY' Analysis of 0 folders with replica on 'OZPDC' completed. Analyzing system folders under 'SCHEDULE+ FREE BUSY' Analysis of 1 folders with replica on 'OZPDC' completed. 1 folders without replica on 'TINETDC'. The analysis was limited by the /n parameter. Adding 'TINETDC' to the replica list of the following 1 folders: /NON_IPM_SUBTREE/OFFLINE ADDRESS BOOK/EX:/o=MSFT/ou=EMS/OAB Version 2/

Page 331: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 37

1 folders updated successfully. Mon Jan 20 17:33:12 EST 2003

--------------------------------------------------------------------------------

After the pfmigrate script finishes, customers will tend to open the report file immediately. However, it is possible for them to see no progress in the report, and they might not see any changes to the Exchange 5.5 replica list because of latency in the scripting engine. Because the pfmigrate report lags behind the actual processing, customers should be patient with the utility until the above text is logged.

What PFMigrate does when adding replicas: 1. Verify there is a WMI server specified 2. Verify the servers are in the same site (or Routing Group), if not, you will

see the fail string: “The Microsoft Exchange Public Folder Migration Tool only works when the source server and the target server are in the same routing group.”

3. Find the folders (via WMI, not using Active Directory) in the public folder store on the source server where the target server is not in the replica list.

4. Verify that there is a MAPI Public folder store on the Target Server and the Source Server. Otherwise, you will receive this string: “The Microsoft Exchange Public Folder Migration Tool only works with MAPI Public Folder stores. There must be a MAPI Public Folder store on both servers specified in the /s: and /t: parameters.”

5. Add the target server to the replica list a. Do not remove ANY servers from the replica list

6. Output the display name of the folder to the screen (and the log) 7. See if we have modified the input number of folders

a. If not, modify the next folder b. If so, output the completion strings to the screen (and the log)

8. If in 5a no more folders can be found, output the completion strings to the screen (and the log)

Steps for PFMigrate to delete replicas 1. Run a report (/R switch) to make sure that the source and target have the

same number of public folders (to indicate that replica list has finished replicating)

2. Prompt the user with a confirmation string to see if they really want to run delete mode. String: “Are you sure you want to want to remove %SourceServer% from the replica list? yes/no”

3. Verify there is a WMI Server 4. Verify that there is a MAPI Public Folder tree on the Target Server and the

Source Server

Note

Page 332: Exchange Server Admin Troubleshooting

38 Module 6: Deployment Tools and ADC Tools

a. Error string; “The Microsoft Exchange Public Folder Migration Tool only works with MAPI Public Folder stores. There must be a MAPI Public Folder store on both servers specified in the /s: and /t: parameters.”

5. Find the folders in the public folder store on the source server where the source server and the target server are in the replica list

6. Remove the source server from the replica list Do not remove any other servers from the replica list.

7. Output the display name to the screen (and the log) 8. Move on to the next folder 9. Run the report mode and show results

The Exchange 5.5 server actually retains the data from the removed replica until the nightly Information Services (IS) maintenance window runs, typically starting at 1:00 A.M. Therefore, even on a barely-used server, public and system folders may take hours to be removed, even after the script returns control to the command prompt.

Note

Page 333: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 39

Lesson 2: ADC Tools

This topic discusses the new Active Directory Connector (ADC) Tools feature, which is a part of the deployment tools process for the “Coexistence with Exchange 5.5” and “Coexistence with Mixed Mode Exchange 2000 and Exchange 5.5” scenarios.

Page 334: Exchange Server Admin Troubleshooting

40 Module 6: Deployment Tools and ADC Tools

A new snap-in

Installed with the Exchange 2003 version of the Active Directory Connector Services management snap-in (ADCTools.dll), an additional node appears on the left-hand side of the management console.. Figure 2.1 illustrates the new snap-in.

Page 335: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 41

Figure 2.1: The ADC Tools node is available after installing the Exchange 2003 version of the Active Directory Connector management snap-in. Half of the ADC tools’ wizards (in gray) may not be launched until prerequisite wizards have finished.

The new snap-in feature will help Administrators easily deploy Exchange 2003 in their existing Exchange 5.5 topologies, as it provides UI and logic that will help them to perform these tasks:

Plan for necessary Connection Agreements (that will synchronize Exchange 5.5 directory and Active Directory) to start the Exchange 2003 rollout.

Get the Resource mailboxes correctly replicated to the Active Directory. Automatically create and configure recipient and public folder Connection

Agreements. However, since the ADC Tools snap-in is separated from exdeploy.chm, the instructions are delivered differently. After each action, instructions appear dynamically in an information window at the bottom of the snap-in. The instructions are dynamic because, depending upon what situations are detected, the order of the steps change. For instance, if all objects in the Exchange 5.5 directory are mapped to unique Windows NT SIDs, then there is no need to run the resource mailbox wizard. Consequently, after the user has run data collection, step 4 becomes available immediately. Otherwise, step 4 will be grayed out until the user completes step 3’s resource mailbox wizard. The

Page 336: Exchange Server Admin Troubleshooting

42 Module 6: Deployment Tools and ADC Tools

dynamic messages that appear in the information window are listed in Chart 2.1, along with their corresponding situations:

Steps Substeps Situation detected

Next step Where logged

UI behavior Message in info section New Message in Info Section

First launch ofthe ADC tools

Everything is disabled except set button

ADC tools Enables you to configure Active Directory Connector, which provides synchronization between the Exchange 5.5 directory and Active Directory. To start ADC tools, under settings, click Set. Then specify an Exchange 5.5 server, port number, and log file location.

ADC Tools helps you configure Active Directory Connector, which synchronizes the Exchange 5.5 directory and Active Directory. To start ADC Tools, in Step 1 Tool Settings, click Set. Then specify an Exchange 5.5 server, LDAP port number, and log file location.

Step 1 User sets the configuration fields

need to run the data collection (step2)

Data validation button will be enabled

Step 1 is complete. Continue with Step 2 Data Collection by clicking Start. Step 2 scans your directories, detects mailbox-matching problems, and identifies recommended connection agreements. Results are stored for use in subsequent steps.

Custom matching rules detected

no Resource Mailbox wizard no Connection Agreement wizard we need to have a success marker

Error: There are custom matching rules found in the ADC installation. The ADC Connection Agreement tools will not be available.

Error: The Data Collection tool found customized object matching rules in the ADC Management properties. ADC Tools cannot be used when customized matching rules exist.

NTDSnomatch damage detected

User need to clean

NTDSNoMatch Damage Detection Scan found unmarked resource mailboxes that have been replicated and need to be cleaned up. Please consult KB articles Q256862 "XADM: How to Correct Mismatched Accounts After Active Directory Connector Replication" and Q278966 "XADM: Unable to Move or Log On to Exchange Resource Mailbox" for instructions to clean up mailboxes.

Warning: Data Collection found resource mailboxes that are mismatched or missing a master account SID. Use the following Microsoft Knowledge Base articles to resolve problems, and then rerun Step 2: Q256862, "XADM: How to Correct Mismatched Accounts After Active Directory Connector Replication" (http://support.microsoft.com?kbid=256862); Q278966, "XADM: Unable to Move or Log On to Exchange Resource Mailbox" (http://support.microsoft.com?kbid=278966).

All objects arereplicated

user do not need to run Resource Mailbox wizard , may still need to run Connection Agreement wizard (optional)

everything is enabled

Step 2 is complete. All objects are already synchronized. You do not need to run Step 3 Resource Mailbox Wizard. However, you may run Step 4 Connection Agreement Wizard to view and create recommended connection agreements.

Data collection stopped for some reason.

Check for the cause and rerun the data collection

everything is grayed

Data Collection stopped. Check the ADC Tools log file for the cause. Then repeat Step 2.

If no resource mailboxes are detected

need to go directly to Connection Agreement wizard

Resource Mailbox wizard will still be enabled Connection Agreement wizard is enabled

Step 2 is complete. No resource mailboxes were detected; therefore, you do not need to run Step 3 Resource Mailbox Wizard. Continue with Step 4 Connection Agreement Wizard.

Step 2 Run Data collection

Objects not replicated from Active Directory to Exchange 5.5

Connection Agreement wizard (maybe the Resource Mailbox wizard if there is ntdsnomatch issues)

Warning: ADUserScan detected that some mail-enabled users, contacts, or groups were not replicated from the Active Directory to the Exchange 5.5 directory.

Warning: The Data Collection tool found mail-enabled users, contacts, or groups that are not replicated from Active Directory to the Exchange 5.5 directory. Running the Connection Agreement Wizard in Step 4 will resolve these issues.

Page 337: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 43

Objects not replicated from Exchange 5.5 to Active Directory

' Warning: ADCObjectCheck detected that some objects were not replicated from the Exchange 5.5 directory to the Active Directory.

Warning: The Data Collection tool found objects that are not replicated from the Exchange 5.5 directory to Active Directory. Running the Connection Agreement Wizard in Step 4 will resolve these issues.

User Hit the cancel button

Rerun the data collection

User has cancelled the operation. The Data Collection tool was cancelled. Repeat Step 2.

Unstamped Resource mailboxes detected

Need to run the Resource Mailbox wizard step 3

NTDSNoMatch detected that some objects need to be marked as resource mailboxes before they can be imported into the Active Directory. Please review and import the changes using the Resource Mailbox wizard.

The Data Collection tool found objects that must be marked as resource mailboxes before they can be replicated to Active Directory. Running the Resource Mailbox Wizard in Step 3 will resolve these issues.

Network problem

To dialog There may be a problem with network or LDAP connectivity. Check network connections and run the Resource Mailbox Wizard again.

Permissions issues

To dialog Ensure your account has the View Only Admin role (at a minimum) in the Exchange 5.5 directory under the organization, site, and configuration nodes. Ensure that the specified Exchange server is running Exchange 5.5 SP3 or a later version. If access is still denied, add the current Active Directory account to the Exchange 5.5 server's Local Administrators group.

Everything is okay

Need to run the validation for Resource Mailbox wizard

Resource Mailbox Wizard is finished. Allow time for the changes to replicate throughout the Exchange 5.5 directory. When replication is complete, click Verify to check directory synchronization.

Run the Resource Mailbox Wizard

Not all resource mailboxes are stamped

1 .Need to allow enough time for the replication to happen, or 2. Need to run the wizard again.3. Need to make sure that the other admins have imported their CSV files

NTDSNoMatch still detects that some objects need to be marked asresource mailboxes before they can be imported to the Active Directory. If the changes recommended by the Resource Mailbox Wizard have just been imported, please allow time for Exchange 5.5 directory replication to complete before proceeding.

Warning: The Exchange 5.5 directory still contains objects that you must mark as resource mailboxes before they can be replicated to Active Directory. If you have just run the Resource Mailbox Wizard, allow time for the changes to replicate throughout the Exchange 5.5 directory. If you are using the CSV file importfeature, make sure all files have been imported into the directory. Otherwise, rerun the Resource Mailbox Wizard.

Validation stopped

Verification stopped. Check the ADCTools log file for the cause.

Cancel Admin will popup a message to warn that user needs to rerun the wizard

Verification was cancelled. Rerun the Resource Mailbox Wizard, and then click Verify again.

All resource mailboxes are stamped

Need to go to step 4

Connection Agreement wizard and final validation are enabled

Verification is complete. Continue with Step 4 Connection Agreement Wizard.

Step 3

Run the validation

The wizard was successfully run

Run the validation (step5)

Connection Agreement Wizard is finished. Allow time for the changes to replicate throughout Active Directory. Then click Verify.

Step4 Run the Connection Agreement wizard

Some objects Need to rerun Warning: ADCObjectCheck still Warning: The Exchange 5.5 directory still

Page 338: Exchange Server Admin Troubleshooting

44 Module 6: Deployment Tools and ADC Tools

are not replicated from the Exchange 5.5 directory to Active Directory

the Connection Agreement wizard to create the missing Connection Agreements or allow enough time for replication to happen.

detects that some objects were not replicated from the Exchange 5.5 directory to the Active Directory. If the connection agreements have just been created by the Connection Agreement Wizard, please allow time for Active Directory replication to complete.

contains objects that are not replicated to Active Directory. If you have just run Connection Agreement Wizard, allow time for directory replication to complete.

Some objects are not replicated from Active Directory to Exchange 5.5

Need to rerun the Connection Agreement wizard to create the missing Connection Agreements or allow enough time for replication to happen.

Warning: ADUserScan detected that some mail-enabled users, contacts, or groups were not replicated from the Active Directory to the Exchange 5.5 directory. If the connection agreements have just been created by the Connection Agreement Wizard, please allow time for Exchange 5.5 directory replication and Active Directory replication to complete.

Warning: Active Directory still contains objects that are not replicated to the Exchange 5.5 directory. If you have just run Connection Agreement Wizard, allow time for directory replication to complete.

Everything is okay

The customer is done with his ADC deployment , need to complete the deployment

ADC Tools are complete and Active Directory Connector is successfully configured. Return to the Deployment Tools or continue your Exchange deployment.

Run the last validation

Chart 2.1: Message flow summary of ADC Tools As the message flow summary suggests, each of the four steps are dependent upon those that precede them.

Page 339: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 45

Overview of the Steps

Because it is supposed to run in scenarios where Exchange 2003 or Exchange 2000 coexist with Exchange 5.5, ADC Tools will not address topologies that have multiple Exchange organizations, as ADC tools will neither recommend nor create inter-organizational Connection Agreements. The ADC Tools are more intrusive than the rest of the deployment tools, as the ADC Tools’ wizards collect, modify, and mass-manipulate directory data more accurately than the rest of the deployment tools. These wizards require strict ordering that could not be accomplished within the help file (exdeploy.chm/hta). It also requires more user interaction, as any mistakes could greatly affect its ability to mass-modify Exchange 5.5 directory objects (custom attribute 10), and also to create connection agreements. ADC tool’s general messages are directed towards the adctools.log file, but these messages are also summarized in the snap-in’s informational window. Exdeploy.log does not get written by the ADC Tools, since ADC Tools runs under completely different context from exdeploy.exe.

Tool Settings overview: Step 1 is fairly simple. The user sets the Exchange 5.5 server that has good knowledge consistency, and the logging path is pointed within the user’s profile by default. (Typically C:\Documents and Settings\username\Local Settings\Application Data\Microsoft\Active Directory Connector). Also, the ADCTools.log file is initialized.

To examine the files generated by ADCTools, copy and paste the path from the snap-in’s window. Inexperienced administrators may not be able to locate this path when navigating through explorer, since “Local Settings” is a hidden directory.

Although the user is able to choose the Exchange 5.5 server, the global catalog used by ADC Tools is not user-selectable. When the ADC Tools chooses a global catalog, it sticks with it unless there are network problems. To determine which global catalog is being used by ADC Tools, enable ADC Tools debugging as discussed later in this module.

Data collection overview: Scans the Exchange 5.5 directory for inconsistencies, such as unreplicated objects or groups of objects that are

Troubleshooting Tip

Page 340: Exchange Server Admin Troubleshooting

46 Module 6: Deployment Tools and ADC Tools

associated with one security identifier. Decides what types of connection agreements are needed, and outputs XML data files, which are used by subsequent ADC Tools. Stamps ADCUserCheck and ADCObjectCheck markers in AD. To run this tool properly, one should be logged on with credentials at least the level of permissions admin in the org, site, and configuration containers in the Exchange 5.5 directory.

Resource Mailbox Wizard and Verification overview: Reads from the ntdsnomatch.xml file, and renders into a user-friendly format for browsing/expanding user accounts that are associated with more than one mailbox object. Like NTDSAtrb, if the Exchange 5.5 alias=samaccountname, the Resource Mailbox Wizard will recommend (in bold) the primary mailbox to be associated to the user account. All non-bolded objects will have their custom attribute 10 modified with “NTDSNoMatch.” These will be treated as resource mailboxes, such that after ADC replication, these will be represented in Active Directory as disabled, mailbox-enabled security principals. Additionally, NTDSNoMatched accounts will have their Primary Windows NT accounts swapped after ADC replication. Resource Mailbox Wizard will output ntdsnomatch_<sitename>.csv into the logging path. Connection Agreement wizard and Validation overview: Primarily reads from the adcobjectcheck.xml and ntobjectcheck files that contain the lists of recommended Connection Agreements. The administrator then has an option of whether or not to continue, in case custom Connection Agreements are found. If the administrator continues with this wizard, their custom connection agreements will be deleted and replaced by the Connection Agreement wizard. The Connection Agreement wizard is not meant to replace existing ways of creating/managing Connection Agreements as there will be customers who need more complex settings (like specific OU settings, custom filters, etc.) Next, the administrator provides credentials for each site/domain where unreplicated users were found, so that they may be transferred to the parameters of the connection agreements upon creation. The validation step is similar to another data collection step, since it generates another set of adcobjectcheck.xml and ntobjectcheck.xml files to verify that the Active Directory Connector service has run the new Connection Agreements and that objects are now synchronized between both directories. However, the validation step is doubly important because it stamps the ADCUserCheck marker again, but this time it should do so with a success (0) flag in Active Directory.

What Steps Can Run? The ADC Tools decide what steps can be run, based on the presence of various XML files.

The ADC Tools cannot run if there are custom matching rules (CustomRules.XML is present).

Step 3, the Resource Mailbox Wizard will run if NtdsNoMatch.xml is present, and customRules.XML is not.

Step 4, the Connection Agreement Wizard will run if adcObjectCheck.xml and ntObjectCheck.xml are present, and customRules.XML is not.

If you ever get into a state where the next step will not unlock then either

1. Wait for replication. The log files should tell you what objects the ADC Tools have not updated yet.

2. Re-run data collection.

Page 341: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 47

Data Collection output files

Although ADC Tools is a straightforward process flow, much of the preparation and decision-making is performed by the tools in Step 2’s Data collection. Although ADC Tools automatically picks a global catalog from which to analyze active directory objects, the data collection tool cannot run without correct entry of server name/port number of Exchange 5.5 from Step 1.

Data Collection may generate up to five possible files utilized by subsequent wizards: customrules.xml, ADCObjectCheck.xml, NTObjectCheck.xml, ntdsnomatch.xml, and badresourcemailboxes.xml. Data Collection also updates (or create if it does not exist) the ADCTools.log with the same summaries and corrective actions as the information window (lower portion of figure 2.1), but it also writes additional information such as lists of problem accounts. The logfile is never deleted or renamed; rather, it is appended upon execution of each of the following buttons:

Data Collection “Run” button. Step 3 “Verify” button. Step 4 ”Verify” button.

Lastly, the Data Collection button will write these two markers on the description attribute of the cn=Microsoft Exchange… object:

ADCUserCheck ADCObjectCheck

Sample ADCTools.log output and explanations of common error outputs during data collection In one execution of the data collection wizard, there are four passes listed in adctools.log. In the following text boxes, these four passes are broken down with descriptions.

Page 342: Exchange Server Admin Troubleshooting

48 Module 6: Deployment Tools and ADC Tools

As its name implies, this pass scans for accounts that need to be designated as resource mailboxes. When it writes to adctools.log, it will not list any accounts that need to be NTDSNoMatched. Instead, the lists of accounts to be ntdsnomatched are written to ntdsnomatch.xml, while adctools.log will only list problems running the tool, and a corrective action.

If the entry says “Error: Security identifier for the associated…could not be resolved” then this means that Exchange 5.5 object’s associated account was deleted or could not be found in the forest. So this associated account could be a deleted account, or an account within another Windows NT 4 or 200x domain in another forest whose trust has been broken from the current forest.

If the entry says “Could not find match to ‘cn=user…’ then this means that the Exchange 5.5 object is associated with a deleted account, or an account outside of this Active Directory forest.

If the entry says “Matched ‘cn=user,cn=recipients,ou=sitename,o=orgname’ to ‘cn=user,dc=domain,dc=com’ then this means that the Exchange 5.5 object’s Primary Windows NT account is an account that was found within the Active Directory forest. However, no connection agreement has yet matched and stamped mail attributes onto the Active Directory account.

If the entry says “The AD Object is tombstoned…” then this means that the Exchange 5.5 object’s associated account was deleted from Active

Current user is 'Administrator\MS' on computer 'Z2' Pass 1 of 4: Resource Mailbox Scan 02/07/2003 23:29:16

Error: Security identifier (SID) for the associated Windows NT account (Assoc-NT-Account) for the mailbox 'cn=deleteguy,cn=Recipients,ou=Remote55Site,o=Microsoft' could not be resolved.

Warning: The Data Collection tool found objects that must be marked as resource mailboxes before they can be replicated to Active Directory. Running the Resource Mailbox Wizard in Step 3 will resolve these issues.

Pass 2 of 4: Active Directory Connector Object Replication Check 02/07/2003 23:29:17 Could not find match to 'cn=remotec,cn=Recipients,ou=Remote55Site,o=Microsoft'. Matched 'cn=remoted,cn=Recipients,ou=Remote55Site,o=Microsoft' to 'CN=remoted,OU=remote55,DC=ms,DC=com' based on SID. Could not find match to 'cn=deleteguy,cn=Recipients,ou=Remote55Site,o=Microsoft'. Matched 'cn=hubuserA,cn=Recipients,ou=HubSite,o=Microsoft' to 'CN=hubuserA,CN=Users,DC=ms,DC=com' based on SID. Matched 'cn=hubuserB,cn=Recipients,ou=HubSite,o=Microsoft' to 'CN=hubuserB,CN=Users,DC=ms,DC=com' based on SID. Warning: The Data Collection tool found objects that are not replicated from the Exchange 5.5 directory to Active Directory. Running the Connection Agreement Wizard in Step 4 will resolve these issues.

Page 343: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 49

Directory but not yet "garbage collected." Viewing the Exchange 5.5 object’s Primary Windows NT Account field should yield “Unknown Account”

If the entry says “Multiple AD objects..” then this means that the 5.5 object was improperly replicated to two different Active Directory objects

This pass lists any Exchange 5.5 objects that have not been replicated to Active Directory, and it has the same algorithm as some of the deployment tools. You can obtain similar output by running this command line:

exdeploy.exe /t:adcusercheck /t:adcobjectcheck

Pass two of four will also write output to the adcobjectcheck.xml file, which is discussed later.

Pass three of four (Active Directory Object Replication Scan) does not output to adctools.log, and instead writes exclusively to ntobjectcheck.xml. It lists any mail-enabled Active Directory objects that have not been replicated to the Exchange 5.5 directory, and it does this to hidden objects as well. It also shares exdeploy code, and you can get the same output from running

exdeploy.exe /t:aduserscan

This pass contains code unique to Exchange 2003, in that it checks for disabled objects in Active Directory that are missing msexchMasterAccountSID values. Such accounts would meet this criteria if they were prematurely replicated by a connection agreement, when they first should have been designated as resource objects. If any such Active Directory objects meet this criteria, the adctools.log

Pass 3 of 4: Active Directory Object Replication Scan 02/07/2003 23:19:49

Active Directory Object Replication Scan completed. No unreplicated objects found.

Pass 4 of 4: Active Directory Unmarked Resource Mailbox Scan 02/07/2003 23:19:50 Warning: Active Directory Unmarked Resource Mailbox Scan found resource mailboxes that are mismatched or missing a master account SID. Use the following Microsoft Knowledge Base articles to resolve problems, and then rerun Step 2: Q256862, "XADM: How to Correct Mismatched Accounts After Active Directory Connector Replication" (http://support.microsoft.com?kbid=256862); Q278966, "XADM: Unable to Move or Log On to Exchange Resource Mailbox" (http://support.microsoft.com?kbid=278966).

Page 344: Exchange Server Admin Troubleshooting

50 Module 6: Deployment Tools and ADC Tools

will instruct which KB articles to use for clean-up. All other output is written badresourcemailboxes.xml.

It is very possible that if customers disable non-migrated accounts as part of their administration process, that the manually-disabled accounts are reported among this list. But you can still confirm whether they were replicated by the ADC by looking at their “Description”, which should say “Disabled Windows Account.”

Sample XML output and explanations of common error outputs during data collection Customrules.xml - This XML file is only created if the ADC is already configured with custom matching rules. The Step 3 and Step 4 wizards in ADC Tools will not run if this file exists. If this is the case, then customers must use the traditional recipient connection agreement configuration pages.

Ntdsnomatch.xml – written during Pass one of four of data collection, this XML file contains a list all of the Windows NT 4 or Active Directory accounts that are associated with more than one Exchange 5.5 directory object. This is necessary to prevent mismatching of resource mailboxes to accounts, since Active Directory objects require 1-to-1 associations, whereas Exchange 5.5 objects were not limited. The NTDSNoMatch.xml file has an organization-wide scope, and it also checks for hidden objects. Here is a sample output:

Note

Page 345: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 51

<?xml version="1.0" encoding="utf-16"?>

<ntdsnomatch exchange55="root55:389" gc="TILAB-DC" date="200301131241"

existingextatt10data="false">

<Account sid="0105000000000005150000002CC77192C6AD34CAF0086B1659040000"

account="TILAB\Bill_the_VP" existingextatt10data="false">

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>resourcemailbox1</Display_Name>

<Alias_Name>resourcemailbox1</Alias_Name>

<Directory_Name>resourcemailbox1</Directory_Name>

<Home-Server>ROOT55</Home-Server>

<Obj-Container>/o=Microsoft/ou=GSXSite1/cn=Recipients</Obj-Container>

</mailbox>

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>conf room resource</Display_Name>

<Alias_Name>conf room resrouce</Alias_Name>

<Directory_Name>conf room resrouce</Directory_Name>

<Home-Server>ROOT55</Home-Server>

<Obj-Container>/o=Microsoft/ou=GSXSite1/cn=Recipients</Obj-Container>

</mailbox>

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>bigbill</Display_Name>

<Alias_Name>bigbill</Alias_Name>

<Directory_Name>bigbill</Directory_Name>

<Home-Server>ROOT55</Home-Server>

<Obj-Container>/o=Microsoft/ou=GSXSite1/cn=Recipients</Obj-Container>

</mailbox>

Page 346: Exchange Server Admin Troubleshooting

52 Module 6: Deployment Tools and ADC Tools

</Account>

<Account sid="0105000000000005150000002CC77192C6AD34CAF0086B165A040000"

account="TILAB\group_ems" existingextatt10data="false">

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>group mailbox1</Display_Name>

<Alias_Name>group mailbox1</Alias_Name>

<Directory_Name>group mailbox</Directory_Name>

<Home-Server>ROOT55</Home-Server>

<Obj-Container>/o=Microsoft/ou=GSXSite1/cn=Recipients</Obj-Container>

</mailbox>

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>group mailbox2</Display_Name>

<Alias_Name>group mailbox2</Alias_Name>

<Directory_Name>group mailbox2</Directory_Name>

<Home-Server>ROOT55</Home-Server>

<Obj-Container>/o=Microsoft/ou=GSXSite1/cn=Recipients</Obj-Container>

</mailbox>

</Account>

</ntdsnomatch>

If mailbox primary = true, this would indicate that ADC Tools is simply

recommending that the current object (typically mailbox class) has an alias that

matches the samaccountname. Otherwise, ADC Tools logic defaults to recommending the

5.5 object as a resource by placing NTDSNoMatch on custom attribute 10. The

recommendation isn’t applied until the Resource mailbox wizard, discussed later.

ADCObjectCheck.xml – In pass 2 of 4, this file contains a list of Exchange 5.5

directory objects (mailboxes, custom recipients, and distribution lists) that have

not replicated from 5.5 to AD. From this output, we also can view what types of

connection agreements are recommended and whether they are primary CAs. The

following is a sample output:

<?xml version="1.0" encoding="utf-16"?>

<ADCObjectCheck exchange55="ozpdc:389" gc="TINETDC" date="200301062103">

<Objects>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=&quot;EX:_O=MSFT_OU=EMSC6B44F55C6B44F55C6B44F55203BF8A3000007&quot;,c

n=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=&quot;EX:_O=MSFT_OU=EMSC6B44F55C6B44F55C6B44F55203BF8A3000008&quot;,c

n=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=EVENTCONFIG_OZPDC4948743F4948743F4948743F182909A6000011,cn=Recipients

,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user1,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

Page 347: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 53

<Sid>010500000000000515000000E07B3345D93C346AD3448378F0030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user2,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F1030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user3,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F2030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user4,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F3030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user5,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F4030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=ResourceU,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F0030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=ResourceG,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F5030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=OAB VERSION

24948743F4948743F4948743F182909A6000BC1,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=E55USER1-

PF4948743F4948743F4948743F182909A600177A,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=PF-DL-

ACLD4948743F4948743F4948743F182909A600177B,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="group">

<ldapDN>cn=55DL1,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=PF-

DELETEDACCOUNTOWNER4948743F4948743F4948743F182909A600177D,cn=Recipients,ou=EMS,o=MS

FT</ldapDN>

</object>

</Objects>

<RecommendedCAs>

<CA CAtype="user" primary="true" site="EMS" >

<exportContainer>ou=EMS,o=MSFT</exportContainer>

Page 348: Exchange Server Admin Troubleshooting

54 Module 6: Deployment Tools and ADC Tools

</CA>

<CA CAtype="group" primary="true" site="EMS" >

<exportContainer>ou=EMS,o=MSFT</exportContainer>

</CA>

<CA CAtype="folder" primary="true" site="EMS" >

<exportContainer>ou=EMS,o=MSFT</exportContainer>

</CA>

</RecommendedCAs>

</ADCObjectCheck>

Extension-Attribute-10 will simply report if the Exchange 5.5 directory object has Custom Attribute 10 populated.

NTObjectCheck.xml – In pass three of four, this file enumerates any users in Active Directory which contain mail-enabling attributes, but do not have ADC-Global-Names present. (Without ADC-Global-Names, one may truly declare that a 2-way Connection Agreement hasn’t fully replicated for this Active Directory object.) Similar to ADCObjectCheck, ntobjectcheck will recommend the connection agreements necessary for proper replication of this object. Sample ntobjectcheck.xml output follows:

Page 349: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 55

<?xml version="1.0" encoding="utf-16"?>

<NTObjectCheck exchange55="ozpdc:389" gc="TINETDC" date="200301131032">

<Objects>

<object domain="ti" site="EMS" primary="false" CAtype="user">

<ldapDN>CN=billy harris,CN=Users,DC=ti,DC=vm</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<ObjectGuid>56E4DEEBC405214FBDA9A71742E3B38B</ObjectGuid>

<LegacyExchangeDN>/O=MSFT/OU=EMS/cn=Recipients/cn=billyh</LegacyExchangeDN>

<msExchHomeServerName>/O=MSFT/OU=EMS/cn=Configuration/cn=Servers/cn=TINETDC</msE

xchHomeServerName>

</object>

<object domain="ti" site="EMS" primary="false" CAtype="contact">

<ldapDN>CN=william taylor,CN=Users,DC=ti,DC=vm</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<ObjectGuid>BE0E225EBA49854CB889DC3655890BCB</ObjectGuid>

<LegacyExchangeDN>/O=MSFT/OU=EMS/cn=Recipients/cn=williamtaylor</LegacyExchangeD

N>

<msExchHomeServerName></msExchHomeServerName>

</object>

</Objects>

<RecommendedCAs>

<CA CAtype="user" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>ou=EMS,o=MSFT</importContainer>

</CA>

<CA CAtype="contact" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>OU=EMS,O=MSFT</importContainer>

</CA>

<CA CAtype="group" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>ou=EMS,o=MSFT</importContainer>

</CA>

<CA CAtype="folder" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>ou=EMS,o=MSFT</importContainer>

</CA>

</RecommendedCAs>

</NTObjectCheck>

The <Objects> section lists any users/mailboxes/groups that have not replicated to the Exchange 5.5 directory. The <RecommendedCAs> section lists any options that should be checked for the connection agreement. For example, if the mail-enabled contact “William Taylor” didn’t exist, you would not expect to see the “contact” entry in the latter section.

For the coexistence with Exchange 5.5 deployment scenario, you should not expect ntobjectcheck.xml to contain any elements, as Exchange 2003 will not yet have been installed. For the “Coexistence with Mixed Mode Exchange 2000 and Exchange 5.5” scenario, you may expect data in the <Objects> section.

Note

Page 350: Exchange Server Admin Troubleshooting

56 Module 6: Deployment Tools and ADC Tools

Badresourcemailboxes.xml – In pass four of four, this file is created if any Active Directory accounts are missing (or had mismatched) msExchMasterAccountSID values. This would have occurred if a Recipient Connection Agreement was replicated before any NTDSNoMatching occurred. To fix the associations, parse through the XML file for all the accounts, follow KB articles Q256862 and Q278966 to reset their states, and then follow through the rest of the ADC tools. You would not expect to have this file generated in the “Coexistence with Exchange 5.5” deployment scenario, but it is probable that this file is generated when customers are deploying in the “Coexistence with Mixed Mode Exchange 2000 and Exchange 5.5” scenario.

Troubleshooting mini lab: How to get missing MSExchMasterAccountSIDs, and then identify them using ADC Tools:

1. Create a mailbox called “primary1” and associate it with an Active Directory account.

2. Create a mailbox called “resource1” and associate it with the same active directory account.

3. Immediately replicate a primary recipient connection agreement, and note Event ID 8281 indicates that “resource1” could not be stamped with msExchMasterAccountSID because that value already existed on the first object.

4. Use ADC Tools’ data collection to scan Active Directory, and note the badresourcemailboxes.xml file gets generated. Note how the information window and ADCTools.log instruct you to utilize KB articles to rollback.

To rollback:

1. Make sure that any Connection Agreements replicating the above containers are unscheduled, or that their owning ADC services are stopped.

2. Remove, using 5.5 raw mode, the ADC-Global-Names values from the “resource1” mailbox object.

3. Delete the disabled “resource1” user from Active Directory. To rectify:

1. Rerun the Step 2 Data Collection. All the XML files from the previous data collection will be renamed to *.bak extensions, and ADCTools.log is appended with a message stating that resource1 needs to be designated as a resource mailbox.

2. Run Step3’s resource mailbox wizard (discussed in the next section) to mark the appropriate objects as resource objects.

3. Replicate the previous connection agreement, or have step 4’s Connection Agreement wizard create and replicate connection agreements for you.

Troubleshooting the XML files: For ADCTools output, note that repeated execution of data collection wizard will not append to existing XML files. Instead, older XML files are renamed with a timestamp and .bak extension before being replaced by newer XML files. To launch the XML file for easy expansion/readability from Internet Explorer, you will need to have the XML parser installed. Otherwise, open the XML file using Notepad.

Page 351: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 57

Resource Mailbox Wizard

In Step 3, the Resource Mailbox wizard button becomes clickable once the Data Collection Wizard has completed. Resource Mailbox wizard will read from NTDSNoMatch.XML and allows customers to toggle which mailbox should be primary or secondary. Figure 2.2 depicts the Resource Mailbox wizard:

Figure 2.2: The interactive portion of Resource Mailbox wizard allows for customers to review recommendations provided from ntdsnomatch.csv. The Resource Mailbox wizard replaces NTDSAtrb.exe for identifying resource mailboxes.

Page 352: Exchange Server Admin Troubleshooting

58 Module 6: Deployment Tools and ADC Tools

When navigating through the mailbox list, you can expand or collapse each node. A node may be a Windows NT 4 account or a security principal within the Active Directory forest, and its child objects are all of the multiple Exchange 5.5 mailboxes pointing to the same assoc-NT-account. Since there should only be one account per mailbox set as the Primary for Exchange 2000 and 2003 to work correctly, that is the reason we have this wizard.

When picking primary/resource mailboxes to avoid object mismatching, the user needs to highlight an account to be designated as the primary. When clicking the button labeled “set as primary,” the highlighted mailbox gets bolded, and any other mailbox object under the same umbrella that was previously bolded will become unbolded. Unbolded child objects are considered to be resource mailboxes, and as such, will get NTDSNoMatch’d on their custom attribute 10 when the wizard completes. In most cases, the Resource Mailbox wizard will already suggest which of the Exchange 5.5 objects will be bolded, because it detects a match between the mailbox’s alias and the Windows user logon name.

In cases that it does not detect that match, it will leave everything unbolded and leave it to the user to decide which one to set as primary. It is not a problem to leave all mailboxes under a common node unbolded, because like in the figure above, since both group mailboxes would simply be NTDSNoMatched, which still allows for the account (or the global group in this case) to continue to have full mailbox access to both mailboxes.

After selecting the primary/secondary mailbox objects, there are two options that the deployment administrator can take:

Option 1 - Use the “Export” button: This option reads the entries from the window and sends them to a Comma-Separated-Values (CSV) file which is easily readable and modifiable by spreadsheet software such as Excel. Customers possessing very large directories may find that the window is too small to manipulate hundreds of potential mismatches. So the next logical step for these customers would be to quit the Resource Mailbox wizard, and then use a spreadsheet or database program to review and modify the CSV file to designate resource mailboxes. Lastly, they would use the Exchange 5.5 admin.exe program to import each CSV into its corresponding site, thereby mass-modifying the Exchange 5.5 mailboxes’ custom attributes. This option is not much different from the NTDSAtrb.exe tool, except that NTDSAtrb is launched from a command prompt. In fact, the CSV file generated by NTDSAtrb and Resource Mailbox wizard follow the same format:

Obj-Class

Extension-Attribute-10 Display Name

Primary Windows NT Account Alias Name

Mailbox NTDSNoMatch resourcemailbox1 TILAB\Bill_the_VP resourcemailbox1

Mailbox NTDSNoMatch conf room resource TILAB\Bill_the_VP

conf room resource

Mailbox ~DEL bigbill TILAB\Bill_the_VP bigbill Mailbox NTDSNoMatch group mailbox1 TILAB\group_ems group mailbox1 Mailbox NTDSNoMatch group mailbox2 TILAB\group_ems group mailbox2

Chart 2.2: The first five columns of an ntdsnomatch_sitename.csv file, as exported from the Resource Mailbox wizard. Note that ~DEL clears the custom attribute 10 field for bigbill, which guarantees that the bigbill mailbox is the primary (personal) mailbox for Bill_the_VP. When this file is imported into the Exchange 5.5 directory, four objects (except bigbill) will be identified as resource mailboxes, such that when the ADC eventually reads them, object matching rules are ignored and object-

Page 353: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 59

creation ensues in Active Directory. The disabled objects will appropriately have the msExchMasterAccountSID populated. When the 2-way Connection Agreement follows the second half of the replication cycle (from Active Directory to Exchange 5.5), it changes the resource mailboxes’ Primary Windows NT Account values to point to the SIDs of the newly-generated Active Directory objects (that are disabled). This process ensures a 1-to-1 association of every mailbox to user account in both directories, thereby preventing many permissions, delegation, and public folder problems.

On the other hand, if these resource mailboxes were not designated, then aside from the possibility of mismatching conf room resource as Bill_the_VP’s primary mailbox, the resource mailbox (as well as the real personal mailbox for Bill_the_VP) will have disabled objects created for them in Active Directory. Disabled objects whose Exchange 5.5 sources were not designated with NTDSNoMatch will not have the msexchmasteraccountSID populated, and as such, 9548 errors will be logged during folder access. Furthermore, when the ADC performs its back-replication, the primary Windows NT accounts are not switched. You can expect to find 9551 error events if these mailboxes were ACLed on folders.

Option 2 – Make window modifications through the GUI, and allow Resource Mailbox wizard to modify the Exchange 5.5 directory. (This is the recommended and most common scenario). This option involves performing the custom attribute 10 sets/unsets in a manner that prevents the administrator from inadvertently setting two primary mailboxes on the same security principal. The administrator would not use the export button, and would instead proceed to the next dialog within the wizard. (Note at the CSV files are exported anyway, but they will be programmatically parsed by the Resource Mailbox wizard GUI.) After providing credentials to the Exchange 5.5 site(s), Resource Mailbox wizard will automatically connect to an Exchange 5.5 server in each site and mass-modify custom attribute 10 for the Exchange 5.5 directory objects. The Resource Mailbox Wizard is completed at this point, and a summary window appears.

Pay close attention to the summary window. If there are any errors in the import process, it will tell you which file to examine. The files will usually be XML files, and you can view them either with notepad, or by double-clicking them to launch them inside Internet Explorer’s XML parser:

Troubleshooting Tip

Troubleshooting Tip

Page 354: Exchange Server Admin Troubleshooting

60 Module 6: Deployment Tools and ADC Tools

Figure 2.3: Error code hidden within an XML file, viewed by the XML3 parser. The parser is installed when the Active Directory Connector Management snap-in is installed.

How to troubleshoot a known issue: 0XC00000CE error with Resource Mailbox wizard By design, Resource Mailbox wizard will impersonate the Exchange 5.5 site account before calling the Exchange Directory Access Programming Interface (DAPI) so that DAPI runs in the correct context to be able to import into the Exchange 5.5 directory.

In more detail, the Exchange 5.5 site account that will be used to import changes into Exchange 5.5 must first read the CSV files. However, the CSV might be unreadable from the logging path, typically located under C:\Documents and Settings\username\Local Settings\Application Data\Microsoft\Active Directory Connector. This is because the ADC Tools admin’s profile directory (locked by NTFS by default) may be inaccessible by the impersonated Exchange 5.5 admin’s security context. So, an error gets logged to a newly-generated XML file called ntdsNoMatch_55SiteNameError.xml. The user will need to view the XML file to see the error (such as in figure 2.3) before performing a Knowledge Base lookup.

The user can always work around this by quitting the Resource Mailbox wizard GUI, copying the CSV file onto the Exchange 5.5 server logged in as someone appropriate, and then use admin.exe to import the CSV. Alternatively, the

Page 355: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 61

person logged-onto the ADCTools machine can change the NTFS permissions to allow the Exchange 5.5 account CHANGE NTFS permissions on the C:\Documents and Settings\username\Local Settings\Application Data\Microsoft\Active Directory Connector folder, thereby allowing the Exchange 5.5 site admin credentials to access the CSVs upon the next execution of Resource Mailbox wizard.

Once the Resource Mailbox wizard completes, or if a CSV file is used to manually import the changes, then the Exchange 5.5 directory needs to be verified to ensure no mailboxes will fail the 1-to-1 association test.

The verify button in Step3 does this by writing a pass/fail summary to adctools.log, and recording any DNs (into ntdsnomatch.xml) of objects that still need to be ntdsnomatched. If VERIFY fails, this could be a misconfigured Connection Agreement, but is more commonly a replication latency issue.

When Resource Mailbox wizard makes changes to multiple sites, the administrator must wait for inter-site replication to complete, because ADC tools settings’ (Step 1) are only pointed at one server. Until replication is completed to the pointed server, the VERIFY button fails and does not allow administrators to proceed to the next step.

Depending on the case, the admin needs to wait for replication or fix whatever accounts did not get marked as secondary before running another verify. Otherwise, the administrator cannot proceed to step 4: Connection Agreement Wizard.

Important

Page 356: Exchange Server Admin Troubleshooting

62 Module 6: Deployment Tools and ADC Tools

Connection Agreement Wizard:

In Step 4, the Connection Agreement wizard automates the creation of connection agreements. In past Exchange 5.5/2000 deployments, customers had difficulties creating Connection Agreements because there were numerous options and property sheets to configure for each recipient connection agreement. Furthermore, it was difficult to determine which options were needed. For example, customers would not know when Connection Agreements needed to be set as primary, as they could not easily determine whether Active Directory objects simply needed matching and stamping, or if object creation would be required to keep directories in sync. Lastly, Connection Agreements are fairly complex and there would be no true base configuration documentation for how the Connection Agreements should have been configured because their ideal configurations depended tremendously upon the existing Exchange 5.5 site and domain topologies, which vary from customer to customer.

The Connection Agreement wizard removes the complexity from the user by parsing the ADCObjectCheck.XML and NTObjectCheck.XML files. The <RecommendedCAs> sections of the XML files define several properties, including the replication flow (1-way or 2-way) of new Connection Agreements, and whether or not they should be primary. For that given moment in time, the Connection Agreement properties defined by the XML files are sufficient to get the Global Address Lists (GALs) replicated, and are thus called “perfect” for that instance. If the Connection Agreement wizard detects that custom connection agreements exist, it compares them to the “perfect” properties of each Connection Agreement it intends to create. (The term “perfect” is what the development team gave to Connection Agreements that are created by ADC Tools because they are ideal for that moment in time to satisfy getting the GALs synchronized). If the properties of the existing Connection Agreement match those of the “perfect” Connection Agreements defined by the XML files, then the Connection Agreement wizard will not show the prompt shown below:

Page 357: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 63

Figure 2.4: When the Connection Agreement Wizard detects custom connection agreements that do not meet the ideal properties defined by the XML files, it gives the user a decision to skip Connection Agreement wizard.

If the administrator chooses to continue the wizard, the listed connection agreements are not immediately deleted. Instead, they are only deleted if the administrator finishes the Connection Agreement Wizard. Companies that have already deployed Exchange 2000 will most likely get this dialog prompt. If the companies have large topologies, or have spent a lot of work creating their custom Connection Agreements to suit their needs, the Connection Agreement wizard will not address their scenario, and they should not allow the Connection Agreement wizard to continue to remove and replace their custom Connection Agreements. Furthermore, the Connection Agreement wizard will not work in scenarios having custom matching rules, though companies who have invested the time to create rules are most likely experienced enough to properly configure their connection agreements without the assistance of ADC Tools. The intended audience for the Connection Agreement wizard is for the small-to-medium customer that does not have expertise or financing to hire consultants. For this audience, continuing to delete any existing misconfigured connection agreements is recommended.

The next section of the Connection Agreement wizard is the staging area selection. The staging area needs to be chosen as a “miscellaneous” container for where any unmatched objects within a forest are dumped. For example, distribution lists in Exchange 5.5 are not associated with Windows NT SIDs, and therefore, will never be matched to existing Active Directory accounts in any other domains. Therefore, there must be some organizational unit (OU) within the current domain where these unmatched objects are created. And so by selecting the staging area OU, it becomes the “Default Destination”

Page 358: Exchange Server Admin Troubleshooting

64 Module 6: Deployment Tools and ADC Tools

parameter (msexchserver1importcontainer) for the recipient Connection Agreement’s “From Exchange” tab. However, choosing the staging area somewhat breaks the multi-site Exchange 5.5 administrative model, since distribution lists (DLs) can replicate into a domain falling outside their realm of administration (due to the fact that customers deploy Windows NT 4 domains and Exchange 5.5 sites very closely). This is by design, since the goal of Connection Agreement Wizard is to sync the Exchange 5.5 GAL with mail-enabled Active Directory objects. So although the administrative model for multi-domain organizations is not ideal, it is sufficient to resolve most bad replication issues Microsoft Product Support Services has seen in Exchange 2000. An alternative to having other sites’ objects from replicating into the staging domain (which is outside their realm of control), is to first deselect some remote sites’ Connection Agreements from being created at the end of the Connection Agreement wizard, and then rerun the Connection Agreement wizard and choose a new staging area residing with those remote Active Directory domains. Alternatively, it may be desirable to plan a neutral domain to become the staging area for all sites and domains.

The next decision in the Connection Agreement Wizard is the Site Connections dialog, which asks whether Connection Agreements are desired to be 1-way or 2-way. If switched to one-way, these Connection Agreements will have their “From Windows” tabs disabled.

Figure 2.5: Site Connections dialog. The Connection Agreement Wizard will never recommend one-way connection agreements.

The Connection Agreement wizard will never recommend a 1-way Connection Agreement, but it will give the option for customers to make the switch. Not all Connection Agreements that the Connection Agreement Wizard intends to create will be shown amongst this list:

Page 359: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 65

Public Folder Connection Agreements are not shown, since they are always 2-way.

Recommended recipient Connection Agreements that are 1-way from “Windows to Exchange” will not be shown. (It is not expected that the ADC Tools will ever detect a situation where this is recommend.)

The next two dialog boxes, labeled “Site Credentials” and “Domain Credentials,” require that the user enter username and passwords so that these will be stored onto the connection agreement properties. The username and server names will be written to the “Connections” tab, while the validated password is written into link state algorithm (LSA) global secrets.

Since Connection Agreement Wizard will say a user is “validated,” it is important to realize that validation simply means that the user’s password has been verified as correct; it does not mean that the specified account credentials have been tested to have the “permissions admin” role to modify the directories.

Figure 2.6: Connection Agreement Wizard needs credentials for each site and domain for purposes of authentication.

Since the Connection Agreement wizard needs to contact Exchange 5.5 directory servers or site replication directory services on every connection agreement which it creates, the Connection Agreement wizard will not work with disconnected sites (or sites where firewalls block port 389), and the validation will fail. In such an instance, one could bypass this problem by removing the disconnected sites’ entries from the XML file used by the Connection Agreement wizard, and then later rerun Connection Agreement wizard from a machine within the disconnected site.

Following the credentials screen, the user has an opportunity to choose not to create some Connection Agreements. Unless they desire to create another staging area, it is not recommended for them to uncheck any planned connection agreements.

Note

Page 360: Exchange Server Admin Troubleshooting

66 Module 6: Deployment Tools and ADC Tools

Figure 2.7: Reviewing which Connection Agreements are to be created.

In the above example, where all Exchange 5.5 sites contain unreplicated mailboxes whose associated Primary Windows NT accounts reside in all domains throughout a forest, then the Connection Agreement wizard will created a perfect mesh where the number of recipient Connection Agreements created by the Connection Agreement wizard is based on (number of sites) * (number of active directory domains). Typically, in a decentralized administration model, the number of recipient connection agreements is lower because most Exchange 5.5 administrators keep their all their directory objects associated with only one Active Directory domain over which they control.

A final summary screen is then shown to the user, and if they select “Finish”, the process to create connection agreements begins. The creation process is the most delicate part of the Connection Agreement Wizard. This process could take several hours in a large topology spanning slow links. During this time, the ADC Tools are not resilient enough to re-query and retry in the event of a serious network error. When an error occurs, the Connection Agreement Wizard still finishes, but the error is reported in an XML file. For example, if the Active Directory Connector service is not running, or isn’t able to be contacted by the ADC Tools’ Connection Agreement Wizard, then it will error-out, and the user must examine the XML output to resolve the issue before starting the Connection Agreement Wizard over again. Some typical errors one might receive are “Constraint Violation” and “ADSI” errors. To troubleshoot these, it is most efficient try again, and obtain a network monitor sniff. After Connection Agreements are created, connection agreements are not immediately listed in the Microsoft Management Console. The user must refresh the view.

Upon creating the connection agreements, Connection Agreement wizard populates the domain controllers with their fully-qualified domain names. The fully-qualified domain name (FQDN) is used to facilitate the Exchange 2003

Page 361: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 67

ADC’s Kerberos requirements. Additionally, the connection agreements no longer allow NTLM (Windows Challenge/Response) authentication, as illustrated in Figure 2.8.

Figure 2.8: Connection Agreement Wizard creates Connection Agreements with fully-qualified domain names. Additionally, Kerberos replaces NTLM as the default authentication method.

In the end, the connection agreements will have “perfect Connection Agreements,” which characteristically export from the Exchange 5.5 site root “OU=”, and export from the domain naming context root “DC=” exporting users, groups, and contacts. “Perfect Connection Agreements” never contain multiple export containers. Connection Agreements start executing as they are created. So if the Connection Agreement Wizard takes several minutes to create the Connection Agreements, the first few Connection Agreements may start replicating before the last set of Connection Agreements are created.

After the connection agreements are created and replicated, the Step 4 VERIFY button needs to be used to perform another pass of adcobjectcheck and ntobjectcheck to ensure that Connection Agreement replication has synced the two directories. Most importantly, clicking the verify button also rewrites the completion marker, ADCUserCheck, onto the description attribute of the Microsoft Exchange object. Upon writing this object, regardless of whether or not there are still unreplicated objects in either directory, Exchange 2003 setup will be able to proceed.

Page 362: Exchange Server Admin Troubleshooting

68 Module 6: Deployment Tools and ADC Tools

Troubleshooting if Verify reports a problem: Although it is optional to troubleshoot, customers may have better confidence in their deployment if they make sure that the adcobjectcheck and NTObjectcheck tests report no errors during verify. The possible reasons that Verify fails may be due to:

1. Not enough time was given for the domain controllers to replicate new objects in Active Directory.

2. Some Connection Agreements were deselected (for example, unchecked from figure 2.7).

3. Credentials entered during the Connection Agreement wizard did not have privileges to read/write to one or both directories.

4. Disconnected site (or remote Exchange 5.5 directory service whose port 389 was blocked).

The solution to the first two causes is simply to wait (perhaps up to several days, depending upon the size of the forest and site link schedules) and then rerun the Verify button. For the third cause, the user should filter for warning/error events from the source MSADC in the application event log. If one or more Connection Agreements have the wrong credentials configured, the MSADC event will show the Connection Agreement’s displayname, and then the administrator may rectify the corresponding Connection Agreement’s “Connections” property sheet with appropriate credentials. For the fourth case, the user should either manually create the Connection Agreement between a domain controller and the Exchange 5.5 directory service at the remote site.

Differences between manual and auto-generated connection agreements: The only attribute that is set by the Connection Agreement wizard that is not changeable by the traditional Connection Agreement -creation UI is the “msExchServer1SearchFilter.” This search filter is modified in such a way that only allows objects from a specific administrative group to replicate over the connection agreement.

Page 363: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 69

ADCTools: Putting the pieces together The process for ADC Tools is not as lengthy as the deployment tools, but it contains more complex conditional scenarios, as illustrated in figure 2.9 below.

Figure 2.9: ADC Tools process flow

Debugging ADC Tools Presently, there are no known issues that would cause the ADC Tools snap-in to hang, so a debugging example is not available. And although it is possible to attach a debugger to the MMC.exe process running the ADC tools snap-in, the debugger may not provide the best information because all the tools were written in Visual Basic.

Nevertheless, should there be a circumstance relating to ADC tools hanging, one may enable ADC Tools debug logging to assist in troubleshooting. In the system properties of the machine running ADC Tools, go to the environment variables option, and create the system variable “DebugADCTools” and set it to a value of 1. Alternatively, type “set DEBUGADCTOOLS=1” at the command prompt and the environment variable is added. Regardless of which method of adding the variable, if the ADC management snap-in is already loaded, you must close and restart it to reflect the change. Afterwards, when any of the ADC Tools are executed, additional logging information will appear in the

Page 364: Exchange Server Admin Troubleshooting

70 Module 6: Deployment Tools and ADC Tools

adctools.log file. The debug output may not make sense unless you have the ADC Tools code right next to you. The code is in DSA\src\deploy\adctools and dsa\src\deploy\walksdll.

Page 365: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 71

Lab 6.1 Exchange 2003 ADC replication featuring Deployment and ADC Tools

Importance: Active Directory Connector (ADC) replication cannot always be described in words. Often, it is easiest to show videos or observe its behavior. Aside from observing ADC replication, students will see the new ADCTools and use it as a troubleshooting tool.

Lab Objectives: At the end of this lesson, students will be able to

Know how to use ADCTools as a troubleshooting tool. Recognize common customer issues involving ADC. Learn the following ADC Behavior:

• Object matching vs. creation.

• Auto-renaming “CN=” values.

• OU-generation by ADC.

• Mismatched accounts.

• How NTDSNoMatch affects assoc-nt-account.

• Rollback steps from Connection Agreement replication.

• Cross-domain matching.

• Perceived “duplicates” created.

Page 366: Exchange Server Admin Troubleshooting

72 Module 6: Deployment Tools and ADC Tools

Exercise 1: Lab preparations 1) Turn on the following virtual machines in this order: LondonDC, Madrid55. When LondonDC shows the login prompt, you may start Dublin55.

Configuration:

• LondonDC – Windows 2003 domain controller of forest root “deptools” domain.

• Dublin55 – Windows NT 4.0 SP6a member server within “Deptools” domain.

• Madrid55 – Windows NT 4.0 SP6a domain controller of domain “NT4domain”.

• The ADC service is installed on LondonDC.

• Exchange 5.5 site called “Site1” consists of only one server installed on Dublin55.

• Exchange 5.5 site called “Site2” consists of only one server installed on Madrid55.

• Both sites are connected via X.400 connectors. 2) On LondonDC, ensure that the Windows 200x support tools are installed. If not, then install them from c:\installation cd\support\tools, and launch suptools.msi. (We will need these tools for future lab exercises.)

3) Familiarize yourself with the topology:

• What domain trusts exist?

• Is directory replication between Exchange 5.5 sites already established?

• How are the display names between Site1 and Site2 different? (Make note for clarity later in the exercise.)

• On which machine can we install Exchange 2003? 4) Use Virtual Server or Virtual PC settings to configure the Exchange 2003 Enterprise Evaluation CD (.ISO) image as a mounted virtual CD within the LondonDC virtual machine. (Your host machine may have the .ISO file located within c:\flats\ISO)

5) After mounting, your virtual machine should pop up the autorun splash screen. If you do not see it, double-click on setup.exe from the root directory of the virtual CD.

6) From the splash screen, choose to run the deployment tools. Follow the instructions as though you were installing Exchange 2003 for the first time.

7) For each tool executed, examine the new files produced under c:\exdeploy logs. Pay particular attention to exdeploy.log, since this file is updated upon each tool execution.

8) When you get to phase 2, step 5 in the help guide, you may skip the installation of the ADC setup, since the Exchange 2003 ADC is already installed. Here, we will temporarily stop using the deployment tools, and temporarily concentrate instead on ADC-specific topics.

Page 367: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 73

Exercise 2a: Observe ADC auto-renaming Before practicing on the ADC and Deployment Tools, it is first important to observe typical ADC behavior that might not be obvious to customers.

1. This lab already has a two-way recipient connection agreement pre-configured on LONDONDC. Open the Active Directory Connector snap-in and examine the connection agreement’s properties.

2. Create an account in Active Directory account called admin. Make it a member of domain administrators. (Do this on LONDONDC.)

• Full Name: admin

• Userlogon name: admin

• OU: OU for Exercises 1-5 DO NOT create a mailbox if the user-creation wizard prompts you. Deselect the checkbox. Finish the wizard, and confirm that “admin” appears under the “Name” column.

3. Use Exchange 5.5 Administrator on DUBLIN55 to create Mindy Martin’s mailbox:

• First name: Mindy

• Last name: Martin

• Display Name: Martin, Mindy

• Alias: MindyM

• Primary Windows NT Account -> select existing acct -> deptool\admin

• Container: “Container for Exercises 1-5”

• Replicate the existing connection agreement. 5. On LONDONDC, refresh Active Directory Users and Computers. Where is

the admin account? Did it get deleted? 6. Open-up the properties of the “Martin, Mindy” account. Go to the account

tab. Note that the logon name hasn’t been affected by ADC replication. Also note that the Exchange General tab has the home server and mailbox alias.

Customers can easily get confused because the ADC not only replicates attributes, but it also modifies the LDAP common/canonical name. (Read 269843 ADC Overwrites Display Name with Exchange Server 5.5 Display Name for information on how to workaround.)

Exercise 2b: ADC-Global-Names comparison

1. Create an Active Directory account for a user:

• Through Active Directory Users and Computers on LONDONDC, navigate to this container: OU for Exercises 1-5

• Create a user account called Steve Knopf.

Note

Page 368: Exchange Server Admin Troubleshooting

74 Module 6: Deployment Tools and ADC Tools

• First name: Steve

• Last name: Knopf

• Full Name: Steve Knopf

• Userlogon name: stevek

• During the wizard, you will be prompted to create the mailbox. Go ahead and let it create it for you.

On DUBLIN55, refresh Exchange 5.5 admin to verify Steve Knopf’s mailbox was created. You may need to refresh the admin program’s view.

2. Open-up ADSIEdit, which is installed on LONDONDC.

Question 1: What effect do the commas have on the LDAP display name of Mindy Martin’s Active Directory account?

Question 2: Compare msexchadcglobalnames attributes between Mindy and Steve’s Active Directory accounts. What is different?

3. Open-up Exchange 5.5 admin program in Raw mode on DUBLIN55. Click Start, then Run, and type the following: c:\exchsrvr\bin\admin.exe /r Go to File -> Raw properties when highlighting Mindy’s and steve’s mailboxes. Compare ADC-global-names attributes between MindyM’s and stevek’s Active Directory accounts. What is different?

4. Delete Steve Knopf’s user account. Right-click on the two-way Connection Agreement and replicate now. Verify that Steve Knopf’s mailbox in Exchange 5.5 administrator disappears.

5. Delete Mindy Martin’s mailbox through Exchange 5.5 administrator. Right-click on the two-way Connection Agreement and replicate now. Why didn’t Mindy Martin’s Active Directory account become deleted?

6. Verify that Mindy Martin’s user account has no more Exchange

attributes/tabs. 7. Optional: create a few more objects in either Exchange 5.5 or Active

Directory, and examine ADC-Global-Names to confirm consistency of observations. Ensure your results are consistent with KB article Q316280.

Page 369: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 75

Exercise 2c: OU creation through the ADC 1. Using admin.exe, create two containers under site1\Container for Exercises

1-5 container called “subcontainer1” and “subcontainer2”. 2. Create a new custom recipient in subcontainer2 called sub2CR. Provide any

external SMTP address. 3. Replicate your Connection Agreement. 4. In active directory users and computers, which container(s) do you see

underneath the “OU for Exercises 1-5” OU? Why don’t we see all of them?

Exercise 2d: When are we prompted to create a mailbox? 1. On the schedule tab of the original two-way Connection Agreement that was

preconfigured for this lab, set the schedule to never. 2. Try to create a user account in the “OU for Exercises 1-5” OU. Are you

prompted to create the mailbox? 3. Set the schedule tab of the original two-way Connection Agreement back to

Always. 4. On the Advanced tab, set the “Primary… to Exchange” to unchecked. 5. Try to create a user account in the “OU for Exercises 1-5” OU. Are you

prompted to create the mailbox? 6. Reverse the previous steps to set the Connection Agreement back to its

original state.

Exercise 3a: Common problem: Mismatched accounts 1. Stop the ADC service. (Click Start, then Run, and then net stop msadc.) 2. Create an account in Active Directory called for our fictitious CFO, Jon

Morris. (Do this on LONDONDC)

• OU: OU for Exercises 1-5

• Full Name should be Jon Morris

• Userlogon name: jmorris

• Provide a complex password. DO NOT create a mailbox through Active Directory Users and Computers user-creation wizard if prompted.

Finish the wizard, and confirm that “Jon Morris” appears under the “Name” column.

3. Create the following Exchange 5.5 mailboxes in this order in the “Container for Exercises 1-5,” beginning with the conference room resource mailbox:

Display Name Alias Primary Windows NT Account

Conference Room conference deptool\jmorris

Jon Morris jmorris deptool\jmorris

Earnings (shared) earnings deptool\jmorris

Page 370: Exchange Server Admin Troubleshooting

76 Module 6: Deployment Tools and ADC Tools

In Exchange 5.5, customers commonly had this configuration so that users could have access to multiple mailboxes. However, this configuration is not 1-to-1, so Exchange 2000/2003 chokes on permissions if the ADC replicates these objects to Active Directory. Let’s go ahead and break the system:

4. Start the ADC service. (Click Start, then Run, and then net start msadc.) 5. Wait up to a minute for the replication cycle, and refresh Active Directory

Users and Computers. (Alternatively, force replication by right-clicking on the recipient connection agreement) Then observe how it appears that the Jon Morris Active Directory account becomes disabled. Open-up his Active Directory account properties, and go to the Exchange General tab. What is the alias? Go to the Account tab. What is the user logon (samaccountname)?

6. What happened with the real Jon Morris account? (Hint: In Active Directory Users and Computers, go to View -> Add/Remove columns. Make the “pre-Windows 2000 logon name” available. Once the view is refreshed, go back into the “Exercises 1-5” container, and check for the existence of the accounts associated/created by the ADC by looking under the newly-added column.)

Short answer: samaccountname remains the same, but the Name, or "cn=” value is renamed because it was matched to the conference room mailbox. Long answer: When the ADC performed an LDAP query against the Exchange 5.5 directory, it starts matching based on the order in which the Exchange 5.5 directory service lists the responses. You can simulate this by opening LDP.exe, connecting to Exchange 5.5, and searching on the value of msexchserver2searchfilter as it is listed on the properties of the connection agreement. Typically the LDAP query filter is (|(objectclass=organizationalPerson)(objectclass=remote-address)(objectclass=groupOfNames)), but value of msexchserver2searchfilter varies depending upon the options the selected on the Connection Agreements “From Exchange” tab. The LDAP response from the Exchange 5.5 directory service is NOT in alphabetical order. In this simple case, the LDAP response is the order in which the objects were created: {Conference Room, Jon Morris, Earnings (shared)}. So from the list of objects returned by the Exchange 5.5 directory service, the ADC queries Active Directory for their associated SIDs. In this simple case, the ADC sees that the first unmatched object (not having a global name) is the conference room mailbox. The ADC parses the SID from the conference room mailbox, and queries Active Directory for any object whose objectSID matches. The matched objectSID happens to be the deptool\jmorris account’s SID. Once matched (through matching rules), the ADC stamps conference room’s attributes onto deptool\jmorris account, and renames its “cn=” value of the x500 distinguished name. The ADC repeats the process with the next Exchange 5.5 object. The next LDAP entry returned from Exchange 5.5 is the actual Jon Morris mailbox. When the ADC tries to match Jon Morris’ mailbox to an Active Directory account, it sees deptool\jmorris has the matching objectSID. But since legacyExchangeDN is populated (from the last match) on deptool\jmorris, the ADC cannot overwrite exchange attributes for this second match attempt. In this case, the ADC must abandon its object matching rules and create a new, placeholder account with a random samaccountname. (In pre-Exchange 2003 versions of the ADC, the placeholder account’s

Page 371: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 77

samaccountname would not be randomized, and so it would have created deptool\jmorris-1 in our example case). Disabled, placeholder accounts can be confusing, and customers tend to believe that the ADC has disabled their existing accounts. So in this case, an administrator might enable the “Jon Morris” placeholder account to achieve parity. Unfortunately, this actually causes more problems (see KB articles Q303180 and Q278966) and confusion for support engineers trying to troubleshoot related problems. Additionally, customers will not realize that their old jmorris account was mismatched and renamed because the pre-Windows 2000 column is not in Active Directory Users and Computers by default. Multiply jmorris by several hundred users having multiple mailboxes, and you can imagine the difficulty in troubleshooting a normal-sized company’s mismatched accounts in their “Exercises 1-5” OU!

When Jon Morris logs onto the domain and Outlook 2000/XP with his DEPTOOL\jmorris account for the first time, which mailbox will he see?

Answer: If Exchange 2000 and 2003 are installed, and Jon Morris logs onto Outlook without generating his profile, Outlook will read the Active Directory account’s mailbox attributes to automatically generate the profile. In this case, Outlook will find that the conference room mailbox has its attributes on DEPTOOL\jmorris account, so Outlook will present our CFO with the contents of the conference room mailbox, rather than his personal (and more important) mailbox. Do we see any application event log events that indicate there was something wrong with replication?

Exercise 3b: Clean-up from ADC mismatching. If there are hundreds of mismatched accounts like Jon Morris’ mailbox, how can we find them and perform a rollback to a state before the ADC replicated the connection agreements?

Three methods to find the mismatches:

a) By observing the pre-Windows 2000 logon name column in Active Directory Users and Computers. However, depending on the number of objects that were stamped in the customer’s Active Directory, this task can be long and tedious. If you don’t enable the extra column, it is not easy to determine groups of Active Directory objects that have been renamed.

b) Using the deployment tools. Running exdeploy.exe with the /adccheck switch will output files, which you can examine to identify which accounts do not have msexchmasteraccountSID.

c) Using the ADC Tools, we can discover accounts with mismatched/missing msExchMasterAccountSIDs. This is the method that we will use in these next steps.

Page 372: Exchange Server Admin Troubleshooting

78 Module 6: Deployment Tools and ADC Tools

1. By glancing at Active Directory Users and Computers, some engineers using method A have deleted either the disabled account or the “conference” account in their attempts to clean up. Do not do this, as the mistakenly deleting these Active Directory accounts causes the ADC to replicate a deletion to the Exchange 5.5 directory service, thereby deleting the corresponding Exchange 5.5 mailbox. Instead, let’s do a clean up by first stopping the ADC service.

Why do we need the ADC service to be stopped while cleaning-up?

Answer: During cleanup, we may need to delete or manipulate objects. If the ADC replicates while we are only halfway finished with cleanup, it can cause objects to become mismatched again, or possibly even become deleted permanently.

What happens if the user deletes the disabled “Jon Morris” account, but not the conference account?

Answer: Jon Morris’ personal mailbox will be deleted, but he will still be able to log on.

More common mistake: What happens if the user deletes the “Conference Room” account? After all, the worst we can do is to just lose an unimportant resource mailbox.

Answer: The conference room mailbox will get deleted from Exchange 5.5. Although Jon Morris’ important personal mail is not lost, his user Active Directory account/SID is gone. What this means is that any other domain resources to which he previously had access are no longer accessible. The CFO cannot even log on anymore. 2. We need to identify which accounts were “damaged” when ADC didn’t find

NTDSNoMatch. Open the Active Directory Connector Management snap-in -> ADC Tools.

3. Set the Exchange server to DUBLIN55, port 389. 4. Run the data collection phase. 5. Examine the adctools.log file. Note any errors, if any. 6. In phase four of four, what articles are mentioned? Open and read those

articles. 7. Examine the badresourcemailboxes.xml file. Does it include all of the

accounts that we need to clean up?

Page 373: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 79

Answer: No, the account on which the ADC performed the initial match is not listed on this list. This is the DEPTOOL\jmorris account (whose display name is Conference Room). In addition to cleaning-up the entries from badresourcemailboxes.xml file, we also need to disassociate the mismatched account – conference room. 8. Stop the ADC Service (net stop msadc). 9. Use Exchange 5.5 admin in RAW mode to view the raw properties for each

of the three Exchange 5.5 mailboxes, and then delete all ADC-Global-Names.

10. You may now safely delete the disabled account “Jon Morris” from Active Directory without the fear of back-replicating a deletion to his Exchange 5.5 mailbox.

Since we delete the “Jon Morris” named account, why doesn’t this prevent our CFO from logging on?

Answer: When we deleted the “Jon Morris” account, we did not delete the SID he uses to log on. Instead, we deleted a random SID that was created by the ADC. As long as the object whose samaccountname==DEPTOOL\jmorris is not deleted, we are safe. 11. Rename the “Conference Room” Active Directory account back to the

Display name “Jon Morris” (Right-click on the Active Directory account and choose rename)

12. The Jon Morris user account still has global names pointing at the conference room mailbox. We still need to disassociate the global name pointer so that the next replication cycle will not be tainted. Either remove the msexchADCglobalnames values manually or use the “Remove Exchange Attributes” feature (documented in article W307350).

Exercise 3c: Associate accounts properly. 1. Open ADC Manager -> ADC Tools 2. Set the Exchange server to DUBLIN55, port 389. 3. Run the data collection phase. 4. Examine the adctools.log file. Note any errors, if any. 5. Run Step2 (Resource Mailbox Wizard). 6. Expand deptool\jmorris and set the appropriate mailbox as primary. (If the

Jmorris mailbox is bolded by default, then it is already set as primary) 7. Click Next, providing credentials and finishing wizard. 8. Examine, in the logging path (c:\documents and settings\administrator\local

settings\Application Data\Microsoft\Active Directory Connector), the ntdsnomatch<sitename>.csv file in notepad or excel. Note the similarities from the Resouce Mailbox Wizard GUI selection.

9. Make sure that NTDSNomatch is on custom attribute 10 of the conference mailbox. When the ADC reads this value present on the conference

Page 374: Exchange Server Admin Troubleshooting

80 Module 6: Deployment Tools and ADC Tools

mailbox, it will NOT match it with any Active Directory accounts, and instead, will create a mailbox-enabled disabled account in Active Directory.

10. Make sure that NTDSNomatch is NOT on custom attribute 10 of Jon Morris’ Exchange 5.5 mailbox. By not having the value present, the ADC will match-up this mailbox with DEPTOOL\jmorris.

11. In Exchange 5.5 admin, examine custom attribute 10 of the Jon Morris and conference mailboxes. Note that the Resource Mailbox Wizard already imported the CSV file for you.

12. Restart the ADC service. The connection agreement will replicate. 13. Refresh Active Directory Users and Computers, and notice how the

disabled-account is now the conferencing resource, while jon morris’ mailbox is associated correctly with his Active Directory account.

14. In Exchange 5.5 admin, open the properties of the conference room mailbox. What changes were made to the Primary Windows NT Account?

Answer: Note that the Primary Windows NT account has been changed to Deptool\conference.

15. Since deptool\jmorris is no longer the Primary Windows NT Account, how can our CFO access his resource mailbox again? (At this point, a customer might be tempted to swap back to DEPTOOL\jmorris. However, no further action is needed. The ADC preserved jmorris’s permission to this mailbox on the Permissions tab. Since the permissions tab is not viewable by default, you can enable it from the Exchange 5.5 Admin menu -> Tools -> Options -> Permissions tab). After enabling the permissions view, confirm that Deptool\jmorris has permissions to the resource mailbox object. Another observation to cite is that the randomization of user logon names will not occur for NTDSNoMatched objects.

Exercise 4: Object matching vs. object creation and other cross-domain replication oddities

1. Create an Exchange 5.5 mailbox called “popeye” on the server Madrid55. (Be sure to create the mailbox in the site2 naming context, and not the site1 naming context) For the Primary Windows NT account, create the nt4domain\popeye user account. Fill-in some other attributes, such as address, city, etc.

2. Create a separate OU in the DEPTOOL domain called “Madrid55 mailbox placeholders”. This OU will contain objects corresponding to the site2’s mailboxes.

3. Create an Active Directory account in your new OU called popeye. Let the full name match the user logon name. Notice how you were not prompted to create a mailbox. This is because your custom OU is not specified on the “From Windows” tab of a connection agreement.

4. Create a two-way recipient connection agreement between the madrid55 site’s Recipients container and the new OU you created in step 3. Name it effectively (i.e. User Connection Agreement - site2\recipients <->

Page 375: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 81

deptool.lab\madrid55 mailbox placeholders) Make sure the connections tab/Exchange Server information section has NT4domain\administrator. Why does this exercise not use DEPTOOL\administrator on the Exchange server information?

Answer: If the Connection Agreement needs to write ADC-Global-Names or create directory objects in the madrid55 site, it would have used DEPTOOL\administrator. Credentials, which have no privilege to read/write to the madrid55’s site naming context, which would have prompted a call to Microsoft Product Support Services stating “Why isn’t my ADC replicating?”

Another common mistake that nearly every customer makes is that they will not use the account browser. Instead, they will type the account name without the domain name prefix. (“Administrator” instead of “NT4domain\administrator”) The connection agreement UI does not check for the validity of the credentials, so “Administrator” would cause the Connection Agreement not to replicate since it is not a domain account.

5. After a replication cycle, do you notice the new disabled account? Why did the ADC create a new account, instead of matching and stamping the existing Popeye account with the site2’s popeye mailbox attributes?

(Hint: Read object matching rules KB article Q303181)

In their attempts to clean-up, would it be wise for customers to get rid of the “duplicate” popeye-1 account?

6. When accounts cannot be matched, you have noticed that disabled placeholder accounts are generated. In this case, the ADC also makes another modification to the Exchange 5.5 mailbox. What do you see on the permissions tab of the popeye mailbox on madrid55?

Exercise 5: Out-of-default destination object matching 1. On madrid55, use Exchange 5.5 admin to create a new mailbox called

remoteadmin. (Be sure to create the mailbox in the site2 naming context, and not the site1 naming context) For the primary Windows NT account, associate this new mailbox with DEPTOOL\administrator.

2. Force the Connection Agreement that you created in Exercise 3, step 4 to replicate.

3. In Active Directory Users and Computers, do you see a new “remoteadmin” object created in your custom OU? Why or why not? The next few steps can help you answer this question “Why.”

Note

Page 376: Exchange Server Admin Troubleshooting

82 Module 6: Deployment Tools and ADC Tools

4. On madrid55, open-up admin.exe in raw mode. Get raw properties on the

remoteadmin mailbox. Go to the Primary Windows NT account attribute. There should be a long security ID. No two SIDs in a forest can be the same; the algorithm for creating SIDs guarantees uniqueness. Although SIDs in a domain nearly always start-off with the same numbers, you will always see variations in the last eight hex numbers in the “attribute value” field. (You may need to scroll to the far right.) Write down the last eight digits.

5. Open-up ADSI Edit on LONDONDC. Locate the DEPTOOL\administrator account and get the properties. Select “mandatory” from the first drop-down list and “objectSID” from the second drop-down list. Scroll to the far right, and compare the last octet from step 4. Note how the syntax is different but the values are the same, especially the last octet where user SIDs usually vary. When the SID in the Active Directory domain matches the SID stored on a mailbox in the Exchange 5.5 directory, you may now conclude that the ADC matches the object in its current OU, and does not need to create an object in your custom OU. Additionally, the ADC does not move the matched object to your custom OU, despite the fact that the custom OU is specified as the “Default Destination” on the connection agreement’s properties.

Exercise 6: Examination of ADC Tools logfiles Now that we’ve become a little more familiar with basic ADC replication, we will use ADC Tools to automatically create connection agreements.

1. Starting from the beginning, re-run Data Collection Wizard. (It never hurts to rerun the wizard over again.)

2. Read the error messages in the information box. 3. Navigate to this directory: C:\Documents and

Settings\Administrator\Application Data\Microsoft\Active Directory Connector (Hint, copy and paste from the “logging path” in Step1 of the ADC Tools Wizard.)

4. Open the ADC Tools logfile, and each XML file. (Note: The XML parser was installed with ADC setup, but is also installed in full installs of Exchange 2003.) In ADCTools.log, you should see a list of mailboxes whose Primary Windows NT accounts have SIDs which could not be matched.

5. Using Exchange 5.5 admin.exe on dublin55, can you determine similarities in the accounts having SIDs that could not be matched as conveyed in ADCTools.log?

Page 377: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 83

7. Compare the ADC Tools snap-in information window output against ADCTools.log. What is different?

8. From the command prompt type “exdeploy.exe /gc:londondc /s:dublin55 /t:adcusercheck” Compare the output file adcusercheck.log against ADCTools.log. Are the log output files complaining about the same accounts?

9. Which command line exdeploy.exe tool switch performs three out of the four passes from ADC Tools’ Data Collection? (Hint: type exdeploy.exe without any parameters.)

10. Re-launch the Exchange 5.5 admin.exe using the /r or –r switch. Examine raw properties of the single account which was mentioned having the unresolveable SID. Why was it unresolveable?

11. View the raw properties (Shift-Enter) of the account. Does the assoc-NT-account have a value? _________ How does the assoc-NT-account of this account differ from the assoc-NT-account of the mailbox named EmptyACCT?

12. Delete the object. 13. In adcobjectcheck.xml, observe the configuration options for planned

connection agreements under the <RecommendedCAs> section. 14. Run the Step 3: Resource Mailbox Wizard. Do not choose the “Export”

button. Make a note of the CSV file(s) produced in the “Logging path” directory.

15. At the “Site credentials” page, why do we only see one domain?

16. When finished importing changes, select the “Verify” button. 17. Why do we still see the unresolved SID error in the information window,

even after deleting the account? Key note: After each configuration change, ADCTools must re-discover the directories via another execution of Data Collection.

18. Rerun Data Collection Wizard. 19. Note that in the logging path, additionally XML files are written per

ADCTools tool session. Note the backups of older XML files from previous runs.

Page 378: Exchange Server Admin Troubleshooting

84 Module 6: Deployment Tools and ADC Tools

20. Rerun Resource Mailbox Wizard. Are there any mailboxes to designate as NTDSNomatch’d? Why or why not?

21. Continue with the remaining ADC Tools before returning to the deployment Tools.

When running Connection Agreement Wizard, the Remote Exchange 5.5 server name must be manually selected or typed.

Immediately after creating Connection Agreements, they will not be visible until you refresh the “Active Directory Connector (LondonDC)” pane.

Exercise 7: Running Exchange 2003 setup. Before proceeding, navigate to the ADCUserCheck marker using ADSIEdit.

1. Within ADSIEdit, select properties of cn=Microsoft Exchange… and go to the description attribute, highlighting the ADCUserCheck

2. Select “Remove” and “OK” and “Apply”. 3. Run Setup, and note the error message block. 4. Exit Exchange 2003 setup. 5. Re-run ADC Tools -> Connection Agreements Wizard’s “Verify” button.

(Alternatively, you may re-run data collection to re-add the completion marker.)

6. Rerun setup. Is Setup still blocked? 7. Optional: Repeat 1-5, but instead of using ADSIEdit or ADC Tools to re-

add the value, use the exdeploy.exe /s:dublin55 /gc:londondc /t:adcusercheck command to add the completion marker on the Microsoft Exchange container. Then simply view the marker again, to see if it has been reapplied with a new timestamp.

8. Proceed through the post-installation steps. This concludes the Lab “Exchange 2003 ADC replication featuring Deployment and ADC Tools”

Note

Page 379: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 85

Learning Measure: Answers are in Appendix B.

How can you identify if an object is first created in the Exchange 5.5 directory or the Active Directory?

a) If ADC-Global-Names is present, it means that the Exchange 5.5 directory sourced the object to Active Directory.

b) If msExchADCGlobalNames exists, it means that the Exchange 5.5 directory sourced the object to Active Directory.

c) If four values exists on a global name, the object was sourced in the opposite directory

d) If two values exist on a global name, the object was sourced from the opposite directory

What are escape characters in the LDAP display name as seen through ADSIEdit or LDP?

a) Underscores (“_”) b) brackets (“< >”) c) commas (“,”) d) backslashes (“\”) e) “dc=”

When are we prompted to create a mailbox upon user account creation? (Check all that apply)

a) If Exchange 2000 or Exchange 2003 is installed in the domain. b) If the machine you are on has Exchange System Manager installed. c) If the Connection Agreement for your OU has “primary for the connected

Windows domain” checkbox checked. d) If the Connection Agreement for your OU has the “primary for the

connected Exchange organization” checkbox checked.

Page 380: Exchange Server Admin Troubleshooting

86 Module 6: Deployment Tools and ADC Tools

Appendix A: Sample log files

Page 381: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 87

Sample logs created by deployment tools process after a complete deployment

Exdeploy.log: #*** Exdeploy began: 05/17/2003 03:28:38 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: DSScopeScan, and OrgNameCheck. + Planning Your Deployment (DSScopeScan) - Exchange 5.5 Directory Configuration Summary (DSConfigSum) DSConfigSum provides summary information about your current topology. Details are logged to dsconfigsum.log. - Total number of servers: 2 - Total number of sites: 2 - Exchange 5.5 Directory Object Summary (DSObjectSum) DSObjectSum provides summary information about the objects available in your Exchange 5.5 organization. - Number of Exchange 5.5 public folders: 7 - Number of Exchange 5.5 distribution lists: 1 - Number of Exchange 5.5 distribution lists with hidden membership: 0 - Number of Exchange 5.5 custom recipients: 0 - Exchange 5.5 Directory User Count (UserCount) UserCount reports the total number of users in each site and the total number of users in the Exchange 5.5 directory. Details are logged to usercount.log. - Total number of users: 12 - Server Version Check (VerCheck) VerCheck verifies if Exchange Server 2003 can be installed into your Exchange organization. Details are logged in vercheck.log. VerCheck completed successfully. - Existing Org Report (OrgReport) OrgReport checks Active Directory for an existing organization. Test completed successfully. An organization was created by ForestPrep. You can rename this organization when you install the first Exchange 2003 server. - Global Catalog Server Version Check (GCVerCheck)

Page 382: Exchange Server Admin Troubleshooting

88 Module 6: Deployment Tools and ADC Tools

GCVerCheck verifies the availability of a global catalog server or domain controller running SP3 or later in the current or adjacent site. Details are logged in gcvercheck.log. GCVerCheck completed successfully. All domain controllers and global catalog servers in the current and adjacent Windows sites are running Windows 2000 SP3 or later. + Cleaning Up the Exchange 5.5 Directory (UserPrep) - Organization and Site Names Check (OrgNameCheck) OrgNameCheck verifies if your Exchange 5.5 organization and site names contain unsupported characters. Details are logged to orgnamecheck.log. Your organization name, or one or more site names, contains unsupported characters. #*** Exdeploy finished: 05/17/2003 03:28:40 ***# #*** Exdeploy began: 05/17/2003 03:32:51 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: PolCheck, and OrgCheck. + Preparing Active Directory for Exchange Server 2003 (OrgPrepCheck) - Organization Readiness Check (OrgCheck) OrgCheck verifies the Exchange extensions to the Active Directory schema, checks the existence and membership of the Exchange Domain Servers group and Exchange Enterprise servers group, and checks that a global catalog server is available in a domain in which DomainPrep has been run. Warning: The Exchange Domain Servers group 'cn=Exchange Domain Servers,cn=Users,DC=deptool,DC=lab' does not contain the local computer 'CN=LONDONDC,OU=Domain Controllers,DC=deptool,DC=lab'. If the local computer is not running Exchange Server 2003, this is not a problem. OrgCheck completed successfully. - Policy Check (PolCheck) PolCheck verifies that the necessary permissions are configured correctly on domain controllers. Details are logged to exdeploy-polcheck.log. PolCheck completed successfully. #*** Exdeploy finished: 05/17/2003 03:32:53 ***# #*** Exdeploy began: 05/17/2003 03:52:46 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: OrgNameCheck, OrgCheck, and PubFoldCheck.

Page 383: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 89

+ Cleaning Up the Exchange 5.5 Directory (UserPrep) - Organization and Site Names Check (OrgNameCheck) OrgNameCheck verifies if your Exchange 5.5 organization and site names contain unsupported characters. Details are logged to orgnamecheck.log. OrgNameCheck completed successfully. + Preparing Active Directory for Exchange Server 2003 (OrgPrepCheck) - Organization Readiness Check (OrgCheck) OrgCheck verifies the Exchange extensions to the Active Directory schema, checks the existence and membership of the Exchange Domain Servers group and Exchange Enterprise servers group, and checks that a global catalog server is available in a domain in which DomainPrep has been run. Warning: The Exchange Domain Servers group 'cn=Exchange Domain Servers,cn=Users,DC=deptool,DC=lab' does not contain the local computer 'CN=LONDONDC,OU=Domain Controllers,DC=deptool,DC=lab'. If the local computer is not running Exchange Server 2003, this is not a problem. OrgCheck completed successfully. - Public Folder DS/IS Check (PubFoldCheck) PubFoldCheck finds and fixes incorrect Access Control Lists (ACLs) on public folders, verifies that every public folder in the Exchange 5.5 public store has a corresponding public folder object in the Exchange 5.5 directory, and removes public folders in the Exchange 5.5 directory that do not have a corresponding object in the Exchange 5.5 public store. PubFoldCheck completed successfully. #*** Exdeploy finished: 05/17/2003 03:52:48 ***# #*** Exdeploy began: 05/17/2003 03:54:08 ***# + Exchange 5.5 Server: dublin55:389 + Global Catalog Server: LONDONDC + Tools run: Exchange installation. + Exchange installation (Tools run during setup) - Organization and Site Names Check (OrgNameCheck) OrgNameCheck verifies if your Exchange 5.5 organization and site names contain unsupported characters. Details are logged to orgnamecheck.log. OrgNameCheck completed successfully. - Organization Readiness Check (OrgCheck)

Page 384: Exchange Server Admin Troubleshooting

90 Module 6: Deployment Tools and ADC Tools

OrgCheck verifies the Exchange extensions to the Active Directory schema, checks the existence and membership of the Exchange Domain Servers group and Exchange Enterprise servers group, and checks that a global catalog server is available in a domain in which DomainPrep has been run. Warning: The Exchange Domain Servers group 'cn=Exchange Domain Servers,cn=Users,DC=deptool,DC=lab' does not contain the local computer 'CN=LONDONDC,OU=Domain Controllers,DC=deptool,DC=lab'. If the local computer is not running Exchange Server 2003, this is not a problem. OrgCheck completed successfully. - Active Directory Connector User Replication Check (ADCUserCheck) ADCUserCheck will verify if it has previously been run as recommended. If this tool has not been run prior to setup, Exchange installation will fail. Warning: ADC Tools detected that some users were not replicated from the Exchange 5.5 directory to Active Directory. - Active Directory Connector Object Replication Check (ADCObjectCheck) ADCObjectCheck will verify if it has previously been run as recommended. Warning: ADCObjectCheck detected that some objects were not replicated from the Exchange 5.5 directory to Active Directory. - Public Folder DS/IS Check (PubFoldCheck) PubFoldCheck finds and fixes incorrect Access Control Lists (ACLs) on public folders, verifies that every public folder in the Exchange 5.5 public store has a corresponding public folder object in the Exchange 5.5 directory, and removes public folders in the Exchange 5.5 directory that do not have a corresponding object in the Exchange 5.5 public store. PubFoldCheck completed successfully. #*** Exdeploy finished: 05/17/2003 03:54:09 ***# #*** Exdeploy began: 05/17/2003 03:54:13 ***# + Exchange 5.5 Server: dublin55:389 + Global Catalog Server: LONDONDC + Tools run: Exchange installation. + Exchange installation (Tools run during setup) - Organization and Site Names Check (OrgNameCheck) OrgNameCheck verifies if your Exchange 5.5 organization and site names contain unsupported characters. Details are logged to orgnamecheck.log.

Page 385: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 91

OrgNameCheck completed successfully. - Organization Readiness Check (OrgCheck) OrgCheck verifies the Exchange extensions to the Active Directory schema, checks the existence and membership of the Exchange Domain Servers group and Exchange Enterprise servers group, and checks that a global catalog server is available in a domain in which DomainPrep has been run. Warning: The Exchange Domain Servers group 'cn=Exchange Domain Servers,cn=Users,DC=deptool,DC=lab' does not contain the local computer 'CN=LONDONDC,OU=Domain Controllers,DC=deptool,DC=lab'. If the local computer is not running Exchange Server 2003, this is not a problem. OrgCheck completed successfully. - Active Directory Connector User Replication Check (ADCUserCheck) ADCUserCheck will verify if it has previously been run as recommended. If this tool has not been run prior to setup, Exchange installation will fail. Warning: ADC Tools detected that some users were not replicated from the Exchange 5.5 directory to Active Directory. - Active Directory Connector Object Replication Check (ADCObjectCheck) ADCObjectCheck will verify if it has previously been run as recommended. Warning: ADCObjectCheck detected that some objects were not replicated from the Exchange 5.5 directory to Active Directory. - Public Folder DS/IS Check (PubFoldCheck) PubFoldCheck finds and fixes incorrect Access Control Lists (ACLs) on public folders, verifies that every public folder in the Exchange 5.5 public store has a corresponding public folder object in the Exchange 5.5 directory, and removes public folders in the Exchange 5.5 directory that do not have a corresponding object in the Exchange 5.5 public store. PubFoldCheck completed successfully. #*** Exdeploy finished: 05/17/2003 03:54:14 ***# #*** Exdeploy began: 05/17/2003 04:28:30 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: ADCConfigCheck. + Completing Deployment

Page 386: Exchange Server Admin Troubleshooting

92 Module 6: Deployment Tools and ADC Tools

- Active Directory Connector Configuration Replication Check (ADCConfigCheck) ADCConfigCheck ensures that Exchange 5.5 directory configuration objects were properly replicated from the Exchange 5.5 directory to Active Directory. Details are logged to adcconfigcheck.log. ADCConfigCheck completed successfully. #*** Exdeploy finished: 05/17/2003 04:28:33 ***# #*** Exdeploy began: 05/17/2003 04:30:47 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: ConfigDSInteg. + Additional tools - Exchange Server 2003 Configuration Object Check (ConfigDSInteg) ConfigDSInteg runs Exchange Server 2003 directory integrity checks on Active Directory configuration objects. Details are logged in e2kdsinteg.log. ConfigDSInteg completed successfully. #*** Exdeploy finished: 05/17/2003 04:31:05 ***# #*** Exdeploy began: 05/17/2003 04:31:45 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: RecipientDSInteg. + Additional tools - Exchange Server 2003 Recipient Object Check (RecipientDSInteg) RecipientDSInteg runs Exchange Server 2003 directory integrity checks on Active Directory mail recipient objects. Details are logged in e2kdsinteg.log. RecipientDSInteg completed successfully. #*** Exdeploy finished: 05/17/2003 04:31:47 ***# #*** Exdeploy began: 05/17/2003 04:31:52 ***# + Exchange 5.5 Server: dublin55 + Global Catalog Server: londondc + Tools run: PrivFoldCheck. + Moving Mailboxes - Private Folder DS/IS Check (PrivFoldCheck)

Page 387: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 93

PrivFoldCheck finds and fixes incorrect Access Control Lists (ACLs) on the private information store, and verifies that every mailbox in the Exchange 5.5 information store has a corresponding object in the Exchange 5.5 directory. PrivFoldCheck completed successfully. #*** Exdeploy finished: 05/17/2003 04:31:53 ***#

dsconfigsum.log: #*** Exchange 5.5 DS Configuration Summary began: 05/17/2003 03:28:40 ***# Site: SITE1-DIRNAME Number of servers: 1 The Key Management server for the Exchange 5.5 site 'SITE1-DIRNAME' is 'DUBLIN55'. Server: DUBLIN55 Version 5.5 (Build 2653.23: Service Pack 4). The public folder store for server 'DUBLIN55' is 'DUBLIN55'. Server 'DUBLIN55' is the directory replication bridgehead for the inbound Exchange 5.5 sites 'ou=SITE2-DIRNAME,o=Microsoft'. Server 'DUBLIN55' is the directory replication bridgehead for the outbound Exchange 5.5 sites 'ou=SITE1-DIRNAME,o=Microsoft'. Site: SITE2-DIRNAME Number of servers: 1 Server: MADRID55 Version 5.5 (Build 2653.23: Service Pack 4). The public folder store for server 'MADRID55' is 'MADRID55'. Server 'MADRID55' is the directory replication bridgehead for the inbound Exchange 5.5 sites 'ou=SITE1-DIRNAME,o=Microsoft'. Server 'MADRID55' is the directory replication bridgehead for the outbound Exchange 5.5 sites 'ou=SITE2-DIRNAME,o=Microsoft'. - Total number of servers: 2 - Total number of sites: 2 #*** Exchange 5.5 DS Configuration Summary finished: 05/17/2003 03:28:40 ***#.

Page 388: Exchange Server Admin Troubleshooting

94 Module 6: Deployment Tools and ADC Tools

Exdeploy-polcheck.log: [21:00:46] #*** Policy Check began: 03/28/2003 21:00:46 ***# [21:00:46] Entering HrMapFileHandle [21:00:46] Leaving HrMapFileHandle [21:00:46] PolicyTest.exe results: This tool will check every domain controller in the local domain to see if the "Manage auditing and security logs" privilege granted to the "Exchange Enterprise Servers" group by DomainPrep has replicated to that DC. If the policy change has not yet replicated to all DCs, then you should avoid making policy changes on any DC that has not received those changes yet. You must have Domain Admin rights to run this tool successfully. If you see an error that says: !! LsaEnumerateAccountRights returned error 5 !! then you don't have permission to open the LSA on the given DC. =============================================== Local domain is "ms.com" (MS) Account is "MS\Exchange Enterprise Servers" ======================== DC = "Z2" In site = "Default-First-Site-Name" Right found: "SeSecurityPrivilege" [21:00:46] Entering HrFindPrintErrorMessage [21:00:46] Leaving HrFindPrintErrorMessage [21:00:46] PolCheck completed successfully. [21:00:46] #*** Policy Check finished: ***#

Page 389: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 95

Exdeploy-progress.log: [03:28:38] ********** Beginning Exchange Deployment Tools ********** [03:28:38] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 03:28:38 05/17/2003 [03:28:38] Entering HrDirPreReq_Initialize [03:28:38] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [03:28:38] Entering HrRegisterAXDLL [03:28:38] Leaving HrRegisterAXDLL [03:28:39] Leaving HrDirPreReq_Initialize [03:28:39] Entering HrRegisterAXDLL [03:28:39] Leaving HrRegisterAXDLL [03:28:39] Entering HrDirPreReq_ConfigInit [03:28:39] Leaving HrDirPreReq_ConfigInit [03:28:39] Entering HrDirPreReq_ObjectInit [03:28:39] Leaving HrDirPreReq_ObjectInit [03:28:39] Entering HrDirPreReq_UserInit [03:28:40] Leaving HrDirPreReq_UserInit [03:28:40] Entering HrDSConfigSum [03:28:40] Leaving HrDSConfigSum [03:28:40] Entering HrDSObjectSum [03:28:40] Leaving HrDSObjectSum [03:28:40] Entering HrUserCount [03:28:40] Leaving HrUserCount [03:28:40] Entering HrVerCheck [03:28:40] VerCheck verifies if Exchange Server 2003 can be installed into your Exchange organization. Details are logged in vercheck.log. [03:28:40] Leaving HrVerCheck [03:28:40] Entering HrOrgReport [03:28:40] Test completed successfully. An organization was created by ForestPrep. You can rename this organization when you install the first Exchange 2003 server. [03:28:40] Leaving HrOrgReport [03:28:40] Entering HrGCVerCheck [03:28:40] GCVerCheck completed successfully. All domain controllers and global catalog servers in the current and adjacent Windows sites are running Windows 2000 SP3 or later. [03:28:40] Leaving HrGCVerCheck [03:28:40] Entering HrOrgNameCheck [03:28:40] Error: Your Exchange 5.5 organization and site names contain unsupported characters. [03:28:40] Leaving HrOrgNameCheck [03:28:40] Entering HrDirPreReq_Terminate [03:28:41] Leaving HrDirPreReq_Terminate [03:32:51] ********** Beginning Exchange Deployment Tools ********** [03:32:51] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 03:32:51 05/17/2003 [03:32:51] Entering HrDirPreReq_Initialize

Page 390: Exchange Server Admin Troubleshooting

96 Module 6: Deployment Tools and ADC Tools

[03:32:51] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [03:32:51] Entering HrRegisterAXDLL [03:32:51] Leaving HrRegisterAXDLL [03:32:51] Leaving HrDirPreReq_Initialize [03:32:51] Entering HrRegisterAXDLL [03:32:51] Leaving HrRegisterAXDLL [03:32:51] Entering HrOrgCheck [03:32:53] Leaving HrOrgCheck [03:32:53] Entering HrRunPolCheck [03:32:53] Entering HrGetDSILog [03:32:53] Leaving HrGetDSILog [03:32:53] Entering HrGetDSILog [03:32:53] Leaving HrGetDSILog [03:32:53] Entering HrMapFileHandle [03:32:53] Leaving HrMapFileHandle [03:32:53] Entering HrFindPrintErrorMessage [03:32:53] Leaving HrFindPrintErrorMessage [03:32:53] Leaving HrRunPolCheck [03:32:53] Entering HrDirPreReq_Terminate [03:32:53] Leaving HrDirPreReq_Terminate [03:52:46] ********** Beginning Exchange Deployment Tools ********** [03:52:46] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 03:52:46 05/17/2003 [03:52:46] Entering HrDirPreReq_Initialize [03:52:46] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [03:52:46] Entering HrRegisterAXDLL [03:52:46] Leaving HrRegisterAXDLL [03:52:46] Leaving HrDirPreReq_Initialize [03:52:46] Entering HrRegisterAXDLL [03:52:46] Leaving HrRegisterAXDLL [03:52:46] Entering HrDirPreReq_ConfigInit [03:52:47] Leaving HrDirPreReq_ConfigInit [03:52:47] Entering HrOrgNameCheck [03:52:47] Leaving HrOrgNameCheck [03:52:47] Entering HrOrgCheck [03:52:47] Leaving HrOrgCheck [03:52:47] Entering HrPubFoldCheck [03:52:47] #*** Public Folder DS/IS Check began: 05/17/2003 03:52:47 Server name: dublin55 ***# [03:52:47] PubFoldCheck completed successfully. [03:52:47] #*** Public Folder DS/IS Check finished: 05/17/2003 03:52:47 ***# [03:52:47] Leaving HrPubFoldCheck [03:52:47] Entering HrDirPreReq_Terminate [03:52:48] Leaving HrDirPreReq_Terminate [04:28:28] ********** Beginning Exchange Deployment Tools ********** [04:28:28] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 04:28:28 05/17/2003 [04:28:28] Entering HrDirPreReq_Initialize

Page 391: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 97

[04:28:28] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [04:28:30] Entering HrRegisterAXDLL [04:28:30] Leaving HrRegisterAXDLL [04:28:30] Leaving HrDirPreReq_Initialize [04:28:30] Entering HrRegisterAXDLL [04:28:31] Leaving HrRegisterAXDLL [04:28:32] Entering HrDirPreReq_ConfigInit [04:28:32] Leaving HrDirPreReq_ConfigInit [04:28:32] Entering HrADCConfigCheck [04:28:33] ADCConfigCheck completed successfully. [04:28:33] Leaving HrADCConfigCheck [04:28:33] Entering HrDirPreReq_Terminate [04:28:33] Leaving HrDirPreReq_Terminate [04:30:47] ********** Beginning Exchange Deployment Tools ********** [04:30:47] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 04:30:47 05/17/2003 [04:30:47] Entering HrDirPreReq_Initialize [04:30:47] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [04:30:47] Entering HrRegisterAXDLL [04:30:47] Leaving HrRegisterAXDLL [04:30:47] Leaving HrDirPreReq_Initialize [04:30:47] Entering HrRegisterAXDLL [04:30:47] Leaving HrRegisterAXDLL [04:30:47] Entering HrE2KDSIntegCfg [04:30:47] #*** Exchange Server 2003 Configuration Object Check began: 05/17/2003 04:30:47 ***# [04:31:05] #*** Exchange Server 2003 Configuration Object Check finished: 05/17/2003 04:31:05 ***# [04:31:05] Leaving HrE2KDSIntegCfg [04:31:05] Entering HrDirPreReq_Terminate [04:31:05] Leaving HrDirPreReq_Terminate [04:31:45] ********** Beginning Exchange Deployment Tools ********** [04:31:45] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 04:31:45 05/17/2003 [04:31:45] Entering HrDirPreReq_Initialize [04:31:45] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [04:31:45] Entering HrRegisterAXDLL [04:31:45] Leaving HrRegisterAXDLL [04:31:45] Leaving HrDirPreReq_Initialize [04:31:45] Entering HrRegisterAXDLL [04:31:45] Leaving HrRegisterAXDLL [04:31:45] Entering HrE2KDSIntegRecip [04:31:45] #*** Exchange Server 2003 Recipient Object Check began: ***# [04:31:47] #*** Exchange Server 2003 Recipient Object Check finished: 05/17/2003 04:31:47 ***# [04:31:47] Leaving HrE2KDSIntegRecip [04:31:47] Entering HrDirPreReq_Terminate

Page 392: Exchange Server Admin Troubleshooting

98 Module 6: Deployment Tools and ADC Tools

[04:31:47] Leaving HrDirPreReq_Terminate [04:31:52] ********** Beginning Exchange Deployment Tools ********** [04:31:52] Starting Exchange 6940 Deployment Tools on Windows 5.2.3790 at 04:31:52 05/17/2003 [04:31:52] Entering HrDirPreReq_Initialize [04:31:52] Init called with Domain Controller londondc and Exchange 5.5 server dublin55. Setup's language ID is 0. ActiveX Path is (null) [04:31:52] Entering HrRegisterAXDLL [04:31:52] Leaving HrRegisterAXDLL [04:31:52] Leaving HrDirPreReq_Initialize [04:31:52] Entering HrRegisterAXDLL [04:31:52] Leaving HrRegisterAXDLL [04:31:52] Entering HrDirPreReq_ConfigInit [04:31:52] Leaving HrDirPreReq_ConfigInit [04:31:52] Entering HrPrivFoldCheckBegin [04:31:52] #*** Private Folder DS/IS Check began: 05/17/2003 04:31:52 Server name: dublin55 ***# [04:31:52] Entering HrPrivFoldCheck(dublin55) [04:31:53] Leaving HrPrivFoldCheck [04:31:53] PrivFoldCheck completed successfully. [04:31:53] #*** Private Folder DS/IS Check finished: dublin55 ***# [04:31:53] Leaving HrPrivFoldCheckBegin [04:31:53] Entering HrDirPreReq_Terminate [04:31:53] Leaving HrDirPreReq_Terminate

Gcvercheck.log #*** GC Version Check began: 05/17/2003 03:28:40 ***# Current Site (Default-First-Site-Name) Global catalog servers running Windows 2000 SP3 or later: LONDONDC #*** GC Version Check finished: 05/17/2003 03:28:40 ***#

Page 393: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 99

Orgnamecheck.log: #*** Organization and Site Names Check began: 05/17/2003 03:28:40 Server name: dublin55 ***# Error: Organization or Site display name 'SITE1 (DisplayName)' in the Exchange 5.5 directory contains a leading space, a trailing space, or one or more of the following unsupported characters: ~`!@#$%^&*()_+={}[]|\:;"'<,>.?/ Error: Organization or Site display name 'SITE2 (DisplayName)' in the Exchange 5.5 directory contains a leading space, a trailing space, or one or more of the following unsupported characters: ~`!@#$%^&*()_+={}[]|\:;"'<,>.?/ #*** Organization and Site Names Check finished: 05/17/2003 03:28:40 *** #*** Organization and Site Names Check began: 05/17/2003 03:52:47 Server name: dublin55 ***# OrgNameCheck completed successfully. #*** Organization and Site Names Check finished: 05/17/2003 03:52:47 *** #*** Organization and Site Names Check began: 05/17/2003 03:54:09 Server name: dublin55:389 ***# OrgNameCheck completed successfully. #*** Organization and Site Names Check finished: 05/17/2003 03:54:09 ***

Usercount.log: #*** Exchange 5.5 DS User Count began: 05/17/2003 03:28:40 ***# Site: SITE1-DIRNAME Number of mailboxes: 11 Site: SITE2-DIRNAME Number of mailboxes: 1 - Total number of users: 12 #*** Exchange 5.5 DS User Count finished: 05/17/2003 03:28:40 ***#

vercheck.log #*** Server Version Check began: 03/28/2003 12:15:29 ***# Site 'HubSite' has an Exchange 5.5 server SP3 or later installed. Server: Z0 Version 5.5 (Build 2653.23: Service Pack 4). Site 'Remote55Site' has an Exchange 5.5 server SP3 or later installed. Server: REMOTE55 Version 5.5 (Build 2653.23: Service Pack 4). #*** Server Version Check finished: 03/28/2003 12:15:29 ***#

Page 394: Exchange Server Admin Troubleshooting

100 Module 6: Deployment Tools and ADC Tools

Adcconfigcheck.log #*** ADCConfigCheck began: 03/29/2003 02:41:42 Server name: z0 ***# Error: Configuration object 'cn=Internet Mail Connector (Z0),cn=Connections,cn=Configuration,ou=HubSite,o=Microsoft' has not replicated to Active Directory. #*** ADCConfigCheck: finished verifying 68 objects: 03/29/2003 02:41:42 ***#

E2kdsinteg.log ********* e2k ds integ began on 03/29/2003 at 02:45:15 (gc = z2, version = 6.5.0) --------> Config results from (objectClass=*) ... << no issues found >> ********* finished verifying 657 objects on 03/29/2003 at 02:46:01 ********* e2k ds integ began on 03/29/2003 at 02:57:58 (gc = z2, version = 6.5.0) --------> Recipient results from (mailNickname=*) ... << no issues found >> ********* finished verifying 22 objects on 03/29/2003 at 02:58:18

Page 395: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 101

Sample logs created by ADC Tools process after a complete deployment:

Adctools.log: Current user is 'Administrator\DEPTOOL' on computer 'LONDONDC'

Pass 1 of 4: Resource Mailbox Scan 05/17/2003 03:33:50

Error: Security identifier (SID) for the associated Windows NT account (Assoc-

NT-Account) for the mailbox 'cn=deletedacct,cn=Recipients,ou=SITE1-

DIRNAME,o=Microsoft' could not be resolved.

Warning: The Data Collection tool found objects that must be marked as resource

mailboxes before they can be replicated to Active Directory. Running the Resource

Mailbox Wizard in Step 3 will resolve these issues.

Pass 2 of 4: Active Directory Connector Object Replication Check 05/17/2003

03:33:51

Matched 'cn=KMSERVER,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft' to

'CN=Administrator,CN=Users,DC=deptool,DC=lab' based on SID.

Matched 'cn=aduser1,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft' to

'CN=aduser1,CN=Users,DC=deptool,DC=lab' based on SID.

Matched 'cn=aduser2,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft' to

'CN=aduser2,CN=Users,DC=deptool,DC=lab' based on SID.

Matched 'cn=resource1,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft' to

'CN=aduser1,CN=Users,DC=deptool,DC=lab' based on SID.

Matched 'cn=resource2,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft' to

'CN=aduser2,CN=Users,DC=deptool,DC=lab' based on SID.

Could not find match to 'cn=deletedacct,cn=Recipients,ou=SITE1-

DIRNAME,o=Microsoft'.

Could not find match to 'cn=xdomain1,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft'.

Could not find match to 'cn=xdomain2,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft'.

Could not find match to 'cn=xdomain3,cn=Recipients,ou=SITE1-DIRNAME,o=Microsoft'.

Could not find match to 'cn=nt4user1,cn=Recipients,ou=SITE2-DIRNAME,o=Microsoft'.

Warning: The Data Collection tool found objects that are not replicated from the

Exchange 5.5 directory to Active Directory. Running the Connection Agreement Wizard

in Step 4 will resolve these issues.

Pass 3 of 4: Active Directory Object Replication Scan 05/17/2003 03:33:52

No mail enabled objects found in Active Directory.

Active Directory Object Replication Scan completed. No unreplicated objects

found.

Pass 4 of 4: Active Directory Unmarked Resource Mailbox Scan 05/17/2003 03:33:55

No mail enabled objects found in Active Directory.

Active Directory Unmarked Resource Mailbox Scan completed. No problems found.

Current user is 'Administrator\DEPTOOL' on computer 'LONDONDC'

Resource Mailbox verify 05/17/2003 03:34:34

Error: Security identifier (SID) for the associated Windows NT account (Assoc-

NT-Account) for the mailbox 'cn=deletedacct,cn=Recipients,ou=SITE1-

DIRNAME,o=Microsoft' could not be resolved.

Page 396: Exchange Server Admin Troubleshooting

102 Module 6: Deployment Tools and ADC Tools

Warning: The staging area domain is not in native mode. Group membership and

security cannot be properly managed unless the staging area is in a native mode

domain

Connection Agreement Wizard is finished. Allow time for the changes to replicate

throughout Active Directory. Then click Verify in Step 4.

Current user is 'Administrator\DEPTOOL' on computer 'LONDONDC'

ADC object replication verify 05/17/2003 03:41:10

Could not find match to 'cn=deletedacct,cn=Recipients,ou=SITE1-

DIRNAME,o=Microsoft'.

Warning: The Exchange 5.5 directory still contains objects that are not

replicated to Active Directory. If you have just run Connection Agreement Wizard,

allow time for directory replication to complete, and then rerun the verification

task in Step 4. Otherwise, run the Connection Agreement Wizard.

Active Directory Object Replication Scan completed. No unreplicated objects

found.

Page 397: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 103

Adcobjectcheck.xml <?xml version="1.0" encoding="utf-16"?>

<ADCObjectCheck exchange55="ozpdc:389" gc="TINETDC" date="200301062103">

<Objects>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=&quot;EX:_O=MSFT_OU=EMSC6B44F55C6B44F55C6B44F55203BF8A3000007&quot;,c

n=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=&quot;EX:_O=MSFT_OU=EMSC6B44F55C6B44F55C6B44F55203BF8A3000008&quot;,c

n=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=EVENTCONFIG_OZPDC4948743F4948743F4948743F182909A6000011,cn=Recipients

,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user1,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F0030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user2,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F1030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user3,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F2030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user4,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F3030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=e55user5,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F4030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=ResourceU,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F0030000</Sid>

</object>

<object site="EMS" primary="true" CAtype="user">

<ldapDN>cn=ResourceG,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<Sid>010500000000000515000000E07B3345D93C346AD3448378F5030000</Sid>

</object>

Page 398: Exchange Server Admin Troubleshooting

104 Module 6: Deployment Tools and ADC Tools

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=OAB VERSION

24948743F4948743F4948743F182909A6000BC1,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=E55USER1-

PF4948743F4948743F4948743F182909A600177A,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=PF-DL-

ACLD4948743F4948743F4948743F182909A600177B,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="group">

<ldapDN>cn=55DL1,cn=Recipients,ou=EMS,o=MSFT</ldapDN>

</object>

<object site="EMS" primary="true" CAtype="folder">

<ldapDN>cn=PF-

DELETEDACCOUNTOWNER4948743F4948743F4948743F182909A600177D,cn=Recipients,ou=EMS,o=MS

FT</ldapDN>

</object>

</Objects>

<RecommendedCAs>

<CA CAtype="user" primary="true" site="EMS" >

<exportContainer>ou=EMS,o=MSFT</exportContainer>

</CA>

<CA CAtype="group" primary="true" site="EMS" >

<exportContainer>ou=EMS,o=MSFT</exportContainer>

</CA>

<CA CAtype="folder" primary="true" site="EMS" >

<exportContainer>ou=EMS,o=MSFT</exportContainer>

</CA>

</RecommendedCAs>

</ADCObjectCheck>

Page 399: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 105

Ntobjectcheck.xml <?xml version="1.0" encoding="utf-16"?>

<NTObjectCheck exchange55="ozpdc:389" gc="TINETDC" date="200301131032">

<Objects>

<object domain="ti" site="EMS" primary="false" CAtype="user">

<ldapDN>CN=billy harris,CN=Users,DC=ti,DC=vm</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<ObjectGuid>56E4DEEBC405214FBDA9A71742E3B38B</ObjectGuid>

<LegacyExchangeDN>/O=MSFT/OU=EMS/cn=Recipients/cn=billyh</LegacyExchangeDN>

<msExchHomeServerName>/O=MSFT/OU=EMS/cn=Configuration/cn=Servers/cn=TINETDC</msE

xchHomeServerName>

</object>

<object domain="ti" site="EMS" primary="false" CAtype="contact">

<ldapDN>CN=william taylor,CN=Users,DC=ti,DC=vm</ldapDN>

<Extension-Attribute-10></Extension-Attribute-10>

<ObjectGuid>BE0E225EBA49854CB889DC3655890BCB</ObjectGuid>

<LegacyExchangeDN>/O=MSFT/OU=EMS/cn=Recipients/cn=williamtaylor</LegacyExchangeD

N>

<msExchHomeServerName></msExchHomeServerName>

</object>

</Objects>

<RecommendedCAs>

<CA CAtype="user" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>ou=EMS,o=MSFT</importContainer>

</CA>

<CA CAtype="contact" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>OU=EMS,O=MSFT</importContainer>

</CA>

<CA CAtype="group" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>ou=EMS,o=MSFT</importContainer>

</CA>

<CA CAtype="folder" primary="false" site="EMS" domain="TI">

<exportContainer>DC=ti,DC=vm</exportContainer>

<importContainer>ou=EMS,o=MSFT</importContainer>

</CA>

</RecommendedCAs>

</NTObjectCheck>

Page 400: Exchange Server Admin Troubleshooting

106 Module 6: Deployment Tools and ADC Tools

Ntdsnomatch.xml <?xml version="1.0" encoding="utf-16"?>

<ntdsnomatch exchange55="dublin55:389" gc="LONDONDC" date="200305170333"

existingextatt10data="false">

<Account sid="010500000000000515000000962C1F5F2DF4160173D1A1FA5D040000"

account="DEPTOOL\aduser1" existingextatt10data="false">

<mailbox primary="true">

<Obj-Class>Mailbox</Obj-Class>

<Display_Name>aduser1</Display_Name>

<Alias_Name>aduser1</Alias_Name>

<Directory_Name>aduser1</Directory_Name>

<Home-Server>DUBLIN55</Home-Server>

<Obj-Container>/o=Microsoft/ou=SITE1-DIRNAME/cn=Recipients</Obj-

Container>

</mailbox>

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>resource1</Display_Name>

<Alias_Name>resource1</Alias_Name>

<Directory_Name>resource1</Directory_Name>

<Home-Server>DUBLIN55</Home-Server>

<Obj-Container>/o=Microsoft/ou=SITE1-DIRNAME/cn=Recipients</Obj-

Container>

</mailbox>

</Account>

<Account sid="010500000000000515000000962C1F5F2DF4160173D1A1FA5E040000"

account="DEPTOOL\aduser2" existingextatt10data="false">

<mailbox primary="true">

<Obj-Class>Mailbox</Obj-Class>

<Display_Name>aduser2</Display_Name>

<Alias_Name>aduser2</Alias_Name>

<Directory_Name>aduser2</Directory_Name>

<Home-Server>DUBLIN55</Home-Server>

<Obj-Container>/o=Microsoft/ou=SITE1-DIRNAME/cn=Recipients</Obj-

Container>

</mailbox>

<mailbox primary="false">

<Obj-Class>Mailbox</Obj-Class>

<Extension-Attribute-10>NTDSNoMatch</Extension-Attribute-10>

<Display_Name>resource2</Display_Name>

<Alias_Name>resource2</Alias_Name>

<Directory_Name>resource2</Directory_Name>

<Home-Server>DUBLIN55</Home-Server>

<Obj-Container>/o=Microsoft/ou=SITE1-DIRNAME/cn=Recipients</Obj-

Container>

</mailbox>

</Account>

</ntdsnomatch>

Page 401: Exchange Server Admin Troubleshooting

Module 6: Deployment Tools and ADC Tools 107

Appendix B: Learning Measure Answers 1. D 2. D 3. A,B,D

Acknowledgments Microsoft Employee d) Vincent Yim e) Harvey Rook, Rafiq El Alami, Neil Shipp, Tejas Patel, Brad Owen

Microsoft Internal Specification f) [Microsoft Corporation. Windows XP Professional Resource Kit, Chapter 6:

Security, MS Press, 2001.]

MSPress books g) [Microsoft Corporation. Windows XP Professional Resource Kit, Chapter 6:

Security, MS Press, 2001.]

Help files (level 100 only) h) [Microsoft Corporation. “To configure a modem for a dial-up connection.”

Help and Support Center, Windows XP]

KB article i) [Microsoft Corporation. “Creating User and Group Reports in Windows.”

Knowledge Base article Q137848, available http:support.microsoft.com.]

Existing Whitepaper j) [Microsoft Corporation. “Best Practices for System Policies in Windows

2000 Networks.” Whitepaper available at http://www.microsoft.com/technet.]

MOC courseware k) [Microsoft Corporation. “Basic Administration of Microsoft Windows

2000.” MOC 2028, Microsoft Training and Certification, 2001.]

Page 402: Exchange Server Admin Troubleshooting

Contents

Overview ................................................................................................................. 1 Lesson 1: Changes to the ADC ............................................................................... 2 Lab 7.1: Troubleshooting LDAP queries .............................................................. 12 Lesson 2: DSAccess Usage and troubleshooting .................................................. 16 Lesson 3: Changes in DSAccess ........................................................................... 28 Lab 7.2: Exomatic tool .......................................................................................... 37 Lesson 4: Other Directory Changes ...................................................................... 38 Lab 7.3: Per-Attribute change troubleshooting ..................................................... 59 Lab 7.4: Post-Setup and SRS replication troubleshooting .................................... 59 Appendix A: Numeric registry keys used by DSAccess and their values ............. 68 Appendix B: Answers to some of the Labs ........................................................... 69 Acknowledgments ................................................................................................. 70

Module 7: Exchange Directory Components

Page 403: Exchange Server Admin Troubleshooting

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, Lotus cc:Mail, Lotus Notes) may be the trademarks of their respective owners.

Page 404: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 1

Overview

For this module, we will discuss the new deployment features available with Exchange Server 2003 and differentiate the deployment process from Exchange 2000.

Topic 1 Changes to the Active Directory Connector (ADC)

Topic 2 DSAccess Usage and Troubleshooting

Topic 3 DSAccess Changes

Topic 4 Other Directory Changes

Prerequisites Experience with troubleshooting ADC and DSAccess. Understanding of the ADCTools/Deployment Tools module

Page 405: Exchange Server Admin Troubleshooting

2 Module 7: Exchange Directory Components

Lesson 1: Changes to the ADC

This topic discusses changes to the Active Directory connector that are not covered in the Deployment and ADC Tools module.

Page 406: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 3

ADC Setup includes the entire Exchange Server 2003 schema

In Exchange 2000, the ADC schema files were a subset of the Exchange 2000 core schema files. So although the ADC’s setup /schemaonly switch extended the schema, customers were required to perform further schema extensions using setup /forestprep. This meant longer lockdown periods for larger customers whose custom applications were sensitive to schema extensions due to the delayed nature of replications and resetting of indexed attributes. (You may refer to KB article 230662 for more information on indexed attributes)

In Exchange 2003, the schema files imported during the installation or upgrade of an Active Directory Connector service are identical to the core Exchange 2003 schema; therefore, the schema is only updated once. So if the Exchange 2003 version of ADC setup detects the existence of the Exchange 2003 schema, then no further schema updates will be applied. On the other hand, if ADC setup detects a schema version below 6870, the Exchange 2003 schema updates will be applied. ADC Setup detects the schema by examining the RangeUpper attribute of this object in Active Directory:

cn=ms-Exch-Schema-Version-Pt,cn=schema,cn=configuration,dc=<DN-of-forest-root-domain>

If a domain controller does not show a value of 6870 for the RangeUpper attribute, the schema extension has not completed. There are no more schema updates beyond RangeUpper, since it is the last entry added to the schema9.ldf file:

dn: CN=ms-Exch-Schema-Version-Pt,<SchemaContainerDN> changetype: modify replace: rangeUpper rangeUpper: 6870

Page 407: Exchange Server Admin Troubleshooting

4 Module 7: Exchange Directory Components

Note: Although Exchange 2003’s ADC setup includes the entire schema, it does not mean it is equal to a setup /forestprep. This is because ADC Setup does not perform many of forestprep’s actions, such as importing the Outlook templates and set access control lists (ACLs) on some Active Directory containers. Additionally, forestprep cleans up some address templates and display specifiers. (The removed display specifiers were Exchange 5.5 classes that were never used nor shown in Active Directory.) Therefore, forestprep is still required if ADC’s setup executable is run first, but customers who follow change-control procedures within large environments will not need to plan for additional administrative lockdowns while waiting for schema changes to replicate, since schema extensions will be skipped upon running forestprep later.

Page 408: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 5

Exchange 2003 ADC upgrades “versionNumber” on connection agreements

When the ADC is upgraded to the Exchange 2003 version, the ADC setup program will not only upgrade the ADC binaries; it will also modify the “versionNumber” attribute on any connection agreements owned by that ADC service.

(To determine what connection agreements are owned by an ADC service, use the Active Directory Connector Services snap-in, and highlight the ADC server indicated by the name “Active Directory Connector (servername)” on the left-hand pane. Its owned connection agreements will be viewable on the right hand pane)

When ADC setup upgrades connection agreements’ versionNumber attribute, the values are set to 16973842. Older ADC management snap-ins (such as Windows ADC and Exchange 2000 Service Pack 3 (SP3) ADC snap-in versions) will not be able to administer these new Connection agreements because they expect the older Major version (versionNumber = 16908296). If an Exchange 2000 or Windows 2000 ADC manager snap-in is used to administer an upgraded or new Exchange 2003 connection agreement, this warning is displayed:

Page 409: Exchange Server Admin Troubleshooting

6 Module 7: Exchange Directory Components

Figure 1.1: Version-incompatibility message when using the Exchange 2000 ADC snap-in to administer an Exchange 2003 connection agreement.

By the same token, whenever an Exchange 2003 ADC Services snap-in is used to open the properties of an Exchange 2000 or Windows 2000 connection agreement, the same popup warning as in Figure 1.1 will appear.

The reasons for increasing the major versions on Public Folder Connection agreements and Recipient Connection agreements are so that:

Windows 2000 ADC services will not be able to run any newer Connection agreements. (Any public folder Controller Agreement re-homed to the Windows 2000 version of the ADC service caused corruption.)

The new connection agreements use Kerberos for authentication, which are not understood by Exchange 2000 ADC services.

In summary, an Exchange 2000 ADC service cannot run a connection agreement whose version is incompatible with its own. Conversely, an Exchange 2003 service cannot run a connection agreement whose version number is below 16973842.

Eventually, all ADC services must be upgraded prior to the installation of the first Exchange 2003 server. Otherwise, Exchange 2003 setup may not proceed. Customers must in-place upgrade all pre-Exchange 2003 ADC services prior to installation, so that all legacy connection agreements are phased out.

Page 410: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 7

ADC randomizes user logon names (Integrated CleanSAM functionality)

In the situation where an Exchange 5.5 object exists, but its primary Windows account (assoc-NT-account) resides in a Windows NT 4 domain or in a separate forest, a properly-configured connection agreement directs the ADC service to perform object-creation. (By contrast, if the mailbox’s “Primary Windows NT Account” pre-existed within the forest, the ADC performs “object matching” and stamps pre-existing user accounts during the initial replication cycle.) In the object-creation case, a disabled account is created by the ADC service. In the past, the Exchange 2000 ADC services would generate the disabled security principal (a.k.a. “samaccountname” or “Pre-Windows 2000 logon name”) that matched the Exchange 5.5 object’s alias name. This caused problems for a couple of reasons:

Customers often had the misunderstanding that ADC object creation was an easy way to migrate Windows NT 4 accounts to Active Directory. Although it was not proper, customers would simply enable these “placeholder” accounts that were generated by the ADC, not knowing that this will cause delegation problems, Public Folder ACL conversion problems, and other permissions problems that may prevent logon or mailbox moves. (For more information regarding problems caused by enabling placeholder accounts, see Knowledge Base articles Q300346 and Q316047.)

ADC-generated objects conflict with the Active Directory Migration Tool’s (ADMT) ability to migrate user logon names from their source domains. (This situation only applies if ADMT is used after initial ADC replication, and if aliasname=userlogonname of the source domain). So when ADMT attempts to create user objects in the target domain, it encounters conflicts with the ADC-generated accounts. ADMT was designed to resolve these conflicts by appending “-1” to each samaccountname it generates – thus satisfying the samaccountname uniqueness within a domain. Although ADMT is a proper and supported migration method for user accounts, the “-

Page 411: Exchange Server Admin Troubleshooting

8 Module 7: Exchange Directory Components

1” object causes an issue for customers because their users prefer not to append a “-1” to their logon process. One may believe that ADClean may be used to merge the two objects into a single account, thereby resolving this issue. However, ADClean excludes transferring samaccountname when it merges the disabled objects’ attributes to the ADMT-generated account. In the end, users are still stuck with different user logon names (i.e. User was accustomed to logging onto source domain as “johnsmith,” but must now logon as “johnsmith-1”).

In Exchange 2003, by randomizing samaccountnames (a.k.a. Pre-Windows 2000 logon name) whenever the ADC generates a placeholder object, both previous problem scenarios are resolved. A typical user logon name for an ADC-generated account would be “ADC_BDZQOKNUIZDWPPHG” where the characters following the underscore are always randomized. Since this random username is difficult to use for any logon prompt, it detracts administrators from improperly enabling the placeholder accounts generated by the ADC. Secondly, the prepared random name will not cause naming conflicts when the future ADMT migrations try to create new users in the Exchange 2003 forest.

Although the Exchange 2003 ADC corrects this issue upon object creation, any existing objects that were created prior to an ADC was upgraded may still need their account names renamed. Cleansam.vbs, a script used by Microsoft Product Support Services to correct the above issues for Exchange 2000 topologies, may be used against accounts residing in Exchange 2003 environments that were upgraded from Exchange 2000. The script may be obtained from a Microsoft Product Support Services Professional. CleanSAM also resolved the behavior where in some instances, ADMT would “match” with the disabled accounts and subsequently merge on top of them, thereby enabling the accounts but failing to clear the msExchMasterAccountSID attribute.

Note

Page 412: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 9

New ADC uses Kerberos and signed LDAP

Connection agreements no longer use the configurable option “Windows NT Challenge/Response” (which is essentially NTLM) for authentication to domain controllers. Instead, the Exchange 2003 ADC uses Kerberos for authentication. This change reflects our “more secure” initiative, since

The ADC contains many valuable passwords, such as domain admin or even enterprise admin-level of privileges.

NTLM and unsigned LDAP are susceptible to replay attacks. This change only affects the ADC server during its communication with a Windows 2000 SP3 domain controller or later. Figure 1.2 illustrates the hard-coded changes to the ADC manager.

Page 413: Exchange Server Admin Troubleshooting

10 Module 7: Exchange Directory Components

Figure 1.2: The Exchange 2003 ADC uses Kerberos against Windows 2000 SP3 or later domain controllers.

Note that in the figure above, Exchange 5.5 LDAP communication remains at the down-level Windows NT Challenge/Response authentication mechanism. Only Windows 2000 SP3 or Windows 2003 domain controllers support signed LDAP on connection agreements.

Lesson 4 discusses how the ADC Microsoft Management Console (MMC) and other admin components use Kerberos with signing, and how to troubleshoot decrypt the data over the wire.

Page 414: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 11

Troubleshooting ADC Setup:

When extending the schema, the installation wizard will call SCRunLDIFScript, which is the wrapper function that invokes ldifde to import any .LDF files from the ADC\i386 folder. If ADC setup fails, the ldif.err or ldif.log file may report reasons or other diagnostic information and can be found in the “local settings\temp” folder of the installer’s profile. If ldif.log/ldif.err cannot be found, another troubleshooting step is to manually execute the import process that ADC setup tries to run. Even if you are not able to find the specific schema ldif file to import, you can always test importing the trial.ldf file (ldifde –i –f trial.ldf), as it is a good test of permissions.

If neither manual import nor examination of the ldif.err or ldif.log file leads you closer towards resolution, examine the Active Directory Connector Setup.log file, which is located on the root of the system drive. This file will detail each step of the ADC installation process, such as from importing the 10 schema?.ldf files, to the registering of the ADC service into the registry hive.

ADC Setup uses the same Backoffice setup engine as Exchange 2000 and Exchange 2003, to record the history of setup sessions to the “Active Directory Connector Setup.log” at the root of the system drive. As such, the logparser.exe utility may be used to navigate through various ADC setup sessions.

Page 415: Exchange Server Admin Troubleshooting

12 Module 7: Exchange Directory Components

Lab 7.1: Troubleshooting LDAP queries

Importance: You will learn how to view Lightweight Directory Access Protocol (LDAP) queries in Exchange 2003, since they are signed and encrypted by default.

Lesson Objectives: At the end of this lesson, students will be able to

Configure debug LDAP setting Use network monitor to view packets over the wire

Make sure that all virtual machines are powered-off. If prompted to commit/discard changes, PLEASE CHOOSE DISCARD!

If you have insufficient memory that requires the virtual machines to exist on two host machines, connect a crossover cable with a partner’s machine, then on the left student machine, start Z2, then remote55. On the right student machine, power-on Z6. When Z2 has completely started, power on Z0 on the left machine and Z3 on the right machine. You may then set each of the machines’ virtual network interfaces to “External” so that virtual machines running on each student machine may communicate with one another. Alternatively, you may run all virtual machines on a single host machine if you tune the memory settings for each virtual machine to a lower number. For example, change Z3 to 160 MB of RAM usage instead of 192 MB.

Page 416: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 13

Exercise 1: Creating a new admin account 1. Log onto Z2 as ms\administrator. 2. Open Active Directory Users and Computers, and use the administrator

account as a template to COPY another user. Name the new user “Admin2”. 3. Open Exchange System Manager, and delegate Admin2 Exchange Full

Administrator at the Organization Level. 4. Log onto Z3 (Exchange 2003) as “Admin2” 5. Verify that you can see the queues node of Z3. 6. Then, highlight server object, and then switch back to Z2’s virtual machine.

Exercise 2: Simulate customer problem: Incorrectly “Locking down” Exchange

Sometimes, customers believe that securing their Active Directory involves restricting access directly on Active Directory objects. This is not the recommended method, and we will see what happens when the modifying a single ACL can cause problems with applications such as Exchange. Although the exact steps below have not been followed by a customer, the following scenario is sufficient to create a cryptic problem that may have been difficult to troubleshoot:

1. Open ADSIEdit.msc. 2. Go to the properties of the “Routing Groups” container underneath hubsite. 3. Select the “Security” tab. 4. Add a Deny access control entry (ACE) under “Full Control” for Admin2.

Apply changes and quit the dialog.

Exercise 3: Troubleshooting using network monitor 1. Netmon is already installed, so go to Start -> Run, and type netmon, and

then click OK. 2. A netmon folder will appear, so double-click on netmon.exe. 3. For the interface, choose the network whose linkspeed is not 0. 4. Get ready to start the capture. Timing is crucial if you want a clean netmon

capture, so read the next three steps to see what you will be doing. 5. Start netmon capture, then immediately switch to Z3 and have Admin2 click

back onto the “Queues” node in Exchange System Manager. 6. You should have received a popup error. As soon as it appears, make sure

ms\administrator stops the netmon capture on Z2. 7. View the capture, and you should not be able to determine what calls are

being made over TCP port 389. (Lesson 4 discusses how the admin components use kerberos with signing, and how to troubleshoot decrypt the data over the wire.)

Page 417: Exchange Server Admin Troubleshooting

14 Module 7: Exchange Directory Components

Exercise 4: Setting debugLDAP regkey 1. On Z3, open the registry editor. 2. Navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange. 3. Create a DWORD called DebugLDAP, and assign it a value of 1. 4. Restart Exchange System Manager so that it can pick up the registry change. 5. Start netmon capture, repeating steps 4-6 in the last exercise. 6. When you’ve stopped the netmon capture, examine it. The TCP traffic over

port 389 should have readable data now.

Page 418: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 15

Page 419: Exchange Server Admin Troubleshooting

16 Module 7: Exchange Directory Components

Lesson 2: DSAccess Usage and troubleshooting

This topic discusses DSAccess as it is applicable to both Exchange 2000 and Exchange 2003. The changes specific to Exchange 2003 are discussed in the next topic. This topic is an attempt to put all DSAccess knowledge needed for everyday operation, support, fixing and troubleshooting.

Page 420: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 17

What is DSAccess?

DSAccess is a common Exchange component used for interactions with Active Directory. DSAccess is loaded by most Exchange processes:

inetinfo.exe (or IIS worker processes in Exchange 2003) mad.exe store.exe winmgmt.exe emsmta.exe

It is worth mentioning that DSAccess is not the only Active Directory interface for Exchange. Transport components use their own Active Directory access mechanism, but it uses the list of domain controllers/global catalogs that DSAccess discovers. Setup and System Attendant typically use ADSI instead of DSAccess.

Exchange components use DSAccess to store and retrieve user information as well as Exchange configuration data.

Page 421: Exchange Server Admin Troubleshooting

18 Module 7: Exchange Directory Components

How DSAccess works

In order to read, and especially write to the Active Directory, Exchange needs special permissions. These permissions are given by setup during the forestprep and domainprep stage to the Exchange Enterprise Servers group and Exchange Domain Server group. During the “Core” setup, the Exchange server’s machine account is made a member of Exchange Domain Servers group, and that group is a member of Exchange Enterprise Servers group. All Exchange services run as localsystem (even the Exchange App Pool on IIS6 runs as localsystem) and thus they access to the Active Directory under the machine account.

Along with the “regular” ACL permissions setup grants all Exchange boxes a “SACL right” – a permission to view ntSecurityDescriptor attribute. This is granted via “SeSecurityPrivilege” privilege.

DSAccess recognizes three types of Active Directory servers:

1. Global catalogs 2. Domain controllers 3. Configuration Domain Controller – a single server (chosen from the list of

domain controllers) that is used to read and write configuration data. At any given moment all DSAccess processes use the same ConfigDC, so that the services do not have to wait for replication of the configuration data.

All these types of servers can be located by DSAccess automatically or manually set in Exchange System Manager’s “Directory Access” tab.

Every 15 minutes DSAccess discovers the Active Directory topology and refreshes the list of Active Directory servers and their properties. DSAccess locates all domain controllers in the current site and the closest sites (in terms of the link cost). Then it applies some checks and determines domain controller properties. The complete table of domain controller properties at a given point in time is logged in the event 2080. Here is a sample event 2080:

Permissions

Types of Active Directory servers used

Topology

Page 422: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 19

Event Type: Information Event Source: MSExchangeDSAccess Event Category: Topology Event ID: 2080 Date: 9/24/2002 Time: 9:57:31 AM User: N/A Computer: TENERIFE01 Description: Process STORE.EXE (PID=2072). DSAccess has discovered the following servers with the following characteristics: (Server name | Roles | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version) In-site: csimison201.cmsdom.extest.microsoft.com CDG 7 7 1 1 1 1 7 1 csimison202.cmsdom.extest.microsoft.com CDG 7 7 1 0 1 1 7 1 Out-of-site: csimison200.cmsdom.extest.microsoft.com CDG 7 7 1 0 1 1 7 1 csimison204.cmsdom.extest.microsoft.com CD- 6 6 0 0 1 1 6 1

As you see, this event lists all domain controllers and global catalogs and their properties. Let’s look at them one by one:

Property Value Meaning Server name String DNS name of the domain controller

Roles 3 characters:

C means server is able to act as a Configuration domain controller

D means that a server is able to act as a domain controller

G means that the server is able to act as a global catalog

Reachability Bitmask 0-7 If bit 1 is set, then the server responds to a TCP ping to the global catalog port 3268. If bits 2 and 3 are set then the server responds to a TCP ping to the domain controller port 389. Thus, if a server is a domain controller only the value will be 6, if it is a domain controller and a global catalog, the value will be 7. If a server is disconnected from network, its reachability will be 0.

Synchronized Bitmask 0-7 Bitmask values are the same as reachability, but it reports the synchronization status of a server.

GC capable Boolean Value is 1 if the server is a valid global catalog. Invalid if global catalog is unreachable, in which case value would be 0.

PDC Boolean Value is 1 if the server acts in the primary domain controller (PDC)

Page 423: Exchange Server Admin Troubleshooting

20 Module 7: Exchange Directory Components

role (only if MinUserDCs regkey is set)

SACL right Boolean Value is 1 if the Exchange server has permission to read SACL on the domain controller.

Critical Data Boolean Value is 1 if the domain controllerhas the Exchange server name in the “Microsoft Exchange” container

Netlogon Bitmask 0-7 Bitmask values are the same as reachability, but they report the success or failure of the DsGetDcName doing the RPC directly to the domain controller (which tests Netlogon service on the domain controller)

OS Version Boolean 1 if the operating system build is good enough (DSAccess Exchange Server 2003 only talks to domain controller that have Windows 2000 SP3 and above).

If any of these properties is 0 (except PDC, which is optional), DSAccess will not use these domain controllers. (If the appropriate bits in the bitmask are 0, DSAccess will not use the domain controller in certain role). For example, in the event above the server csimison204 is not a global catalog, and therefore its reachability is 6, not 7 (the last bit is set to 0). This means that it may be used as a domain controller or Config DC, but cannot be used as a global catalog.

Connection pool Since all services use just one user account (localsystem), it is possible for DSAccess to reuse the LDAP connections from one request to another. Upon the startup or topology change DSAccess opens LDAP connections to all suitable domain controllers/global catalogs in topology. By default DSAccess will use up to ten domain controllers and up to ten global catalogs at the same time. The registry parameter that dictates this behavior is ldapconnections (Appendix B).

Failover Whenever WLDAP32 returns an error on an LDAP operation, DSAccess analyzes it and determines if it means that the domain controller is having problems. If so, the domain controller is immediately marked as unsuitable and the user operation is automatically retried on a different server. This way, as long as there is at least one working domain controller in the topology, DSAccess can continue flawless operation.

DSAccess topology always contains two lists: in-site and out-of-site. Initially DSAccess only uses servers from the local site, but when all local servers from certain category (for example, GCs) are not suitable, DSAccess immediately starts using the out-of-site list and logs event 2084 or 2085. After that it keeps checking the local site every five minutes and fails back as soon as it is possible. The user requests are retried on the out-of-site servers immediately and seamlessly for the users. If a DSAccess caller placed a notification, and the target server was marked as

Page 424: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 21

being down, the notification gets reissued and the client is notified of a change, because the monitored value could have changed while we were reissuing the search.

Page 425: Exchange Server Admin Troubleshooting

22 Module 7: Exchange Directory Components

Versioning DSAccess.dll is the main DLL that implements DSAccess, but it has to have matching dscmsg.dll and dscperf.dll that store event messages and perfcounters respectively. So whenever dsaccess.dll is replaced – either through a hotfix application or through a service pack - it’s better to replace all three DLLs at once. Replacing dscperf.dll is optional but doing so requires a matching dscperf.ini and dscperf.h; in turn, a hotfix or service pack update will need to unload the existing counters by running “unlodctr MsExchangeDSAccess” and then “lodctr dscperf.ini”.

: Do not replace these DLLs unless a hotfix package fails to apply the new binaries, or if many DSaccess-related performance counter warnings/errors flood the application event log (such as when a service pack install could not properly unload counters).

Troubleshooting Tip

Page 426: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 23

Troubleshooting sequence

Get and examine the eventlogs The event log is the easiest and best way to troubleshoot DSAccess. The system log may contain the information about the system-wide events, like being out of memory or not being able to contact domain controllers (which obviously causes DSAccess problems not apparent to customers). The application log often contains DSAccess events that explain exactly what went wrong. By default only the most serious issues affecting the mail flow are logged. If the issue is periodic, increase the logging level and try to reproduce it – eventlog will contain lots of details. In a new deployment, or if a customer has any kind of trouble that might be related to the Active Directory, it is suggested to run with diagnostic logging categories Topology, LDAP and Config at level “Minimum” (or above). This way every 15 minutes you will get a report of the topology state, plus all the details for each server.

See “Topology” section above for the details on the 2080 event. It is the first thing to look at when DSAccess has any problems.

When analyzing topology event 2080 make sure that the Active Directory topology DSAccess is using corresponds to Customer expectations.

When looking at the LDAP category events, looking for reports on LDAP protocol errors like 0x34, 0x51 and 0x52. (A list of these errors are referenced at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/netdir/ldap/return_values.asp

Try to correlate these errors in the log or other network events. Try to find out if they are periodic (i.e. logged every x number of minutes).

If the logging is turned on and events 2080 are logged, check that all events are logged in 15 minute intervals. If there is more than 15 minutes between two events 2080, it may mean a DNS or network issue during topology discovery,

Page 427: Exchange Server Admin Troubleshooting

24 Module 7: Exchange Directory Components

for example, inability to get an IP address or ping certain domain controller. Other events (and their time) should help to pinpoint the root cause.

If the event 2080 is not logged, look for other events, like 2114 that prevented topology discovery from happening. Look up the error in that event. For example, error 0x80040920 is a DSAccess way to report LDAP error 0x20 – LDAP_NO_SUCH_OBJECT. It means that DSAccess did not find any Active Directory servers in the local site or closest sites – probably due to the permission problem or misconfiguration. As with any failed searches, the best way to find out what is happening is to repeat the same search in the LDP utility (see next paragraph).

New in Exchange Server 2003, DSAccess also includes a set of DNS troubleshooting events.

Check permissions If some searches return unexpected results (missing attributes or no results at all), permissions are the first thing to check.

First, check if the Exchange server you are having problems with belongs to this domain’s Exchange Domain Servers group, and that group is a member of Exchange Enterprise Server groups in this and other domains (check “member” attribute to find that out).

If there is a particular object that has these problems, check “inherit permission” flag on the object itself and all levels walking up

For Exchange 2003, check if the Exchange domain servers group has been removed from the Pre-Windows 2000 compatible Access group to ensure that Exchange 2003 servers have the cross-domain access they need (222420 and 226063), as customers might have removed the domain servers group from the pre-W2kcompat group membership after misinterpreting the domainprep warning as “unsecure.” If the domain servers group is missing, you can rerun domainprep.

Dump the Windows NT Security Descriptor of the “broken” object and of the closest “working” object and compare them. Look for an ACL that could have broken things. Exchdump/Exchutil is a great utility that can dump ACLs on Active Directory objects.

If this is Exchange 2000, run the policytest.exe tool that ships on the Exchange 2000 CD (in “Support Tools”) to verify that “Manage auditing and security logs” privilege is granted to the Exchange Enterprise Servers group and replicated to all domain controllers in the local domain. If using Exchange 2003, policytest has been replaced by a deployment tool. So on the Exchange 2003 CD, run exdeploy.exe /t:polcheck to achieve the same results. The tool should find the policy that grants Exchange servers the ability to read SACL on all Domain Controllers that have been domainprep’ed.

The ultimate permission test is to install the Windows support tools on the Exchange server console and launch LDP.exe using “at /interactive” command. This way LDP will be running under localsystem account and the results will exactly match what Exchange services see. While doing that, it is often helpful to turn on auditing on a domain controller. (HOW TO: Audit Active Directory Objects in Windows 2000, http://support.microsoft.com/?id=314955) Be sure to use the same domain controller/global catalog as the one that DSAccess used at the time of failure (this information is usually present in the events or traces relating to the specific object which could not be accessed).

Page 428: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 25

The existing permission model for Exchange 2000 has some defects. For example, you cannot read certain attributes for an object that lives in domain A from a global catalog that belongs to domain B. Although most of the attributes are “public information” there are a few attributes that are not public and therefore not visible:

msExchOmaAdminWirelessEnable; modifyTimeStamp; whenCreated; adminDIsplayName; GarbageCollPeriod; msExchMailboxFolderSet; msExchRequireAuthToSendTo networkAddress; versionNumber

Exchange 2003 domainprep fixes this permission model (discussed in the Exchange 2003 Setup changes module). If the problem reoccurs, rerun domainprep.

Check KB or existing bugs It is possible that there is a known problem or maybe even a hotfix available.

Get the traces (regtrace.exe) Even though the eventlog is very informative, the traces contain the most detailed and precise information. DSAccess uses regtrace – a common tracing mechanism used by multiple Exchange components. You can turn it on and off without restarting a service. See Appendix A for details on how to get useful traces.

Note

Page 429: Exchange Server Admin Troubleshooting

26 Module 7: Exchange Directory Components

DSAccess events

DSAccess has a rich eventing system designed for easy troubleshooting. It should be possible to diagnose a problem by looking at the eventlog only.

The events are logged under the source name “MSExchangeDSAccess” and are divided into five categories:

General Cache Configuration Topology LDAP

The verbosity of events in each category can be set independently, and by default only the most critical events are logged (which corresponds to level “None”). It is easy to change the level for any category by launching Exchange System Manager, right-clicking on the server, selecting Properties and going to the “Diagnostics logging” tab. When any Active Directory-related malfunction is happening, it is a good idea to increase Configuration, Topology and LDAP to Minimum or Maximum.

Chart 3.1 contains a list of all new Exchange 2003 events.

Page 430: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 27

DSAccess Tips:

When logging level is increased to troubleshoot topology problems, it is not necessary to wait 15 minutes for the next topology discovery. It is sufficient to create any subkey with a bogus name in MsExchangeDSAccess\Profiles\Default key (which is the same as adding a server through the “Directory Access” tab), and the topology will be rediscovered immediately.

DSAccess reads updated registry values immediately, and then ignores all registry settings for 15 seconds. It means that if you are inputting values slowly they may be partially read by DSAccess as you type, and the latest changes have been ignored (not forever, just for the next 15 seconds). However, if you wait 15 seconds before submitting the last change, you are guaranteed that all changes will be picked up immediately. For this reason, it is best to import a .regfile, or even better: using the Directory Access tab.

If Exchange System Manager cannot be launched (for example, if System Attendant service won’t start) logging levels can be changed directly in registry. They are located in MsExchangeDSAccess\Diagnostics Logging regkey.

Page 431: Exchange Server Admin Troubleshooting

28 Module 7: Exchange Directory Components

Lesson 3: Changes in DSAccess

Although the behavior of DSAccess has not changed, its logging abilities have been greatly enhanced in Exchange 2003. This topic discusses the new event log entries logged by DSAccess, as well as its new performance monitor counters developed for Microsoft Operations Manager (MOM).

DSAccess includes the following new or modified eventlog entries. A severity “W” or “E” would indicate a warning or error event, respectively. A level “7” means that the event will only be logged if the highest diagnostics logging level is set. In contrast, a level “0” means that no diagnostics logging needs to be set before the event is written to the application event log. There are three categories in which new data appear:

General Category

Event ID Problem Event text: problem description and actions required

Data in event Severity Level Periodic

DSC_EVENT_IMPERSONATED_CALLER Process name

2117 PID

Account name

Impersonated thread called into DSAccess

Impersonated account %3 is calling into DSAccess with the following callstack:

Callstack

W 7

New Events in DS Access

Page 432: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 29

Topology Category

Event ID Problem Event text: problem description and actions required Data in event

Severity Level Periodic

DSC_EVENT_BAD_OS_VERSION The Domain Controller %3 is running %4 %5. DSAccess requires that

Process name

2116 Domain Controllers that run Windows 2000 have at least Service Pack 3 installed.

PID

DC Name

OS version

Domain controller is running Windows 2000 SP2 or below.

SP version

W 0 Yes

DSC_EVENT_CDC_CHANGED Process name

2095 PID

Old CDC name

ConfigDC had been changed

The Configuration Domain Controller has been changed from … to …

New CDC name

I 1 Yes

DSC_EVENT_DISCOVERED_SERVERS

DsAccess has discovered the following servers with the following characteristics:

Process name

2080 (Server name | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)

PID

In-site: Site name

Every time DSAccess discovers servers, it dump the whole list

Out-of-site: List of servers with suitabilities

I 1

DSC_EVENT_DISCOVERY_FAILED Process name

2114 PID

Topo discovery could not generate a topology

Topology Discovery failed, error 0x%3.

Error code

E 0

DSC_EVENT_DNS_DIAG_SERVER_FAILURE

Error DNS_ERROR_RCODE_ Process name

2118 SERVER_FAILURE PID

(0x%3) occurred when DNS was queried for the service location (SRV) resource record used to locate a domain controller for domain %4%n%n

Error code

DNS server configured incorrectly.

The query was for the SRV record for %5%n%n Domain name

E 1 Yes

Page 433: Exchange Server Admin Troubleshooting

30 Module 7: Exchange Directory Components

Common causes of this error include the following:%n%n DNS path

The DNS servers used by this computer contain incorrect root hints.

Addresses of DNS servers used

This computer is configured to use DNS servers with following IP addresses:%n%n%6%n

List of DNS zones

One or more of the following zones contains incorrect delegation:%n%n%7%n%n

For information about correcting this problem, type in the command line:

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageF.htm

DSC_EVENT_DNS_NAME_ERROR Error DNS_ERROR_RCODE_NAME_ERROR (0x%3) occurred when DNS was queried for the service

Process name

2119 location (SRV) resource record used to locate a domain controller for domain %4%n%n

PID

The query was for the SRV record for %5%n%n Error code

Common causes of this error include the following:%n%n Domain name

The DNS SRV records required to locate a domain controller for the domain are not registered in DNS.

DNS path

These records are registered with a DNS server automatically when a domain controller is added to a domain.

Addresses of DNS servers used

They are updated by the domain controller at set intervals.

List of DNS zones

This computer is configured to use DNS servers with following IP addresses:%n%n%6%n

One or more of the following zones do not include delegation to its child zone:%n%n%7%n%n

For information about correcting this problem, type in the command line:

Domain unknown to DNS.

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageE.htm

E 1 Yes

DNS_EVENT_DNS_NO_ERROR DSAccess is unable to connect to any domain controller in domain %3 although DNS was successfully queried for the service location

Process name

2121

DNS info for domain is fine, but we still can’t (SRV) resource record used to locate a domain controller PID

W 1 Yes

Page 434: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 31

for that domain%n%n

The query was for the SRV record for %4%n%n Domain name

The following domain controllers were identified by the query:%n%n%5%n

DNS path

Common causes of this error include:%n%n List of DCs

Host (A) records that map the name of the domain controller to its IP addresses are missing

or contain incorrect addresses.%n%n

Domain controllers registered in DNS are not connected to the network, are not running or have Netlogon service stopped.%n%n

For information about correcting this problem, type in the command line:

connect to any domain controllers.

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageHa.htm

DSC_EVENT_DNS_NO_ERROR_DC_FOUND

DSAccess is unable to connect to domain controller %3 although its service location (SRV) resource record is found in DNS %n%n

Process name

2123 The query was for the SRV record for %4%n%n PID

The following domain controllers were identified by the query:%n%n%5%n

Error code

Common causes of this error include:%n%n DC name

- Host (A) records that map the name of the domain controller to its IP addresses are missing or contain incorrect addresses.%n%n

DNS path

- Domain controllers registered in DNS are not connected to the network, are not running, or have Netlogon service stopped %n%n

List of DCs

For information about correcting this problem, type in the command line:

A particular domain controller we’re looking for is in the DNS but we can’t contact it.

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageHa.htm

W 1 Yes

DNS_EVENT_DNS_NO_ERROR_DC_NOT_FOUND

Domain controller %3 was not found when DNS was queried for the service location (SRV) resource records for domain %4%n%n

Process name

2124 The query was for the SRV record for %5%n%n PID

The following domain controllers were identified by the query:%n%n%6%n

Error code

Common causes of this error include the following:%n%n DC name

Domain controller we’re looking for is not in the DNS. Other domain controllers can be looked up - The DNS SRV records required to locate a domain DNS path

W 1 Yes

Page 435: Exchange Server Admin Troubleshooting

32 Module 7: Exchange Directory Components

controller for the domain are not registered in DNS.

These records are registered with a DNS server automatically when a domain controller is added to a domain.

List of DCs

They are updated by the domain controller at set intervals.

Addresses of DNS servers used

This computer is configured to use DNS servers with following IP addresses:%n%n%7%n

List of DNS zones

- One or more of the following zones do not include delegation to its child zone:%n%n%8%n%n

For information about correcting this problem, type in the command line:

fine.

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageE.htm

DNS_EVENT_DNS_OTHER Error 0x%3 occurred when DNS was queried for the service location

Process name

2122 (SRV) resource record used to locate a domain controller for domain %4%n%n

PID

The query was for the SRV record for %5%n%n Error code

For more information, type in the command line: Domain name

DNS error occurred

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageA.htm DNS path

E 1 Yes

DNS_EVENT_DNS_TIMEOUT Error ERROR_TIMEOUT (0x%3) occurred when DNS was queried for the service location

Process name

2120 (SRV) resource record used to locate a domain controller for domain %4%n%n

PID

The query was for the SRV record for %5%n%n Error code

The DNS servers used by this computer for name resolution are not responding.

Domain name

This computer is configured to use DNS servers with the following IP addresses:%n%n%6%n

DNS path

Verify that this computer is connected to the network, that these are the correct DNS server IP addresses,

Addresses of DNS servers used

and that at least one of the DNS servers is running.%n%n

List of DNS zones

DNS Query timed out

For more information on how to correct this problem, type

E 1 Yes

Page 436: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 33

in the command line:

hh tcpip.chm::/sag_DNS_tro_dcLocator_messageB.htm.

DSC_EVENT_GOT_SACL Exchange Server %3 now has Audit Security Privilege on Process name

2113 Domain Controller %4. PID

Server Name

A domain controller regained its SACL (logged if it previously didn’t have one)

DC Name

I 1

DSC_EVENT_NO_SACL Exchange Server %3 does not have Audit Security Privilege on

Process name

2112 Domain Controller %4. This Domain Controller will not be used by DSAccess.

PID

Server Name

A domain controller lost its SACL right (logged if it previously had it)

DC Name

W 0

LDAP Category Event ID Problem Event text: problem description and actions

required Data in event Severity Level Periodic

DSC_EVENT_KERBEROS_TIMEOUT Received LDAP_SERVER_DOWN (0x51) from Directory Server %3 due to

Process name

2111 Kerberos ticket timeout PID

Broken connection due to Kerberos ticket expiration (every 10 hours)

Server used

I 3

DSC_EVENT_FATAL_ERROR DSAccess needs to close a connection to the Domain Controller %3

Process name

2115 due to error 0x%4. PID

Server used

Got a “fatal error”, i.e. the connection needs to be closed and reopened; does not necessarily mean that the target domain controller is having problems.

Error

W 3

Page 437: Exchange Server Admin Troubleshooting

34 Module 7: Exchange Directory Components

Chart 3.1: DSAccess events new to Exchange 2003 Within the chart above, the only event log entries that are not new in Exchange 2003 are the 2080 and 2095 events. The 2080 event is updated to show an additional field pertaining to the domain controller’s operating system version. The updated 2095 now displays the previous “Config DC” name upon a change.

DSADiag.exe tool no longer supported The Exchange 2000 support tool, DSADiag, allowed Microsoft Product Support Services and customers to examine the list of suitable domain controllers chosen by DSAccess. DSADiag no longer functions in Exchange 2003, as the global catalog and domain controller list will be empty. What replaces this tool are the new monitoring and diagnostic events in the application event log, and the Directory Access tab that was introduced in the Exchange 2000 SP2 timeframe.

To examine the list of working domain controllers and global catalogs and Config DC, one would use of the Directory Access tab in the System Manager snap-in.

To examine the current suitability states, increase diagnostics logging for MSExchangeDSAccess (Toplogy Category), and examine the latest 2080 in the application event log. Refer to chart 3.1’s “Topology” category section for details on these events.

Customers may also utilize the exomatic.hta tool, which leverages the new WMI and monitoring providers in Exchange 2003. Although it doesn’t provide as much information as the event logs out-of-the-box, it is very easy to execute the DSADiag-like WMI provider to obtain a quick glance of working domain controllers/global catalogs/Config DCs and their associated up/down/fast/in sync states fairly quickly.

Although Exchange_DSAccessDC is the class to select within the exomatic.hta tool, the HTA also provides a complete list of other Exchange classes and properties that are new to Exchange 2003. Moreover, it has the added option to automatically generate scripts for you to integrate Exchange WMI access into your own programs. Exomatic.hta is available from Microsoft Product Support Services

New DSAccess perfmon counters (207222) New DSAccess counters allow Microsoft Operations Manager to monitor Active Directory domain controller health and Exchange topology changes. The counters relevant to the directory topic are the new domain controller latency/rate counters that may be used for domain controller troubleshooting. These counter instances contain names of domain controllers for easy use.

Some of these counters present the information for the last minute, i.e. this is per-minute-rate, and it is updated once a minute. The sampling interval (1 minute by default) will be configurable in registry, so that we could tune granularity in place.

DSAccess will allow monitoring of up to 64 domain controllers at a time.

The domain controller name (as well as process name) is limited to 63 characters (it has to have fixed size). If the name is longer than that, it will be truncated.

Note

Page 438: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 35

Counter name Changes Type

DC_TIME_Read Average latency counter (in millisec) PERF_AVERAGE_TIMER, also needs PERF_AVERAGE_BASE counter

DC_TIME_Search Average latency counter (in millisec) PERF_AVERAGE_TIMER, also needs PERF_AVERAGE_BASE counter

DC_RATE_Read Reads per second PERF_COUNTER_COUNTER

DC_RATE_Search Searches per second PERF_COUNTER_COUNTER

DC_RATE_TIMEOUTS LDAP_TIMEOUTs during last minute PERF_COUNTER_RAWCOUNT (per-minute rate will be calculated by DSAccess)

DC_RATE_FATAL_ERRORS How many times during last minute we needed to completely close the connection (such as after getting LDAP_LOCAL_ERROR). Excludes Kerberos timeout

PERF_COUNTER_RAWCOUNT (per-minute rate will be calculated by DSAccess)

DC_RATE_DISCONNECTS How many times during last minute we get LDAP_SERVER_DOWN or any other socket disconnect error

PERF_COUNTER_RAWCOUNT (per-minute rate will be calculated by DSAccess)

DC_RATE_SEARCH_FAILURES How many times during last minute user search fails, mostly with non-fatal and non-disconnect error. (Disconnect error might get there if there are no domain controllers available at all)

PERF_COUNTER_RAWCOUNT (per-minute rate will be calculated by DSAccess)

DC_RATE_BIND_FAILURES How many times during last minute bind to a domain controller failed PERF_COUNTER_RAWCOUNT (per-minute rate will be calculated by DSAccess)

DC_OUTSTANDING_REQUESTS Instant counter of outstanding searches or reads. PERF_COUNTER_RAWCOUNT

DC_TIME_NETLOGON How fast the last dsgetdcname (our “netlogon ping”) returns (in msec).

PERF_COUNTER_RAWCOUNT

DC_TIME_GETHOSTBYNAME Last gethostbyname call duration (in msec) PERF_COUNTER_RAWCOUNT

DC_AVG_TIME_KERBEROS Average Kerberos ticket lifetime for 16 last connections that had Kerberos expired

PERF_COUNTER_RAWCOUNT (average for 16 connections is made by DSAccess)

DC_AVG_TIME_CONNECTION Average connection lifetime for 16 recently closed connections PERF_COUNTER_RAWCOUNT (average for 16 connections is made by DSAccess)

DC_LOCAL_SITE 1 means domain controller is in the local site, 0 otherwise PERF_COUNTER_RAWCOUNT

DC_STATE_REACHABILITY Bitmask, 3 bits for reachability as domain controller, ConfigDC and global catalog (critical)

PERF_COUNTER_RAWCOUNT

DC_STATE_SYNCHRONIZED 1 is domain controller is synchronized, 0 otherwise (critical) PERF_COUNTER_RAWCOUNT

DC_STATE_GC_CAPABLE 1 if it is a global catalgo, 0 otherwise PERF_COUNTER_RAWCOUNT

DC_STATE_IS_PDC 1 if it is a PDC, 0 otherwise PERF_COUNTER_RAWCOUNT

DC_STATE_SACL_RIGHT 1 if Exchange server has SeSecurityPrivilege on this domain controller, 0 otherwise(critical)

PERF_COUNTER_RAWCOUNT

DC_STATE_CRITICAL_DATA 1 if the domain controller contains this Exchange server object (critical)

PERF_COUNTER_RAWCOUNT

DC_STATE_NETLOGON 1 if DsGetDcName to the domain controller succeeds (critical) PERF_COUNTER_RAWCOUNT

Page 439: Exchange Server Admin Troubleshooting

36 Module 7: Exchange Directory Components

1. Other useful counters (global)

GLOBAL_TIME_DISCOVERY Last topology discovery duration time (in millisec) PERF_COUNTER_RAWCOUNT

GLOBAL_TIME_DNSQUERY Last DNS query duration time (in millisec) PERF_COUNTER_RAWCOUNT

GLOBAL_DC_IN_SITE Number of suitable domain controllers in site after last discovery PERF_COUNTER_RAWCOUNT

GLOBAL_GC_IN_SITE Number of suitable global catalogs in site after last discovery PERF_COUNTER_RAWCOUNT

GLOBAL_DC_OUT_OF_SITE Number of suitable domain controllers out of site after last discovery PERF_COUNTER_RAWCOUNT

GLOBAL_GC_OUT_OF_SITE Number of suitable global catalogs out of site after last discovery PERF_COUNTER_RAWCOUNT

Chart 3.2: Domain controller Counters and Global perfmon counters

Page 440: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 37

Lab 7.2: Exomatic tool

Importance: In this lesson, we will observe the replacement for DSADiag: the exomatic tool. Additionally, we will explore additional monitoring interfaces exposed through WMI that are queried using the exomatic tool.

Lesson Objectives: At the end of this lesson, students will be able to use the exomatic tool.

Exercise 1: Observing DSADiag against Exchange 2003 1. From the Exchange 2003 virtual machine, mount the “admin_labfiles.ISO”

CD image. 2. Search for DSADiag on the mounted virtual CD drive. 3. Copy DSADiag to the \exchsrvr\bin folder. 4. Run DSADiag with option 1. What output do you see? How is it different

from the output if you ran it on the Exchange 2000 server? (Answer is provided in Appendix C: 7.1.4)

Exercise 2: Exomatic tool 1. Navigate to the \exomatic directory in the mounted CD image. 2. Copy exomatic.hta to a directory on the local machine. 3. Double-click on exomatic.hta 4. It may take a few seconds to launch, but when it appears, select the drop-

down “Exchange_DSAccessDC” 5. How does this output compare with DSADiag against Exchange 2000?

6. Try running exomatic against an Exchange 2000 server. Will it work?

7. Try running additional scripts and examine their corresponding output. You may also edit the script (such as redirecting output to a message box) or save your customizations to a text file.

Page 441: Exchange Server Admin Troubleshooting

38 Module 7: Exchange Directory Components

Lesson 4: Other Directory Changes

This topic discusses other directory-related changes.

Page 442: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 39

LDAP traffic is now signed

In Exchange 2003, LDAP traffic is signed (and thus, mostly encrypted) for security purposes. Specifically, the following flags are used when binding to ADSI:

ADS_USE_ENCRYPTION - Forces ADSI to use encryption for data exchange over the network.

ADS_USE_SIGNING - Verifies data integrity to ensure the data received is the same as the data sent.

ADS_USE_SEALING - Encrypts data using Kerberos. Network monitor captures will show encrypted packets, which are impossible to troubleshoot. To override the new security behavior when, for example, troubleshooting typical Exchange System Manager error messages returned by Active directory, one must first enable the DebugLDAP key to prevent Exchange from encryption. In this key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange

Create a DWORD called “DebugLDAP” (without quotes) with a value of 1.

After restarting Exchange 2003 Exchange System Manager, the override behavior takes effect, and a network monitor capture will show (in clear text) the LDAP calls made to the domain controller, therefore revealing why a domain controller returned an error.

Signing applies for all admin binaries, not just Exchange System Manager. This means that the ADC Services MMC, as well as the maildsmx extensions for the Active Directory Users and Computers MMC are affected.

Note

Troubleshooting Tip

Page 443: Exchange Server Admin Troubleshooting

40 Module 7: Exchange Directory Components

If you ever need to hardcode the Directory Access property sheet for a server within Exchange System Manager for troubleshooting, ensure that you do not specify a pre-Windows 2000 SP3 domain controller. This is because older domain controllers do not support signing; yet there is no check in Exchange System Manager to prevent administrators from selecting pre-SP3 Windows 2000 domain controllers.

Exchange 2000 ADC services: Although the Exchange 2000 ADC service may be installable in a domain with Windows 2003 domain controllers, there is no built-in check in its setup code to detect a supportable domain controller. A design change in Windows 2003 Forest-Functional mode causes problems with the Exchange 2000 ADC. Specifically, in Windows 2003 forest-functional mode, group membership changes in the Active Directory may not back replicate to Exchange 5.5. Therefore, the Exchange 2000 ADC cannot run against Windows 2003 forest-functional domain controllers. (Forest functional levels are discussed further in this lesson.) The Exchange 2003 version of the ADC does not have this problem..

Page 444: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 41

New logging for Recipient Update Service

Previously in Exchange 2000, proxy address generation was sometimes a mystery when customers couldn’t determine which recipient policies were stamping their users. There also wasn’t a way to tell the progress of the recipient update service (Recipient Update Service) when it was applying a recipient policy. In Exchange 2003, the Recipient Update Service logging is more explanatory:

What Is the Recipient Update Service Doing Now and Why? In Exchange 2003, you can view the object the Recipient Update Service is processing by setting “Proxy Generation” diagnostics to maximum Using Exchange System Manager, pull up the server properties of the Recipient Update Service machine. Choose Diagnostic Logging, MSExchangeSA, Proxy Generation. Set that value to maximum. Now in the event log you will start to see 3006 messages. Each message will contain the object being processed, which polices that apply to the object, the proxies the object has, and the proxies the Recipient Update Service will apply.

If the Recipient Update Service is doing a full rebuild, you can see how far along the Recipient Update Service is, by setting the “Address List Synchronization” diagnostics to medium.

Using Exchange System Manager, pull up the server properties of the Recipient Update Service machine. Choose Diagnostic Logging, MSExchangeAL, Address List Synchronization. Set that value to medium. Now in the event log look for 8329, 8332 and 8330. The Recipient Update Service logs 8329 at the beginning of a rebuild, and 8330 at the end. During the rebuild, the Recipient Update Service will log a 8332 message for every unit of time that follows this formula: Max( ( highest commited USN - initial USN)/10, 50000) units. For example. if the highest commit USN is less than 100000, you will only see two 8332s. So during a “rebuild” operation against a large directory, one might

Page 445: Exchange Server Admin Troubleshooting

42 Module 7: Exchange Directory Components

summarize that an 8332 event occurs every 10% of the way (approximately). For smaller directories, you may only see one 8332 event.

Troubleshooting with Meta-data and Repadmin The metadata is the data about the data. On every Active Directory object, the Active Directory records the metadata. This info contains when how many times an attribute was modified. What domain controller and when the attribute was last modified. This information is valuable if other applications than the Recipient Update Service are administrating the Active Directory. Because there are many processes that can change object attributes (ADC, Recipient Update Service, manual changes by admins), we often wonder “who or what process made that change?”

Example: On this topology, the following command

Repadmin.exe /showmeta “CN=_User105532-16,CN=Users,DC=hrut,DC=extest,DC=microsoft,DC=com” hrookut1dc yields…40 entriesLoc.USN Originating DC Org.USN Org.Time/Date Ver Attribute======= =============== ========= ============= === =========

Page 446: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 43

2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 objectClass 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 cn 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 instanceType 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 whenCreated 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 displayName 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 homeMTA 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 2 proxyAddresses 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 homeMDB 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 nTSecurityDescriptor 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 mailNickname 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 1 replicatedObjectVersion 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 name 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 2 userAccountControl 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 codePage 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 countryCode 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 dBCSPwd 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 logonHours 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 unicodePwd 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 ntPwdHistory 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 pwdLastSet 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 primaryGroupID 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 objectSid 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 accountExpires 2674425 Default-First-Site-Name\HROOKUT1DC 2674425 2003-02-07 10:55:35 1 lmPwdHistory 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 sAMAccountName 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 sAMAccountType 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 showInAddressBook 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 2 legacyExchangeDN

Page 447: Exchange Server Admin Troubleshooting

44 Module 7: Exchange Directory Components

2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 objectCategory 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 textEncodedORAddress 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 mail 2674424 Default-First-Site-Name\HROOKUT1DC 2674424 2003-02-07 10:55:35 1 msExchHomeServerName 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 1 replicationSignature 2677104 Default-First-Site-Name\HROOKUT1DC 2677104 2003-02-07 11:47:51 2 msExchALObjectVersion 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 1 msExchADCGlobalNames 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 2 msExchMailboxSecurityDescriptor 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 msExchUserAccountControl 2674463 Default-First-Site-Name\HROOKUT1DC 2674463 2003-02-07 10:55:59 1 msExchMailboxGuid 2676961 Default-First-Site-Name\HROOKUT1DC 2676961 2003-02-07 11:47:29 1 dLMemDefault 2677104 Default-First-Site-Name\HROOKUT1DC 2677104 2003-02-07 11:47:51 2 msExchPoliciesIncluded 1 entries.

Suppose you were trying to track down a problem with the proxy addresses. From the metadata you can see that the last time the Recipient Update Service modified the object was on 2003-02-07 11:47:51, according to msExchALObjectVersion. In searching additional timestamps on other attributes, you can also declare that at the same time, the Recipient Update Service also modified msExchPoliciesIncluded. From this trace evidence, we can conclude that anything else wasn’t modified by the Recipient Update Service. Therefore, the proxy addresses were last modified on 11:47:29 by the ADC (since the msExchADCGlobalNames were modified at the same time).

Page 448: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 45

Changes to the Offline Address Book (OAB)

In Exchange 2003, a new system folder called "OAB Version 3a" becomes available. This system folder is used solely by Outlook 2003 and later clients, as Outlook now has most of its code ported to Unicode. The reason for the change was because OAB generation by a single server defaults to the server’s system codepage, so characters that cannot be converted to a codepage of that single server become question marks. By generating the Offline Address Books in Unicode, Outlook preserves the Unicode data and is able to render Unicode in the address book UI.

Though it is still possible that the OAB version 2 may be used, Outlook 2003 prefers to download the offline address lists in Unicode format. One minor issue is that the new OAB is created without replica information filled-in. So when Exchange 2000 customers have already manually set replication info on the "OAB Version 2" folder, they may not realize they need to do it again when upgrading to Exchange 2003 for the new folder.

Running GUIDGEN.exe has, in the past, been a troubleshooting step to produce new site folders when the site folder GUID was lost – typically through server deletion from Active Directory. The new OAB v3a follows the same behavior, in that it will be regenerated when the admin group’s sitefolderGUID is updated.

The Unicode OAB also contains the following MAPI address book properties for each record:

New Unicode OAB

Page 449: Exchange Server Admin Troubleshooting

46 Module 7: Exchange Directory Components

PR_USER_CERTIFICATE PR_BUSINESS_TELEPHONE_NUMBER_W PR_GIVEN_NAME_W PR_INITIALS_W PR_STREET_ADDRESS_W PR_LOCALITY_W PR_STATE_OR_PROVINCE_W PR_POSTAL_CODE_W PR_COUNTRY_W PR_TITLE_W PR_COMPANY_NAME_W PR_ASSISTANT_W PR_DEPARTMENT_NAME_W PR_EMS_AB_TARGET_ADDRESS_W ** PR_HOME_TELEPHONE_NUMBER_W PR_BUSINESS2_TELEPHONE_NUMBER_W_MV ** PR_HOME2_TELEPHONE_NUMBER_W_MV ** PR_PRIMARY_FAX_NUMBER_W PR_MOBILE_TELEPHONE_NUMBER_W PR_ASSISTANT_TELEPHONE_NUMBER_W PR_PAGER_TELEPHONE_NUMBER_W PR_COMMENT_W PR_EMS_AB_PROXY_ADDRESSES_W PR_USER_X509_CERTIFICATE PR_EMS_AB_X509_CERT PR_EMS_AB_HOME_MDB_W ** PR_EMS_AB_MANAGER_W ** ** = These properties were added to the OAB Version 3a files. Previously, these properties were not included in the OAB.

Note: Although the PR_EMS_AB_MANAGER_W field is in the V3a OAB files, it is not visible from the client when running offline.

The list of properties in the OAB is hard-coded.

Permissions change to OAB In Exchange 2000, it is possible to get to \non_ipm_subtree\OFFLINE ADDRESS BOOK\ through Outlook Web Access (OWA). Additionally, the “Default” user had “Editor” role permission, so Exchange 2003 is more secure out-of-the-box by demoting the “Editor” role to “Reviewer.” This ACE modification is also set on upgrades.

Smaller, Leaner OAB In Exchange 2000, OAB Generation stored certificates not used by Outlook in the OAB. Certificates accounted for more than half of the OAB size internally at Microsoft. Specifically, most of the size was due to unusable EFS and expired certificates in the UserCertificate attribute.

In Exchange 2003’s OAB generation filters the certificates in the UserCertificate attribute to remove expired, invalid, or those that cannot be used for e-mail based on the usage and enhanced usage flags.

Page 450: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 47

Differences between Windows 2003 and Windows 2000 forests/domains

Different Domain/Forest Functionality Levels Windows 2003 server introduces a more granular approach to setting functionality levels than Windows 2000. The forest can remain in Windows 2000 mode while individual domains can be upgraded to Windows 2003 mode.

For a multi-domain forest, it is possible to promote child domains to Windows 2003 mode while the root remains in Windows 2000 native mode. However, if the root domain is converted to Windows 2003 mode, all child domains will be promoted to Windows 2003 mode.

Page 451: Exchange Server Admin Troubleshooting

48 Module 7: Exchange Directory Components

Some other new features are introduced with new functional levels, such as domain and domain controller rename. However, these new abilities will cause problems with Exchange 2000 and 2003 (discussed below). For a list of functionality levels and their available features and restrictions, open Windows 2003 server “Help and Support,” and search on the string “Domain and forest functionality” with quotes. The table should be contained within the first link found in the search.

InetOrgPerson class The inetOrgPerson object-class is added with Windows 2003 Active Directory. InetOrgPerson objects are based on the standard defined in RFC2798, and are provided primarily for interoperability with third-party systems. This object is designed to be used as an “outward facing” security context. Because of this, these objects are ideal for use as e-mail recipients for external users or internet access to mail in a hosting scenario. However, there are some restrictions depending upon the detected scenario, and whether the maildsmx extensions are registered. The MailDSMX.dll is available on machines which have an Exchange 200x component, such as Exchange System Manager or the ADC services snap-in – which enables the ability to invoke “Exchange tasks” against an InetOrgPerson object. Yet Exchange tasks are limited when operating on this class depending upon the scenario detected:

In Exchange 2000, attempts to mailbox-enable these objects are blocked with “no tasks available.” In fact, the existence of any pre-Exchange 2003 servers will cause the Exchange 2003 admin components to block with this error:

<summary isWarning="false" errorCode="0xc1034957">In order to use Exchange with inetOrgPerson objects is it necessary to upgrade all servers. InetOrgPerson is not supported in the presence of Exchange 5.5 or Exchange 2000 servers.</summary>

Depending upon the Exchange versions detected in the organization, there are only certain times when mail-enabled or mailbox-enabled inetorgperson objects are supported:

Exchange 2003/2000/5.5 – mail-enabled inetorgperson not supported Exchange 2003/5.5 – mail-enabled inetorgperson not supported Exchange 2003/2000 – mail-enabled inetorgperson not supported Exchange 2003 – Supports mail-enabled inetorgperson objects PureADC – mail-enabled inetorgperson not supported. (See below)

For the PureADC scenario, no Exchange 200x servers exist, but Exchange 5.5 and Active Directory Connector are installed. In this configuration, the Active Directory Users and computers snap-in allows for the creation of mailbox-enabling attributes on InetOrgPerson objects, provided that the OU is listed as an export container of a connection agreement. Fortunately, connection agreements will not replicate InetOrgPerson objects in the direction of Active Directory Exchange 5.5. In the opposite direction, it is possible for customers to configure Exchange 5.5 mailboxes to have assoc-NT-account (Primary Windows NT accounts) that are InetOrgPerson objects. Through that process, the ADC will replicate these mailbox-enabling attributes to Active

Page 452: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 49

Directory, but rather than matching and placing attributes on an inetorgperson object, the ADC will fail its matching rules and create a disabled account.

Note that as soon as Exchange 2003 or Exchange 2000 is installed into the pureADC topology, the ability to create mailbox-enabled inetOrgPersons disappears. This is by design, because there is no support for InetOrgPerson objects whenever an ADC is installed.

To become supportable again, customers should either a) remove Exchange attributes from the InetOrgPerson objects created when the topology used to be PureADC, or b) quickly migrate all Exchange 5.5 mailboxes to Exchange 2003, and decommission all Exchange 5.5 servers and Active Directory Connectors.

Non-Compliant Attributes Attributes from Exchange 2000 schema fixed The Exchange 2000 schema defined three non-Request for Comment (RFC)-compliant attributes: houseIdentifier, Secretary, and labeledURI. The Windows 2000 InetOrgPerson Kit and the Adprep tool in Windows 2003 redefined these attributes as described in Request for Comments (RFC) 2798. Unfortunately, running ADPrep /forestprep could cause conflicts because the pre-existing Exchange 2000 version of these attributes in a forest would contain incorrect ldapdisplaynames. Article 314649 explains the problem and how to use the InetOrgPerson toolkit to correct this problem. Specifically, the toolkit imports the correct ldapdisplaynames.

With the Exchange 2003 schema, the new setup /forestprep will import the same corrections to these three attributes, so there is no need to use the InetOrgPerson toolkit prior to running ADPrep /forestprep.

Domain rename Domain Rename (rendom.exe) allows for renaming the DNS and NetBIOS names of domains. This new Windows 2003 feature allows complete restructuring of the domain hierarchy, with the exception of the root domain. In the root domain, the DNS and NetBIOS names can be changed, but it must remain the forest root. Additionally, the forest must be at the Windows 2003 forest functional level. Domain Rename does not support domain “grafting”, that is, domains cannot be moved between forests. One interesting option is that a child domain can become a new tree during the rename process. For more details on domain rename, refer to the documentation at http://www.microsoft.com/windowsserver2003/downloads/domainrename.mspx

Domain rename is not supported when Exchange 2000 or Exchange 2003 exist in the forest. However, there is no mechanism to block customers from renaming domains where Exchange is installed. After renaming a domain’s DNS namespace, anything that uses DSAccess will cease to function properly. Specifically, when an Exchange 2000 or Exchange 2003 server comes online, the System Attendant will fail to start. The only resolution for RTM is for the customer to re-use rendom.exe to rename the domains to their former names.

Despite the problems with Exchange when domain rename is used, the RENDOM.EXE utility may prove to be a good troubleshooting utility. “Rendom /list” will help to identify all directory partitions, domains and NetBIOS names.

Page 453: Exchange Server Admin Troubleshooting

50 Module 7: Exchange Directory Components

DC Rename A new feature available with new Windows 2003 functional modes is the ability to rename domain controllers. Renaming a domain controller requires that the domain must be raised to the Windows 2003 functional level (no pre-Windows 2003 domain controllers can exist)

To rename the domain controller DC to altDC in the example.com domain, use the following syntax: netdom computername dc /makeprimary:altdc.example.com

Unfortunately, like domain rename, there are no safety checks to guard against problems with Exchange 2000 or 2003. So some negative side effects of renamed domain controllers are:

ADC connection agreements no longer replicate because their “connections” tab still points to the old domain controller names.

Domain controllers that were manually specified through Exchange Serve Manager’s “Directory Access” tab become invalid. As such, DSAccess can no longer query domain controllers properly, so all Exchange services will cease to function.

Domain controllers that are automatically discovered by DSAccess may not be used until the Exchange servers are restarted. This is because DSAccess still has the old domain controller names in memory.

Linked Value replication In Windows 2000, when a change was made to a member of a group (one example of a multivalued attribute with linked values) the entire group had to be replicated throughout the forest. For large distribution groups, this caused excessive replication traffic. In Windows 2003 Interim Forest mode (forest level 1) and Windows 2003 Forest mode (forest level 2), linked value replication is a tremendous bandwidth-saver which will allow only the membership changes of a group to be replicated instead of the entire membership list. This allows for removal of the recommended 5000 user per group restriction. Because this requires Windows 2003 forest functionality level, it is incompatible with Windows 2000 domain controllers (even domain controller with Windows 2000 SP3). See: Q265400 for more information on this issue in Windows 2000.

Dynamic NSPI Enable/Disable NSPI is the remote procedure call that is used primarily by clients to interface with Active Directory. This feature provides the interface for Outlook logons and Address Book views. The NSPI interface provides access to a limited set of objects and their attributes in Active Directory.

Prior to Windows 2000 SP3, adding the global catalog role to a domain controller required a reboot to register the name service provider interface (NSPI) endpoint. On Windows 2003 and Windows 2000 SP3 domain controllers, the NSPI endpoint will register dynamically without requiring a reboot. This eliminates the problem of a global catalog not having been rebooted after it was assigned the global catalog role. (Note that while NSPI can be dynamically registered, the Windows 2003 global catalog will still not advertise as a global catalog until it has been fully replicated. To determine if it has been fully replicated, connect to it over port 3268. Without even binding

Page 454: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 51

with credentials, the right-hand pane will say isSynchronized: TRUE for an up-to-date global catalog.)

This benefits Exchange 2003 by removing some additional reboot scenarios. Fewer reboots = happier admins. It also prevents the situation where a global catalog is advertised but has not registered the NSPI interface and therefore remains inaccessible to clients.

Group Membership Caching (“no GC logon”) With “No GC Logon” the domain controller caches complete group membership of a user. This cache is populated at first logon – so you can logon to the domain once, and the local domain controller caches all group memberships to your token. Subsequent logons use the information in the cache stored on a local domain controller, without needing to contact a global catalog.

By default, the cache is refreshed every eight hours from nearest (or selected) site and global catalog. It observes the replication schedule on site link, so if the site link schedule is closed, it could take even longer to reflect new group membership changes. Knowledge that a user has logged on to a site replicates amongst the domain controllers, yet the actual cache itself is not replicated. The domain controller that holds the cache will retain the cache, even after reboots because it is written to disk.

No GC Logon is enabled as an attribute of a site object. Configuration:

1. Go to the properties of NTDS Site Setting object. 2. Use the checkbox to “Enable Universal Group Membership Caching”. 3. There is also a selection for the site from which to refresh.

Three attributes have been added to the Active Directory schema to support this. The values are:

CachedMembership CachedMembershipTimeStamp SiteAffinity

No GC Logon can be enabled immediately upon installation of Windows 2003 domain controllers. No domain or forest functionality mode changes are required. These values are simply ignored by Windows 2000 domain controllers.

Impact to Users: This feature allows users to logon to their domain utilizing a global catalog to populate the universal group memberships. They can then continue to logon to that domain successfully in the future – even if there is no global catalog locally available to provide the universal group memberships.

Impact to Exchange: Even though the intent of this feature is to allow branch office scenarios without any global catalog installed, Exchange still requires a global catalog to function. So customers that leverage this new Windows 2003 feature may still result in bandwidth saturation, since all Exchange DSAccess traffic must go across the WAN to the nearest global catalog. Thus, it is not recommended to use “No GC Logon” whenever WAN bandwidth is slow or costly.

Page 455: Exchange Server Admin Troubleshooting

52 Module 7: Exchange Directory Components

Differences in ADMT 2.0 ADMT 2.0 that is available with Windows 2003 allows for the ability to mass-modify the “Primary Windows NT” account fields in the Exchange 5.5 directories. Customers must first use the user migration with SIDHistory, which behaves the same as Windows 2000’s ADMT 1.0, and then they may use the new “Exchange Directory Migration Wizard” task. The ADMT 2.0 “Exchange Directory Migration Wizard” task connects to an Exchange 5.5 directory and converts all of the Exchange 5.5 objects’ associated NT4domain\useraccounts into ADDomain\useraccounts, where the latter set was created from the initial user migration with SIDHistory. This process facilitates user logons onto their new Active Directory accounts, without any loss of fidelity in delegation and public folder permissions.

ADMT 2.0 also includes a password migration feature, though this subject falls outside of the scope of Exchange 2003 directory changes. For more information on ADMT 2.0, refer to the readme.doc within the Windows 2003 CD\i386\ADMT directory.

Page 456: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 53

Schema changes

Microsoft Mobile Information Server (MMIS schema extensions are not reused in Exchange 2003’s mobility components – that is, the two sets of schema extensions do not overwrite each other, nor do MIS and Exchange 2003 share any schema attributes. This means that an organization migrating from Exchange 2000 + MMIS to Exchange 2003 can have both mobile systems operating side by side without conflict. Thus, migration can be done over time rather than overnight. The only caveat is that no MMIS components may exist on the same machine as Exchange 2003.

To determine all schema changes between Exchange 2000 RTM (4417.5) and Exchange 2003: 1. Open Schema9.LDF from the RC0 media with notepad 2. Find "dn: CN=ms-Exch-CalCon-Client-Wait,<SchemaContainerDN>" 3. From this line to the end of the file are the schema changes between

Exchange 2000 and 2003 As of build RTM, Exchange 2003 introduces approximately 12 new classes and 60 new attributes. The changes in the schema are documented within the SDK documentation, and can be referenced in MSDN online at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/e2k3_ldf_diff_ad_schema_intro.asp

Page 457: Exchange Server Admin Troubleshooting

54 Module 7: Exchange Directory Components

New Admin Tracing Capabilities

Improved admin tracing capabilities exist in Exchange 2003. Notably, you may use regtrace.exe to capture trace information on any communication with Active Directory using Exchange System Manager. This is useful for troubleshooting the cause of any unexplained popup errors while using System Manager. To enable retail tracing, drop a trace.ini file into the Windows directory.

The trace.ini file should contain

[Base] Error=16 [DS] DS Error=16 DS Flow=16 Data Received=16

Then enable tracing with regtrace.exe.

Though not related to directory changes, there are several other admin binaries which may be traced using the following file which you may copy and paste from here:

Page 458: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 55

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; ; Exchange 2003 Admin Retail Traces ; ; This file will allow you to enable additional tracing information for ; the admin binaries in E2K3. ; ; How to use: ; 1. Put this file in the Windows directory (not system32), that's ; the directory pointed to by the %windir% environment variable. ; 2. Uncomment the traces you want. See below for lists of common ; traces to enable. ; 3. Run regtrace.exe (should already be on the server) and select ; the following options: ; "Recoverable Error Conditions", ; "Debug Statements", and ; "Function Entry/Exit". ; On the "Output" tab, select "File" and make sure the filename ; points to a valid drive and directory and change the "Max ; Trace File Size" to 40 megabytes. Save the changes and exit ; out of regtrace. ; 4. Perform the actions you want to be traced. ; 5. Turn off tracing (with regtrace) because tracing does add some ; minimal overhead to the processes. ; 6. Use tracevwr.exe to view the trace results. Tracevwr can ; filter the traces based on the process id so it can be very ; helpful to also get the process id of the binary you wish to ; trace. ; ; The following files are effected by this file: ; exadmin.dll -- ESM (Exchange administration MMC snapin) ; exsetdata.dll -- Setup ; exmgmt.exe -- E2K3 WMI providers ; cdoexm.dll -- CDOEXM com objects ; mad.exe -- System Attendant ; maildsmx.dll -- Exchange extension to ADU&C ; adcadmin.dll -- ADC administration MMC snapin ; excluadmin.dll -- Exchange administration cluster extension ; ; Only mad.exe can pick up changes to the trace.ini file while it is ; running. To get mad.exe to use the new trace.ini values you must change ; and save the trace.ini file and then make some change to the regtrace

Page 459: Exchange Server Admin Troubleshooting

56 Module 7: Exchange Directory Components

; parameters. All the other admin binaries only look at the trace.ini ; file when regtrace is enabled and they are started. ; ; Helpful combinations of traces to enable for certain areas and the ; binaries they effect. ; ; General Errors (all admin binaries): ; Base/Error ; ; Public Folder (exadmin.dll, exmgmt.exe): ; Base/Error ; HTTP/Data Sent ; HTTP/Data Received ; ; Mailbox Clean (mad.exe): ; Base/Error ; Base/MapiSession ; Base/MapiProfile ; Base/EventLog ; MBClean/Values ; ; Proxy Generation (mad.exe): ; Base/Error ; Base/EventLog ; RPC/RPC Calls ; DS/CProxySession ; DS/CRecipientPolicy ; ; Active Directory communication (all admin binaries): ; Base/Error ; DS/DS Error ; DS/DS Flow ; DS/Data Received ; ; Message Tracking provider (exmgmt.exe): ; Base/Error ; WMIPROV/MTC ; ; E2K Monitoring WMI providers (exwmi.dll): ; Base/Error ; EXWMI/General ; EXWMI/Details ; ; E2K3 WMI Providers (exmgmt.exe): ; Base/Error ; BaseWMI/General ; BaseWMI/Context ; BaseWMI/Token ; ; Store interaction (create/delete/mount mdbs - Exadmin.dll, cdoexm.dll) ; Base/Error ; Logic/Store ;

Page 460: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 57

; Exchange Cluster extension (Excluadmin.dll) ; CLUSTERUTILITY/Cluster Utility ; CLUSTERUTILITY/Cluster Property List ; ; Under retail tracing only value 16 (reg trace output) is supported. ; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; [Base] ;Error=16 ;MapiSession=16 ;MapiProfile=16 ;EventLog=16 [BaseWMI] ;General=16 ;Context=16 ;Token=16 [CLUSTERUTILITY] ;Cluster Utility=16 ;Cluster Property List=16 [DS] ;DS Error=16 ;DS Flow=16 ;Data Received=16 ;CProxySession=16 ;CRecipientPolicy=16 [EXWMI] ;General=16 ;Details=16 [HTTP] ;Data Sent=16 ;Data Received=16 [LOGIC] ;Store=16 [MbClean] ;Values=16 [RPC] ;RPC Calls=16 [WMIPROV] ;MTC=16

Page 461: Exchange Server Admin Troubleshooting

58 Module 7: Exchange Directory Components

Page 462: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 59

Lab 7.3: Per-Attribute change troubleshooting

This lab will build the engineer’s skill in determining if an Active Directory Object may have been stamped by the Recipient Update Service, or perhaps through a user script.

1. Pair-up with a partner.

2. Create a mailbox-enabled user.

3. Quickly ask your partner to modify an attribute of your new user account, and then set it back to the way it was (i.e. partner can add a telephone #, and then immediately delete it).

4. Can you determine which happened first?

a. Recipient Update Service stamping, or

b. partner’s attribute modification?

5. Can you determine which attribute your partner modified?

(Hint in 7.3.5 in Appendix C if you cannot)

Lab 7.4: Post-Setup and SRS replication troubleshooting

Post Exchange server installation, ADC replication involves system/connector/server objects. Although the SRS and config Controller Agreement facilitate this replication, there are still common support issues involving the ability for these objects to replicate. Again, behavior cannot always be described in words. Often, it is easiest to learn concepts by observation.

This lab contains exercises that are not discussed within this module’s lessons. However, the exercises highlight common Exchange 2000 and Exchange 2003 directory component problems.

At the end of this lesson, students will be able to

Describe the difference between configuration and manual connection agreements

Create a site topology, describing which sites are mixed vs. pure Use Active Directory Clean Resolve the common customer problem of replicating users from pure

Administrative Groups into the Exchange 5.5 global address list (GAL) Break and fix customer issue: How to recover orphaned Controller

Agreements Resolve customer issue: How to rehome site connectors Describe what types of servers may be possible targets of directory

replication connectors.

Importance:

Importance:

Lesson Objectives:

Page 463: Exchange Server Admin Troubleshooting

60 Module 7: Exchange Directory Components

Exercise 1: Basic observations of config Controller Agreement 1. Open up Exchange System Manager and modify the properties of the

mailbox store on Z2. Enable a 50 MB warning storage limit. 2. Open-up Exchange 5.5 admin, and connect to the SRS database on Z2. In

the properties of the mailbox store, do you see the storage limits appear? 3. If not, what component is responsible for pushing changes to server and

configuration information between Active Directory and 5.5?

4. Open-up ADC Manager, and force a replication on the config Controller Agreement.

5. Go back to Exchange 5.5 admin that is connected to the SRS database. Do you see the storage limits on Z2’s mailbox store replicate?

6. Let’s replicate in the other direction: Use Exchange 5.5 admin to connect to Z0, and make a change to Z0’s mailbox limits, or place an administrative note on the message transfer agent.

7. Follow the replication path. The next component that should receive these updates is the SRS database on Z2. (to speed up intrasite directory replication, use Exchange 5.5 admin to navigate underneath Z2, and then do an “update now” on the properties of the site replication service) Then use Exchange 5.5 admin to connect to the SRS, and verify that the changes in step 6 carried-over.

8. Replicate the config Controller Agreement, and confirm the storage limit (or optionally an admin note) from step6 carried-over to Active Directory.

Exercise 2: How the ADC/SRS handles new admin groups 1. Witness config Controller Agreement/SRS arbitrate a new naming context.

On either Z3 or Z2, open-up Exchange System Manager and create a new Admin Group called “Pure AG”

2. Notice how pure Exchange 5.5 sites are listed in white. When a new Exchange 2000 server joins the pure Exchange 5.5 sites, they become mixed admin groups and will no longer appear white.

3. Open-up ADSIEdit, and observe the new Admin Group “pure AG”. Do you see how configuration/services/Microsoft exchange/Microsoft/administrative groups reflects in Exchange System Manager?

4. Open-up the ADC manager program on either z3 or z2. Go to the properties of the config Controller Agreement. Does the pure admin group appear on the “from windows” tab? (if not, the SRS/config Controller Agreement hasn’t gone through arbitration. To speed up the new site naming context discovery, force a site Knowledge Consistency Checker and stop and restart

Page 464: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 61

the ADC service. Forcing a site Knowledge Consistency Checker is discussed in kb article 285177)

5. On the “from Exchange” tab of the config Controller Agreement, how can you list which sites have only Server 5.5 servers installed?

6. On the “From Windows” tab, can you list which sites have Exchange

2000/2003 installed? 7. Comparing both tabs, can you list which admin groups are mixed? 8. Eventually, the config Controller Agreement will replicate this new Admin

group to the SRS. Use the Exchange 5.5 admin program to confirm that the “Pure AG” naming context appears in Exchange 5.5 as a new site.

9. If you desire, use Exchange System Manager to create as many pure Exchange 2000/2003 admin groups as you desire, and witness their propagation to the Exchange 5.5 directories.

Exercise 3: When a pure Exchange 2000/2003 Admin Group exists 1. Install Exchange 2003 on z6, and install it into the pure AG 2. In the childvm domain, create a few mailbox-enabled Active Directory

accounts, and have their mailboxes reside on Z6. 3. Using the Exchange 5.5 admin.exe program, connect to the Exchange 5.5

server Z0. Will we expect to see the new mailboxes objects appear in the Exchange 5.5 GAL? No, there is not Controller Agreement replicating that object.

4. Using the Exchange 5.5 admin.exe program, connect to the SRS on Z2. Why do we see the new mailbox-enabled objects in the global address list?

This is by design, since the SRS does not show the Exchange 5.5 site naming context; instead, its representation of the GAL is a query to the local global catalog for mail-enabled objects. To truly see what Exchange 5.5 site naming context the SRS has stored (rather than a query to Active Directory), use LDP or LDIFDE to browse/dump the directory objects stored within SRS.edb.

5. Common customer problem: How do we get the pure Administrator Group’s users replicated to the Exchange 5.5 GAL? Create a recipient Controller Agreement for these new mail-enabled users, and source the childvm\users OU. On the connections tab, make sure the Exchange server listed is the SRS database (z2/port379). Why did we select this instead of an Exchange 5.5 server?

Page 465: Exchange Server Admin Troubleshooting

62 Module 7: Exchange Directory Components

Reason1) The SRS can accept a write operation for other naming contexts as long as it is responsible for them. To confirm that this SRS (and not another) owns the “pure AG” naming context, look at the config Controller Agreement’s msexchserver1exportcontainers attribute.

Reason2) It does not matter what you specify on the default destination because the legacyexchangeDN attribute of these Exchange 2000 mailbox-enabled Active Directory objects will be sorted-out by the SRS and placed into the proper Exchange 5.5 site naming context.

6. When you are finished, power-down z6. (We no longer use this virtual machine in any future exercises, but you may opt to suspend it if you would like to experiment after class.)

Exercise 4: Using ADClean 1. On remote55, create an account called Nt4domain\ex4user 2. On z0, using Exchange 5.5 admin.exe, connect to itself. 3. Create a mailbox called ex4user, and assign the Primary Windows NT

Account to “nt4domain\ex4user” 4. Replicate the hubsite recipient connection agreement. You should see a

disabled account appear in Active Directory called “ex4user” 5. Install ADMT from the ISO image of the Windows 2003 CD. 6. Run ADMT, and migrate ex4user from NT4domain to MS domain. Be sure

to migrate sidHistory as well. 7. Examine the ADC-generated account and the ADMT-generated account. Do

they have the same samAccountName (pre-Windows 2000 login name)? How is this different from Exchange 2000’s ADC? (Answer in Appendix C: 7.4.7)

8. From the start menu, open Microsoft Exchange Deployment Active Directory Account Cleanup Wizard

9. Practice merging the two ex4user accounts’ source mail attributes into the target ADMT-generated account.

10. Verify that the ADC-generated account no longer exists. 11. (Optional) ADMT Exchange Directory Migration Wizard: Stop your hubsite

Controller Agreement, and create another nt4 account w/hubsite mailbox. Then rerun ADMT 2.0 to migrate the user, and then run the new “Exchange Directory Migration Wizard feature” to see how it changes the Primary Windows NT Account on the Exchange 5.5 mailbox. Then replicate your hubsite Controller Agreement to confirm that a placeholder account does not get created.

Page 466: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 63

Exercise 5: Msexchuseraccountcontrol/masteracctsid 1. In the MS domain, create a new Active Directory account called

“remotewithADacct” Do not create a new mailbox when the wizard prompts you to.

2. In the remote55site, create a new mailbox called remotewithADacct. Associate the MS\remotewithADacct account with this mailbox.

3. In remote55site, create a new mailbox called “remotewithNTacct” Create a new Windows NT 4 account in the NT4domain when Exchange 5.5 admin prompts you.

4. Create a recipient Controller Agreement between the remote55 server and Z2.

5. Verify the recipient Controller Agreement replicates, and that a “remotewithADacct” is matched and stamped, while “remotewithNTacct” is created is a disabled user account.

6. Use ADSIEdit, and go to the properties of these two accounts. Which one uses msexchmasteraccountsid? Let’s also take a look at msexchuseraccountcontrol. When msexchuseracctcontrol is 0, what does msexchmasteraccountsid contain? In Active Directory Users and Computers, viewing advanced features, what are the mailbox permissions on this account?

7. When msexchuseracctcontrol is 2, what does Msexchmasteraccountsid contain? In Active Directory Users and Computers, viewing advanced features, what are the mailbox permissions on this account?

MasterAccountSIDS

The msExchMasterAccountSID is the real SID of a mailbox. It should only be stamped on disabled user objects. For enabled user objects, the objectSID is used For example a Windows NT4 user logs into his Windows NT4 domain to access his mailbox, which happens to be represented in an Active Directory Domain by a corresponding disabled user account. That disabled, placeholder account contains the msExchMasterAccountSID value matching the SID from his Windows NT4 user account. Though the placeholder account contains an objectSID, that objectSID won’t be used by the store in calculating permissions; instead, msExchMasterAccountSID is used.

The msExchMasterAccountSID has three common uses…

1) When an Exchange 5.5 mailbox is upgraded to Exchange 2000, if there is an msExchMasterAccountSID then the mailbox store will be ACLed with the msExchMasterAccountSID. This is the big reason that changing the msExchMasterAccountSID does not always yield pleasant results. The old SID is still present on permissions for the mailbox.

2) When you log on, the store or OWA can hunt down your mailbox via the msExchMasterAccountSID

3) When you give someone delegate access to your mailbox, the SID that is stamped is the msExchMasterAccountSID.

Page 467: Exchange Server Admin Troubleshooting

64 Module 7: Exchange Directory Components

Exercise 6: How to determine what ADC servers are installed in the forest.

1. Open ADC Manager and open properties of two connection agreements. On the general tab, switch the ADC service that runs the Controller Agreement from Z2 to Z3. This effectively “moves” the Controller Agreement. (If you cannot pick Z3 from the drop-down list, this means that the ADC service is not installed on Z3. Install it if necessary.)

2. Open-up Active Directory Sites and Services. 3. Expanding the list of sites, which machines have “Exchange Settings” as a

subcontainer? Should be z2 and z3 (and z6 if virtual PC is being used).

4. Open-up ADC manager on Z3, and examine the left-hand pane. What correlation is there?

You should see the same number of Controller Agreements in ADC MMC as you see in “Exchange Settings”, except for default ADC policy. You should also see Controller Agreements that don’t have a responsible server and are hidden from ADC UI.

Exercise 7: Customer problem: Customer deletes ADC from Active Directory Sites and Services in an attempt to “clean up” their Active Directory. This could be due to a planned decommissioning of the Z3 server, or just by accident.

1. To reproduce the issue, switch back to the “Exchange Settings” container in Active Directory Sites and Services.

2. Delete the ADC Service object under Z3. 3. Reopen the ADC Manager. Can we recover the Controller Agreements that

were running on the deleted ADC server? 4. To fix, first find the missing Connection agreements that are not viewable

through ADC Manager by opening ADSI Edit or LDP. 5. Browse to configuration\services\microsoft exchange\active directory

connections container within the Configuration Naming Context. Find the missing Controller Agreements in there.

6. Browse through the properties of the missing Controller Agreements. What can we do to make Z2 as the server that replicates this Controller Agreement? (Hint: The Controller Agreement’s need a new HOME.)

7. Once you have rehomed the Controller Agreement’s to Z2, refresh the MMC UI to ensure it is viewable once more.

Page 468: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 65

Exercise 8: How to view other Connection Agreements that exist in a forest

1. What Controller Agreements exist in Active Directory that do not appear in the ADC manager?

2. Investigate whether or not the default ADC policy is assigned a msexchhomesyncservice attribute by default? (Comment in Appendix C: 7.8.2)

Exercise 9: Premature deletion of Exchange 5.5 servers 1. Power-off Z0. This emulates a common customer problem where their

Exchange 5.5 server is taken offline and/or formatted. Customers tend to do this prematurely after they have finished moving all their mailboxes to the Exchange 2000/2003 server. Mailflow and 5.5 dirrep is now broken between hubsite and remote55site. To fix: we must rehome site connectors and dirrep connectors. (Note: There are no more lab steps utilizing the Z0 VM, so discard changes when powering off. However, if you wish to perform additional investigation after this lab, you may suspend Z0.)

2. In Exchange 5.5 admin on remote55site, go to the properties of the directory replication connector. Choose a new remote bridgehead. (Z2 will be the only other option). Why doesn’t Z3 appear as an option to be a bridgehead? (Answer in Appendix 7.9.2)

3. While connected to remote55site, change the site connector target

bridgehead, if applicable. 4. Use Exchange 5.5 admin and connect to the SRS on Z2. You should still see

Z0’s server object. Attempt to change the local bridgehead settings for site connector and dirrep connector so that they no longer rely on Z0.

5. Remove the Z0 server object from the SRS. 6. Test intersite replication by making a few configuration container changes

in remote55site, and verifying that they end-up in the SRS on Z2. 7. What else is broken? (review q152959 and 284148) 8. Fix the site folders. (GUIDGen is located in \labfiles\guidgen.) 9. Test mailflow between the two sites.

Exercise 10: Viewing site recipient objects in the SRS 1. Stop all ADC services.

2. Create a mailbox-enabled user on Z3.

3. Connect to the SRS on Z2.

4. Go to the server recipients container underneath Z3.

Page 469: Exchange Server Admin Troubleshooting

66 Module 7: Exchange Directory Components

5. If the ADC service is stopped, how did this newly-created mailbox-enabled user get written into the SRS?

A) This is a trick question. The object did not get replicated into the SRS. Instead, the Exchange 5.5 admin.exe view shows a referred query LDAP to Active Directory for mailboxes contained on that server.

6. To truly view the objects contained within the SRS, open LDP and connect to port 379 of Z2.

7. Navigate to the global address list. Do you see your newly-created mailbox object?

8. Practice running an ldifde query against port 379 of Z2, and confirm you have the same results.

Exercise 11: Multiple Site Replication Services (This lab is optional due to its lengthy process)

1. Create an SRS on z3. Wait for the new SRS directory object to replicate to remote55’s copy of the “hubsite” naming context. (You can check on progress of interim replication by connecting to z3’s SRS using Exchange 5.5 admin.exe. If you see “Microsoft DSA” or are unable to connect, then you must wait

2. In remote55site, request update on the dirrep connector. 3. When viewing properties of the dirrep connector, do we now see z3 as an

available remote bridgehead? 4. Install a new Windows 2003 server called zremote. If you had limited

resources and are running the virtual machines across several host machines, run this virtual machine on the physical machine that has most free memory. Join it as a member of the root domain.

5. Install Exchange 2003 onto zremote, but when prompted to choose admin group membership, choose to join the remote55 site.

6. Now there are three SRS’s in the organization. Try creating additional pure 200x Administrative Groups with various names, each from one of the three Exchange System Manager servers.

7. Do these new AG’s get arbitrated to only one SRS? (View configuration connection agreements to confirm)

8. Practice using PreferredSRS by reading and implementing article 315408

Learning Measure: In the figure illustrated in Lab 7.1, why isn’t there a directory replication connector between PureAG and Hubsite?

a) To Exchange 5.5, the directory replication connector is implied. b) There is no Exchange 5.5 interface with which to create the directory

replication connector in PureAG. c) Exchange 2003 does not yet exist in Hubsite. d) A config Controller Agreement will create it when another SRS is installed.

Page 470: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 67

e) A and B f) C and D g) All of the above Correct Answer is in Appendix C

Page 471: Exchange Server Admin Troubleshooting

68 Module 7: Exchange Directory Components

Appendix A: Numeric registry keys used by DSAccess and their values

This section lists DSAccess-specific registry keys under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeDSAccess.

Value Name Default value Min Max

CacheTTLUser 5*60 (5 min) 1 24*60*60 (24 hrs)

CacheTTLConfig 15*60 (15 min) 1 24*60*60 (24 hrs)

MaxMemoryUser 0 (disabled) 1 1000 * 1024 * 1024 (1000 MB)

MaxMemoryConfig 0 (disabled) 1 1000 * 1024 * 1024 (1000 MB)

CacheSizeUser 140 *1024 *1024 (25 MB) 1 1000 * 1024 * 1024 (1000 MB)

CacheSizeConfig 5 *1024 *1024 (25 MB) 1 1000 * 1024 * 1024 (1000 MB)

MaxEntriesUser 0 (disabled) 0 0xFFFFFFFF

MaxEntriesConfig 0 (disabled) 0 0xFFFFFFFF

LdapKeepAliveSecs 30 5 0xFFFF

TopoForceChangeMins 24*60 (24 hrs) 1 0xFFFFFFFF/1000/60

TopoRecheckSecs 15*60 (15 min) 1 0xFFFFFFFF/1000

TopoImprovementPercent 10 1 100

ConfigDCAffinityMins 8*60 (8 hrs) 1 0xFFFFFFFF/1000/60

UtilityTimeSecs 30 1 0xFFFFFFFF/1000

MaxNotifyLdapConnections 50 1 10000

NumberOfDCsUsedSimultaneously 10 1 10000

NumberOfGCsUsedSimultaneously 10 1 10000

LoadBalanceWeightDNS 100 0 10000

LoadBalanceWeightRoundRobin 10 0 10000

LoadBalanceWeightOutstandingRequests 1 0 10000

LoadBalanceWeightDegradedConnections 1000 0 10000

MinUserDc -1 (disabled) 0 0x7FFFFFFF

LdapConnections 1 1 10

AsyncLdapConnections 1 1 10

PortNumber 0 0 0xFFFF

Page 472: Exchange Server Admin Troubleshooting

Module 7: Exchange Directory Components 69

LockWaitTimeout 10*1000 (10 sec) 1000 10*6*1000 (10 min)

DisasterCleanupThreshold 5 (times) 1 0xFFFFFFFF

StartingSleepPeriod 15 (ms) 1 500 (0.5 sec)

MaxSleepPeriod 1000 (ms) 1 60000 (1 min)

SlowThreshold 2000 (ms) 1 120000 (2 min)

Appendix B: Answers to some of the Labs

Lab Answers: 7.1.3: The config Controller Agreement in conjunction with the SRS are responsible for replicating configuration data. 7.1.4 - On Exchange 2003, it will only output config DC (of which there can only be one in the list at a given point in time). Further, the topology rediscovery option doesn't work. If you run it on an Exchange 2000 server (such as Z2), all options work. 7.2.5: PureExchange 5.5 only if they don't exist on "From Windows Tab" 7.2.6: PureAG only if they don't exist on "From Exchange Tab" 7.2.6: Mixed AGs (having both Exchange 5.5 and Exchange 2000/2003 servers) will appear on both "From Windows" and "From Exchange" tabs. 7.3.5: The student needs to use repadmin /showmeta, as discussed in the previous lesson. 7.4.7: Exchange 2000’s ADC doesn’t scramble the samAccountName like the Exchange 2003 ADC version. This is a GOOD thing. 7.8.2: The default ADC policy is a different class from other Controller Agreements. Its class does not have the msexchhomesyncservice defined. 7.9.2: Z3 does not have an SRS, and only servers with Exchange 5.5 Directory Services may be specified to become directory replication bridgeheads. Learning Measure: e) A and B

Page 473: Exchange Server Admin Troubleshooting

70 Module 7: Exchange Directory Components

Acknowledgments Microsoft Employee

Vincent Yim Harvey Rook, Neil Shipp, Vladimir Grebenik

MSPress books

[Microsoft Corporation. Windows XP Professional Resource Kit, Chapter 6: Security, MS Press, 2001.]

Help files (level 100 only)

[Microsoft Corporation. “To configure a modem for a dial-up connection.” Help and Support Center, Windows XP]

KB article

“XADM: How the Recipient Update Service Applies Recipient Policies” Knowledge Base article 328738 available at http://support.microsoft.com

Existing Whitepaper

[Microsoft Corporation. “Best Practices for System Policies in Windows 2000 Networks.” Whitepaper available at http://www.microsoft.com/technet.]

Page 474: Exchange Server Admin Troubleshooting

Contents

Overview ................................................................................................................. 1 Lesson 1: The Cross-forest Specification................................................................ 2 MIIS Components ................................................................................................... 8 Lab 8.1: Getting to know MIIS 2003 and GAL Sync Management Agent .......... 26 Lesson 3: Cross-Forest SMTP Mailflow.............................................................. 32 Lesson 4: InterOrg Public Folder Replication....................................................... 35 Lab 8.2: Cross Forest Practice............................................................................. 47 Appendix A GAL Sync Log and Error Messages ................................................. 50 Appendix B: GAL Sync Mapping Types .............................................................. 54 Appendix C: GAL Sync Provisioning Rules......................................................... 67 Acknowledgments ................................................................................................. 76

Module 8: Cross-Forest/Multi-Forest

Page 475: Exchange Server Admin Troubleshooting

ii Module 8: Cross-Forest/Multi-Forest

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2003 Microsoft Corporation. All rights reserved. Microsoft, MS-DOS, Windows, Windows NT, Active Directory, ActiveX, Excel, Exchange Server 5.5, Exchange 2000 Server, Exchange Server 2003, Internet Explorer, Internet Information Server, Word are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein (Groupwise, IBM, Lotus cc:Mail, Lotus Notes, Netscape, Oracle) may be the trademarks of their respective owners.

Page 476: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 1

Overview

Topic 1 The cross-forest Specification Topic 2 Microsoft Identity Integration Services (MIIS) 2003 and Global

Address List (GAL) Sync Topic 3 Cross-forest SMTP mail flow Topic 4 Inter-Org public folder replication

At the time of this writing, only RC1 of MIIS 2003 was available for reference. As such, many of the screenshots may be out of date. Additionally, the MIIS product name was renamed prior to the Release To Manufacture RTM in July 2003. Thus, any references to MMS or “Microsoft Meta Directory Services” should be considered MIIS or “Microsoft Identity Integration Services.”

Introduction

Page 477: Exchange Server Admin Troubleshooting

2 Module 8: Cross-Forest/Multi-Forest

Lesson 1: The Cross-forest Specification

Introduction Why do customers deploy multiple forests? Customers deploy multiple forests because it is more secure. Since an Active Directory domain was no longer a true security boundary, forests became the entities in which future topologies were planned.

Customers want administration autonomy and data isolation. They could achieve this in Exchange 5.5, by simply attaching any new Exchange 5.5 site to an existing organization, and thus form a chain of peer sites with a common GAL. Yet trusts were not necessary between each domain that was the realm of an Exchange 5.5 site. Therefore data could be isolated, and administration could be left in the hands of each site/Windows NT domain administrator.

Exchange 2000 introduced an architectural regression. Customers could no longer achieve administration autonomy and data isolation because Exchange 2000 organizations were each bounded by a single forest. This meant that each Global address list was limited to this same boundary. There was no way to attach administrative groups to existing organizations outside of the forest to share a common GAL. Thus, it was extremely difficult to replace traditional site/domain boundaries with multi-domain, single-forest model. For those that tried, they found that transitive trusts and Exchange groups were so invasive that any domain administrators could potentially gain access to other domains’ mailboxes within the forest.

Goals: The goal is to deploy multiple forests to achieve core messaging functionality as it works in the single-forest case. This includes

- Basic mail functionality

- Calendaring

Page 478: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 3

- Common GAL

Important: There are certain features which are not available in Cross-forest deployment, such as the ability to view distribution group membership whose members are stored in another forest. Further, Delegate Access will not be supported across forests. (Currently, a contact cannot be given delegate access to a mailbox). Thus, customers should not expect to achieve full full-feature parity with single-forest installations.

Added benefits:

- Customers have an alternative to using the Active Directory Connector’s inter-organizational connection agreements, which were not supported for coexistence between different organizations.

- Provide an administrative model somewhat similar to peer-to-peer directories as was the case in Exchange 5.5, so that Exchange 5.5 customers may ease into Exchange 2003 without destroying their existing administrative model.

The bare requirements to make a multi-forest topology work for just basic messaging (basically sending messages and no free/busy features) is Directory Synchronization and mail flow. Essentially, a synchronized global GAL needs to be available to all forests and a transport route needs to be in place to allow mail to flow between them.

Since there is no built-in feature to automate sharing of GAL information between forests, the cross-forest spec combines unrelated technologies to achieve the goals. These technologies include

Microsoft Identity Integration Server (MIIS) to achieve directory synchronization.

Customized SMTP connectors for mailflow between forests.

Optional components (IORepl, x-forest movemailbox) may be added to the cross-forest scenario so that the multi-forest deployments come closer to parity with a single-forest scenario.

Components

Page 479: Exchange Server Admin Troubleshooting

4 Module 8: Cross-Forest/Multi-Forest

MIIS 2003 and GAL Sync

Definition This topic covers Microsoft Identity Integration Server version 2003 and the Global Address List Synchronization (Galsync) process, which perform the directory synchronization. Once set up, users in one forest may look up recipients in different forests, and the infrastructure for cross-forest mail flow is established.

History of MIIS •Previous version bought from Zoomit, Inc.

•Version 2.2 is no longer for “sale.” “Sale” is in quotes because if a customer has a Windows 2000 Advanced Server license, then this product is available at no extra cost. However, in order to implement this product they will need to engage Microsoft Consulting Services (MCS) or an Authorized MIIS Partner. Version 2.2 is a high-touch, complex, difficult to deploy product. As it is particularly difficult to setup and configure, in the hands of the untrained, it is possible to do considerable damage to the connected systems. For this reason MMS 2.2 was never actually sold as a “product” in the true sense. It did not have a SKU, and was not on any parts list from Microsoft. However, Microsoft would license the product to customers who had a Windows 2000 Advanced Server license (at no additional cost), and took a services engagement from MCS, or from an Authorized MIIS Partner.

•Version 3.0 (renamed to MMS 2003, then renamed to MIIS 2003) – Rewritten from the ground-up. There is still an MIIS 2003 partner program, and http://www.microsoft.com/windows2000/partners/mms2003.asp lists those that have been trained on MIIS 2003 and are ready to help customers. The 3.0/2003 version is considerably improved, and Microsoft believes customers can use the information (based on scenarios) that ships on the product CD to conduct their own evaluation, design, test and deployment.

Page 480: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 5

Intro to MIIS 2003

Companies often store data about users and objects in multiple data sources – some within Active Directory forests, but others within other types of data repositories. Therefore, the identity of a user could be scattered across several different locations on several different incompatible platforms. Often, companies need to aggregate (combine) that information into a logical view that represents the sum of all the identify information for a given object.

Thus, Metadirectories are utilized for identity management, or for massaging complex data from multiple data sources so that they may be managed easily. Because the concept is so universal, IBM®, Oracle®, Netscape®, and others market their own Metadirectory products. Microsoft’s latest offering is called Microsoft Integrity Information Services (MIIS) version 2003.

MIIS 2003 is a service that collects information from different data sources throughout an organization and then combines all or part of that information into an integrated, unified view. This unified view presents all of the information about an object, such as a person or network resource, that is contained throughout the organization. In most organizations, the sources data is typically stored in different directories, databases, and other data repositories throughout the Information Technology (IT) infrastructure.

After all of the information about a person or resource is combined in the metadirectory, rules can be applied to decide how this information is managed and how changes to this information flow out to all of the directories that are connected to the metadirectory. Therefore, the metadirectory propagates any changes that originate in one directory to the other directories in the organization.

Page 481: Exchange Server Admin Troubleshooting

6 Module 8: Cross-Forest/Multi-Forest

Glossary: To use MIIS to manage data, it is important to become familiar with key concepts and definitions that will assist you in understanding the architecture and function of MIIS.

Objects and Attributes - Each entry in the Metadirectory is an object, of a specific type, which is comprised of a number of attributes. For example, the entry for Kim Rallsis a specific “Person” object, which possesses attributes (e.g. displayName, Title, Department, telephoneNumber) that are defined for the “Person” Object Type. Examples of other object types include “Computer,” “Group,” and “Organizational Unit.” Although Object Types are similar in function to Lightweight Directory Access Protocol (LDAP) Object Classes, they do not exhibit inheritance, and an object may be of only one type.

Metaverse and connected directories - The external sources of data for MIIS are referred to collectively as connected directories. The collection of objects and attributes which are stored and managed in the MIIS system is the metaverse. Because one object in the metaverse is likely to represent entries in more than one connected directory, there will be a conflict if attributes differ for that object in each connected directory. Therefore, it is important to understand which connected directory has authority for each object type and attribute, and therefore which source takes precedence if there is a conflict. This understanding comes primarily from discussions with the business owners of the various systems, and is one of the key requirements for success of a metadirectory implementation project.

Management Agents and Connector Space - The data flow between MIIS and a specific connected directory is defined in a Management Agent, and data flows only when a Management Agent is run. Each Management Agent has a connector space. The connector space is an area (separate from the metaverse) in which a management agent stores holograms from which changes in data-state are calculated. A Management Agent may handle all objects in a connected directory or may be configured to manage only a subset of objects and their attributes. Every object and attribute from the connected directory handled by a Management Agent is represented in the connector space. The Management Agent configuration defines which objects are attributes are synchronized into the metaverse.

Joins and Projection - If an object from a connected directory is represented in the metaverse, it is said to be joined. The process by which the synchronization engine makes the association between an entry in a connector space and a corresponding entry in the metaverse is known as joining. If an entry in the connector space is to be represented in the metaverse, but no corresponding entry in the metaverse can be found by the Join process, a new metaverse object may be created for this entry (according to how the Management Agent is configured). The creation of a new metaverse object is known as projection, and following projection, the connector space entry will be joined to the new metaverse object.

Connectors and Disconnectors - An object in the connector space may or may not be joined to an object in the metaverse. If a connector space object is joined to a metaverse object, it is referred to as a Connector. If it is not joined to a metaverse object, it is referred to as a Disconnector.

Page 482: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 7

GUIDs and Anchor IDs - Each metaverse object is identified by a unique value know as its Globally Unique ID (GUID). The Join between a connector space object and its corresponding metaverse object is achieved by storing the GUID with each connected connector space object.

Similarly, there must be a link between each object in the connected directory and the connector space, so that the Management Agent can manage the entries in the connector space with the corresponding entries in the connected directory. This is achieved through a unique attribute (or a unique combination of attributes) which originates in the connected directory, and is referred to as the Anchor ID.

Provisioning and Deprovisioning - It is possible to define object flow rules in MIIS that imply that the presence of an entry in one connected directory requires the presence of a corresponding entry in another connected directory. For example, the existence of an entry in a Human Resources system (representing an employee) might require the existence of an Active Directory account for this employee. In order to enforce this rule, MIIS can be configured to create and remove entries in a connected directory (for example, in Active Directory). The creation of entries is referred to as provisioning and the removal of entries is referred to as deprovisioning.

Page 483: Exchange Server Admin Troubleshooting

8 Module 8: Cross-Forest/Multi-Forest

MIIS Components

There are three basic areas of storage. The unified view (metaverse), connector space (where data manipulation takes place), and the connected data sources from where data is pulled/pushed. The actual manipulation (joins/adds/extension rules) are performed by the Management Agents. There are different Management Agents for different types of data sources. Pertaining to Exchange 2003, there is the Active Directory Global Address List Synchronization Management Agent (GAL Sync Management Agent) where our attention will focus later.

Connected DirectorySource and/or destination for synchronized objects and attributes

Connector Space (CS)Staging area for all objects that synchronize with MMSPersists the importand export state on each objectEach MA has its own CS

Metaverse (MV)Central (SQL) store of identity informationMatching CS entries to a single MV entry is called “join”

Active DirectoryActive Directory

OracleOracle

IPlanetIPlanet

SAP,JDEdwardsSAP,JDEdwards,,PeoplesoftPeoplesoft, etc., etc.

ConnectedConnectedData SourcesData Sources

MetaverseMetaverse

UserUser

ConnectorConnectorSpaceSpace

Page 484: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 9

Management Agents can also read/write to/from flat files. For Exchange 2003, we use cell-based Management Agents. Regardless of the Management Agent type, the connector space and metaverse components are stored in SQL. Each Management Agent is capable of performing bidirectional replication between a data source and the metaverse. However, at least two Management Agents are necessary for end-to-end replication from a source connected directory to a target connected directory.

Information flow example This graphic shows how data flows from a data source through the metaverse, onward to another system. This is just an example of the ability to merge objects in different data sources which represent the same entity.

MetadirectoryMetadirectoryConnector Namespace Metaverse Namespace

Sue FineNamePost OfficeLocationEmployee #

Sue FineNamePost OfficeLocationEmployee #

Suzan FineFull NameTitleEmployee #

Suzan FineFull NameTitleEmployee #

Forest1Forest1Forest1 111 Full Name Title

Employee #

Full Name Title

Employee #

Suzan FineSuzan FineSuzan Fine

NamePost Office

LocationEmployee #

NamePost Office

LocationEmployee #

Sue FineSue FineSue Fine

222

= Management Agents= Management Agents

Full Name Title

Employee #

Full Name Title

Employee #

Suzan FineSuzan FineSuzan Fine33Full Name Title

Employee #

Full Name Title

Employee #

Suzan FineSuzan FineSuzan Fine

NamePost Office

LocationEmployee #

NamePost Office

LocationEmployee #

Sue FineSue FineSue Fine

Full Name Title

Employee #Name

Post OfficeLocation

Full Name Title

Employee #Name

Post OfficeLocation

Suzan FineSuzan FineSuzan Fine

44

55

555Suzan FineNamePost OfficeLocationEmployee #

Suzan FineNamePost OfficeLocationEmployee #

Suzan FineSuzan FineSuzan Fine

Forest2Forest2Forest2

1. The management agent for the first connected directory creates entries in the connector namespace. The management agent creates these entries (and their attributes) in its portion of the connector namespace.

2. The management agent for the second connected directory creates entries in its portion of the connector namespace. Note that in the preceding illustration, there is an entry for Kim Ralls each of the connected directories. Therefore, there will be two entries in the connector namespace that represent Kim Ralls.

3. The Management Agent for one of the connected directories creates an entry in the metaverse namespace that corresponds to the entry in its connector namespace. Note that an administrator determines which connected directory to use to initially create entries in the metaverse namespace.

4. The management agent for the second connected directory then attempts to match its entry in the connector namespace with the corresponding entry in the metaverse namespace. This action is called a join because each entry in the connector namespace that represents the same real-world object is joined together, or integrated, as one entry in the metaverse namespace.

Page 485: Exchange Server Admin Troubleshooting

10 Module 8: Cross-Forest/Multi-Forest

To ensure that the entries that are joined together in the metaverse namespace represent the same real-word object, you can configure MIIS to use a unique attribute (such as an employee number) as the criteria by which entries are joined.

At this point, as shown in the preceding illustration, Kim Ralls represented by an entry in each of the connected directories, by an entry in the portion of the connector namespace for each connected directory, and by the integrated entry in the metaverse namespace.

5. After entries for the same real-world object are joined together in a single entry in the metaverse namespace, you can determine which attributes the metaverse namespace entry contains and from which connected directory the values for these attributes originate. This is determined by something called attribute flow rules.

When attribute values differ, an attribute flow rule specifies which attribute value (from the metadirectory or from the connected directory) takes precedence. For example, in the preceding illustration, if the values for a common attribute in the entries for Kim Ralls are different, attribute flow rules inform the management agents of the value that takes precedence so that the all the entries for Kim Ralls can be synchronized.

If an attribute does change in the entry in the metaverse namespace, the appropriate management agent makes that change in the entry in the connector namespace and then propagates the change to the connected directory.

Although the Management Agent above was customized for HR databases, a pre-built Management Agent will be available to accomplish replicating global address lists.

Page 486: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 11

The GAL Sync Management Agent

The pre-built GAL Sync Management Agent will allow Exchange administrators to easily configure MIIS 2003 without having the need of an MCS consultant to help design the attribute and rules. Once configured with the data sources (FQDNs of domain controllers), two GAL Sync Management Agents will be used to replicate mail-enabled objects from each Active Directory forest into the metaverse. As in the example above, it also has the ability to join different objects if certain attributes match. In the GAL Sync Management Agent, flows and joins are discussed below:

The flows that are addressed are the one-to-one mappings listed in the following table.

Source Active Directory Metaverse Target Active Directory User Person Contact

Contact Contact_forest Contact

Group Group Contact

Most of the attributes from the Source Active Directory are preserved and copied into the target Active Directory. For example, a telephone number of UserA in Forest1 will be synchronized to the target contacts UserA in Forest2 and Forest3. However, there are some exceptions where additional rules are called. More detailed information on the mapped attributes and rules are provided in Appendix B: Mapping Rules.

The flows that are not addressed are the one-to-one mappings in the following table, and any one–to-many or many-to-one mappings listed the preceeding or following tables.

Source Active Directory Metaverse Target Active Directory User Person User

Group Group Group

Page 487: Exchange Server Admin Troubleshooting

12 Module 8: Cross-Forest/Multi-Forest

Microsoft Metadirectory Services 2003 does not handle multi mastering of these objects in any environment. All target attributes will typically be overwritten by attributes from the source. Exceptions are proxyAddresses and legacyExchangeDN, which have values assigned in the target forest that must be preserved.

The decision to exclude multi mastering arises from the fact that GAL synchronization should not have multi-mastering of objects as it is necessary for exchange functionality to have a single source object.

When MIIS looks for objects in the “Source Active Directory” column, the objects must match certain criteria to be written into the metaverse:

Object Conditions

User The homeMDB or TargetAddress attribute is defined.

Contains at least one proxyAddress attribute value.

The MSExchHideFromAddressList is not set to ‘true’.

Contains a LegacyExchangeDN attribute value.

Contact The TargetAddress attribute is defined.

At least one proxyAddress attribute value is defined.

The MSExchHideFromAddressList is not set to ‘true’.

Must have a LegacyExchangeDN attribute value, but only if the method of creation for the object was projection and not provisioning.

Group The MSExchHideFromAddressList attribute is not set to ‘true’.

A LegacyExchangeDN attribute value is defined.

Join rules are applied to contact objects only. When you run Management Agents, they will search for joins by using a list of metaverse object attributes. The search attempts to match metaverse object attributes with connector space object attributes. If any of the search criteria are met, then a join is established.

When the Apply Rules run profile is run, the following join rules are applied to contact objects only:

If there is more than one candidate in the metaverse for a join, an exception occurs and an error is logged. The Management Agent cannot determine where to flow the data because it cannot determine which object from which to export data.

If a contact that exists in the authoritative contact organizational unit in the source forest is joined to any object in the metaverse, an error is logged. Contacts in the authoritative contact organizational unit in the source forest are used for projection only. An exception does not occur because contacts are not supposed to flow into the target forest. The error is shown in the Event Viewer logging section.

A join to a metaverse object that is already joined is only allowed if the joined metaverse object is a provisioned contact. In this case, provisioning clears up this duplicate. A warning is logged when this join occurs. In all other cases (where the joined metaverse object is a provisioned contact), a

Page 488: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 13

join to an already joined metaverse object is not allowed and an error is logged.

The following table lists and describes the joins between connector space attributes that represent target forest attributes and metaverse attributes. These joins are performed during provisioning, when you run the export run profile and contacts are created in the target forest.

Active Directory Contact Attribute Metaverse Attribute Description

TargetAddress TargetAddress If an Active Directory contact object has the same target address as the metaverse person object, then it represents the same entity and a join occurs.

ProxyAddresses LegacyExchangeDN If the proxy address is an x500 proxy: If the LegacyExchangeDN of a metaverse object matches a value in the proxyAddresses of contact, then it represents the same entity and a join occurs.

ProxyAddresses ProxyAddresses If any of the proxyAddresses values in the target forest match any of the proxyAddress value matches in the metaverse, then the proxyAddresses values in the target forest represent the same entity (and a join occurs) or a miscofiguration.

If an object is joined outside the target forest organizational unit, MIIS 2003 logs a warning with information about the object and requires the user to move the data into the target forest organizational unit.

Requirements For GAL Sync, we require an Exchange 2003 schema. An Exchange 2003 server is also required because we need an Exchange 2003 Site Replication Service (SRS) bridgehead server to bring in HomeMDB and HomeMTA attributes from Exchange 5.5, if it is present in the topology.

We do not require that the organization names of each forest be the same. However, if the organization names are not unique, and objects reside in identically-named administrative groups in each forest, there is the potential for duplicate legacyExchangeDNs.

Sample Installation After SQL and MIIS 2003 have been installed, the following steps are sufficient to get the GALs replicated between two forests that have Exchange 2003 installed. To create the Management Agent that will import objects from forest1, select “Management Agents,” and choose “Create.”

Page 489: Exchange Server Admin Troubleshooting

14 Module 8: Cross-Forest/Multi-Forest

A designer dialog will appear, and the user is prompted through a wizard-like configuration

Page 490: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 15

Page 491: Exchange Server Admin Troubleshooting

16 Module 8: Cross-Forest/Multi-Forest

If either organization uses special address types (Notes/Gwise/custom), the checkbox for “Route mail sent to contacts exported into this forest through the source forest” should be checked. This will be propagated to the target contact in forest2. If forest2 does not actually know or can misunderstand the address, then you get an Non Delivery Receipt (NDR). So checking the box forces the mail to be routed to the source forests, which then decodes the mail from the SMTP proxyaddresses to get the new target address of Groupwise/Notes/etc.

The “Specify an administrative group…” checkbox describes how to generate legacyExchangeDNs for any imported contacts. The choice for legacyExchangeDN will be chooseable from a drop-down list enumerating each administrative group in the connected forest. The legacyExchangeDN will eventually be created in the format “/o=org/ou=sitename/cn=recipients” where only the “sitename” field is customizable. This will affect the contacts’ site membership.

Note that the last seven dialogs in the GAL Sync Management Agent configuration are OK with their defaults, so click “Next” on each one until finished. They are shown for reference purposes:

Page 492: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 17

This dialog is displayed in its unaltered, default state. You can tell if any changes have been made (even those initially hidden by the “Show All” checkbox) by examining this screen.

This dialog is displayed in its unaltered, default state. You can tell if any changes have been made (even those initially hidden by the “Show All” checkbox) by examining this screen.

Page 493: Exchange Server Admin Troubleshooting

18 Module 8: Cross-Forest/Multi-Forest

Page 494: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 19

Page 495: Exchange Server Admin Troubleshooting

20 Module 8: Cross-Forest/Multi-Forest

After selecting the defaults and finishing the Management Agent Designer, create another Management Agent, using the same previous steps as depicted, but rather than supplying credentials/containers from forest1, supply information specific to forest2.

When two Management Agents have been created, go to Tools -> Configure Extensions, and select the checkbox for Enable Provisioning Rules extension. (without this step, metaverse objects will not be projected into target forests).

The Management Agents may now be run. When running the Management Agents, you will be given an option to run certain Run Profiles.

Run Profiles Run Profiles specifies the parameters with which to run Forest1 GAL Sync Management Agent and Forest2 GAL Sync Management Agent. For this sample GAL synchronization scenario, five run profiles are created for each Management Agent. The following list describes each run profile type and the order in which it is run.

Run profile Description

Full Import All specified data flows from the Active Directory data source to the Microsoft Identity Integration Services 2003 connector space and metaverse.

Delta Import All changed data flows from the Active Directory data source to the MIIS 2003 connector space and metaverse.

Export All specified data flows from the MIIS 2003 metaverse and connector space to the Active Directory data source.

Full Synchronization Once all specified data source data is staged, all specified data flows from the MIIS 2003 connector space to the metaverse.

Delta Synchronization Once changed data source data is staged, changed data flows from the MIIS 2003 connector space to the metaverse.

Provisioning is disabled before the Full Import and Delta Import run profiles, and then enabled before running the Run Full Synchronization, Export, and Delta Import run profiles. Provisioning is managed in this order because all objects must be in the metaverse before rules may be applied or erroneous states can result.

When all of the imports and exports have been run, they are provisioned in the metaverse and the end result is that any mail-enabled objects in a source forest (users/groups/contacts) are created as contacts in the target forests, as in the following illustration

Page 496: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 21

Additionally, any master objects that disappear (get deleted) from the source forest will eventually become de-provisioned and disappear from their target forests as well. Provisioning and de-provisioning rules are detailed in Appendix C.

forests,DC=adpreptest2,DC=internal 1> cn: Contoso User; 1> displayName: Contoso User; 1> mail: [email protected]; 1> givenName: Contoso; 1> instanceType: 4; 1> distinguishedName: CN=Contoso User,OU=Contacts from other forests,DC=adpreptest2,DC=internal; 1> objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=adpreptest2,DC=internal; 4> objectClass: top; person; organizationalPerson; contact; 1> objectGUID: d4f5ead6-d6e2-4cd6-b9ef-651cd96ec0ab; 3> proxyAddresses: X500:/o=Microsoft/ou=First Administrative Group/cn=Recipients/cn=ContosoUser; X400:c=US;a= ;p=Microsoft;o=Exchange;s=User;g=Contoso;; SMTP:[email protected]; 1> name: Contoso User; 1> sn: User; 1> uSNChanged: 36706; 1> uSNCreated: 36706; 1> whenChanged: 5/20/2003 5:1:3 Central Standard Time Central Daylight Time; 1> whenCreated: 5/20/2003 5:1:3 Central Standard Time Central Daylight Time; 1> mailNickname: ContosoUser; 1> mAPIRecipient: TRUE; 1> targetAddress: SMTP:[email protected]; msExchPoliciesExcluded: {26491CFC-9E50-4857-861B-0CB8DF22B5D7}; 1> msExchOriginatingForest: darkforest.internal;

Key observations:

You can determine from which forest an object originated by examining the msExchOriginatingForest attribute. In this example, the object came from darkforest.internal.

The legacyExchangeDN from the source forest is added in the form of the X500 address.

Page 497: Exchange Server Admin Troubleshooting

22 Module 8: Cross-Forest/Multi-Forest

From the msExchPoliciesExcluded, MIIS-generated contacts, by default, have the following checkbox unchecked: “Automatically Update E-mail Address based on recipient policy”.

Versioning: Different versions of MIIS:

Microsoft Identity Integration Server 2003, Enterprise Edition: This version has a large variety of Management Agents which allow customers to connect different data sources to provide identity integration/directory synchronization, account provisioning/de-provisioning and password management. This version includes the GAL Sync Management Agent.

Identity Integration Feature Pack for Microsoft Windows Server Active Directory: The feature pack provides fewer management agents, but it still includes the GAL Sync Management Agent. The feature pack is also a free download, unlike the Enterprise Edition. This version was once called the “Standard Edition” before its release.

MIIS 2003 Enterprise or Standard can be used since there are no additional features in the Enterprise release that Exchange 2000 is dependent on.

Similar Management Agents:

Exchange 5.5 Management Agent Exchange 5.5 Bridgehead Management Agent

Do not confuse the GAL Sync Management Agent with the two Exchange 5.5 Management Agents. The Exchange 5.5 Management Agents are only included with MIIS 2003 Enterprise, but are not supported by Enterprise Messaging Support.

Platform Support:

MIIS or Identity Integration Feature Pack may be installed on Windows 2003 Enterprise Edition.

Moving Mailboxes with Migration Wizard: In all versions of Exchange, mailboxes can only be copied across organizations, and there is no built-in function to move mailboxes. Therefore, customers typically resorted to using Exmerge, initially extracting data from the source Exchange organization, and then importing into the target organization. Then, the administrator would delete the mailbox from the source organization.

Though not a significant feature improvement, the Exchange 2003 Migration Wizard has been modified to allow for cross-forest “mailbox moves.” This is somewhat of a misnomer, as the source mailbox is not deleted from its source organization. Nevertheless, Exchange 2003’s Migration Wizard now has a detection mechanism that matches an MIIS-generated contact with a source

Note

Note

Note

Page 498: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 23

mailbox. When such a match is found, the Migration Wizard will automatically delete the MIIS-generated contact and replace it with a mailbox-enabled user object. The purpose of the deletion is to ensure that the identity does not exist more than once in the Global Address list of the target forest. However, the administrator must remember to manually delete the source mailbox. Failure to do so would cause problems with forking of mailboxes (since the original mailbox might still be receiving messages).

Once the source forests’ mailbox-enabled user object is deleted, then MIIS may be turned on again to perform another synchronization – 1) removing the old object from the metaverse and 2) replacing the original mailbox-enabled user object with a mail-enabled contact representing the target forest’s mailbox. This ensures reply-ability to older messages, as well as preventing forking of messages addressed to that mailbox.

List of tested/supported scenarios Single Instance (Hub/Spoke) - Only Single Instance MIIS has been tested. We are only testing where there is a single instance of MIIS among all the forests (hub configuration).

Hub/Spoke configuration is supported.

Page 499: Exchange Server Admin Troubleshooting

24 Module 8: Cross-Forest/Multi-Forest

Active Directory Connector (ADC) - Mixed mode Exchange 5.5/2003 will be supported in the source and target forests. In the source forest, Exchange 5.5 objects will be brought into the source forest Active Directory through the ADC. The ADC-created objects will then be synced to the target forest via MIIS. If Exchange 5.5 is in the target forest, the ADC will bring the MIIS-created objects into the target Exchange 5.5 Directory. The target forest ADC will not do any transformations of the objects when bringing then into the Exchange 5.5 Directory.

All the following topology conditions have been tested, although not necessarily together: Exchange 2003 Native Exchange 5.5/2003 Mixed Exchange 2000 Native Exchange 2000/2003 Mixed Parent Child/Sibling Second Tree Grandchild Windows 2003 Native Forest Windows 2003 Native domain Windows 2000 Native domain Windows 2000 Mixed Domain Windows 2003 Mixed Domain Japanese Domain Controllers German Domain Controllers Japanese Exchange 2003 Japanese Exchange 5.5 German Exchange 2003 Separate SMTP Domain Namespaces Matching SMTP Domain Namespaces Windows 2-Way Trusts 1-Way trusts No Windows Trusts Resource Forests Exchange Tools/Components Tested With: Only SMTP connectivity between Forests ADClean (but some issues found: basically an issue where the object naming conventions due to conflicts are not optimal.) ADMT (but some issues found: IO Repl (InterOrganizational Public Folder Replication Tool

List of untested/unsupported scenarios Hybrid Resource is unsupported.

Mesh configuration of MIIS servers is not supported by the masses; only MCS-approved mesh configurations may be supported.

Page 500: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 25

Resource Forest scenario In this scenario, one forest holds the Exchange mailboxes and the other forest holds the user accounts. The mail-enabled user SID matches MSExchMasterAccountSID of disabled user accounts with mailboxes in another forest. Since there is already a directory replication mechanism (e.g. the intraorg ADC connection agreements), GAL Sync must not be used against these directories.

Looping Forming a replication loop using two non-GAL Sync Management Agents into Active Directory (thereby possibly causing duplicates) will not be supported. The exception to this rule is if there is MCS engagement/approval.

Example of loop:

AD/Exchange 2003 <-- Lotus Notes Connector --> Notes’ Names.nsf <--|

| |

|-------> Gal Sync MA --- Metaverse --- Lotus Notes MA <-----------|

Mail-enabled Public Folders via GAL Sync not tested/supported.

Key Management Server/SMIME certificates via GAL Sync not tested/supported.

Page 501: Exchange Server Admin Troubleshooting

26 Module 8: Cross-Forest/Multi-Forest

Lab 8.1: Getting to know MIIS 2003 and GAL Sync Management Agent

Introduction In this practice lab, you will log onto your virtual PC and briefly examine the MIIS installation.

Log on to the MIISSQLDC Virtual PC 1. From the host computer, navigate to Microsoft Virtual Server’s “Master

Status” page. 2. Power-on the virtual machine called MIISSQLDC. 3. When the Log on dialog box appears, log on to the MIISSQLDC as

Administrator using Password1 as the password. 4. Note that the equivalent of Ctrl-Alt-Delete is RightAlt-Delete, and that you

can toggle full-screen mode with RightAlt-Enter if you are using the Virtual Machine Remote Control Client.

Run Metadirectory Manager 1. Note that there are five main Tools (Windows or Panes), accessible via

toolbar buttons or through the Tools menu; take a look at each: a. Operations (history of Management Agent-run profile execution) b. Management Agents (note the “Search Connector Space” option, and

the panes for displaying Synchronization Statistics and Flow Errors) c. The Metaverse Designer (you can examine the default Object Types

and Attributes) d. Metaverse Search e. The Joiner

Test with simple replication 1. Logon to Forest1DC 2. Open Active Directory Users and Computers. 3. In the Users OU, create a mail-enabled contact

([email protected]) as well as a mailbox-enabled user account ([email protected]).

4. Switch back to the Metadirectory Manager. 5. On the upper-left pane, right-click on the “Forest1DCGALSyncMA”

and choose “Run”. 6. Choose the full import run profile

Page 502: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 27

7. When the Management Agent has finished running, examine the “Synchronization statistics” pane to see what new objects have been added to the metaverse.

8. Does your contact appear to have been added? If not, open the properties of the Forest1 Management Agent, and go to the “Configure GAL” page. Does external.microsoft.com appear in the SMTP-mail suffix managed in the forest? Alternatively, does the Users OU appear in the “Authoritative contacts” container? If neither apply, apply ONE of them.

Examine the MIIS database in SQL Server (you are encouraged to do this from time to time as you practice more)

1. Open SQL Server Enterprise Manager 2. Use the console tree to navigate to the Microsoft Metadirectory Services

database 3. Take a look at some of the Tables by right-clicking the table (e.g.

mms_metaverse or mms_connectorspace) and selecting Open Table and Return all rows

4. Look for the two objects you created, but note that you are advised not to modify these entries directly!

Complete replication into second forest 1. Return to the Metadirectory Manager, and right-click on the Forest2 –

Metaverse Management Agent. 2. Right-click and choose to Run the export profile. 3. Switch to the virtual machine that runs the second forest. 4. From the Administrative Tools menu, load the Active Directory Users

and Computers snap-in. 5. Examine the container structure, and locate where the new contacts have

been replicated. 6. Open the properties of the new contacts, and examine their e-mail addresses

tab. What address types are there?

Page 503: Exchange Server Admin Troubleshooting

28 Module 8: Cross-Forest/Multi-Forest

Q&A/Troubleshooting Question: If I have a problem with synchronization, how would I begin troubleshooting?

Answer: Click on the Synchronization errors for an object

When navigating into the problematic object, select the “Stack Trace” button to view the reason for the sync error.

Question: How can I find errors from previously run profiles?

Answer: Select the operations button, and look at the history of previous Runs.

Page 504: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 29

Question: How can I tell which objects are going to be exported to other forests?

Answer: After you run a “Full Import” on the Management Agent connected to the source forest, look at the synchronization statistics pane (lower left-hand corner of Metadirectory Manager).

You should see “Outbound Synchronization” - <name of target MA>. Underneath that, click on “export attribute flow,” which will be clickable if there are objects pending export into the target forest. You are then presented with a dialog of all the objects, and navigating into their properties will show which attributes will be created/modified.

Page 505: Exchange Server Admin Troubleshooting

30 Module 8: Cross-Forest/Multi-Forest

Question: How do I view objects in a connector space, in general?

Answer: There are a couple of ways:

• View the "mms_connectorspace" table in SQL enterprise manager.

• In metadirectory manager, highlight the Management Agents option at the top of the screen. Then, on the upper-right hand corner, select "Search Connector Space" (To see which of these objects exist in the metaverse, sort by the “Connector” column for values with the “Yes” value.

Question: Under the “Configure GAL” page, what does "Mail to contacts exported from this forest will be routed through this forest" do?

Answer: When checked, it changes how the targetaddress of a contact gets constructed. (Need better answer)

Question: Under the “Configure GAL” page, when selecting the option “Specify an administrative group to enable interoperability with Exchange 5.5 and Active Directory Connector environments,” if I choose “edit,” how come I do not see my admin group the list of administrative groups?

Answer: Customers will often rename their administrative groups after switching to native mode. In LDP or ADSIEdit, search the admin groups’ “Displayname” entries to correlate with the desired admin group.

Question: What happens if same contact is created in both forests, before MIIS is used to sync?

Answer: Join rules apply here, provided that one of the contacts resides within an “authoritative” contacts container, as configured within the “Configure GAL” property sheet. Since the contact will have the same proxy address in both forests, MIIS will know how to "join" the two objects in both forests as the same object in the metaverse.

Question: If there is an error in sync, I can click the "preview” button. But how can I see the preview button for an object that doesn't have an error?

Answer: Search connector space -> open the object found, and select preview button on lower-left dialog. Alternatively, after you have executed a run profile (such as an import), you can preview objects within the underlined statistics in the Synchronization Statistics pane.

Page 506: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 31

Question How can I automate the schedule of the Management Agents so that it acts like an Active Directory connector?

Answer: There’s no integrated functionality to do this. A VB script can spawn WMI code to do this. However, the script would need to be scheduled using the AT command.

Question: Is the GAL Sync Management Agent designed to create objects in different OU’s within a target forest?

Answer: No, all mail-enabled objects from multiple source forests will get dumped into a single OU in a target forest. (Is there an exception?)

Page 507: Exchange Server Admin Troubleshooting

32 Module 8: Cross-Forest/Multi-Forest

Lesson 3: Cross-Forest SMTP Mailflow

This topic covers mailflow and configuration of SMTP connectors used with the cross-forest spec. Additionally, conditional forwarding, a new Windows 2003 feature in DNS, may be used.

Scenario 1: No two Exchange organizations share the same SMTP namespace SMTP connectors are not required if each Exchange organization can resolve the other through DNS. However, if the two forests are separate divisions within the same company, each company division may not be resolvable through internet DNS servers. For that reason, or SMTP connectors may be set up in this fashion:

Introduction

Page 508: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 33

This illustrates one SMTP connector in the source forest, where “Darkforest” is a forest outside of the current forest where the SMTP connector is created. An SMTP connector in the Darkforest organization would also need to be created pointing to the source forest.

Scenario 2: One or more Exchange organizations share SMTP namespace If the SMTP namespaces of each organization is the same, then routing will be difficult since each Exchange organization will believe that it is authoritative for that SMTP domain, and will queue for remote delivery to another organization.

In this circumstance, a secondary proxyaddress must be added to all mail-enabled users, contacts, and groups to distinguish forest membership and allow SMTP routing outside of the forest.

Example: Assuming that domain.com is the shared SMTP domain for two forests, the following steps are used

1. On the recipient policy of forest1, modify the recipient policies to give all GAL objects an additional (secondary) proxyaddress to have a unique SMTP domain that is not present in any other forest. You could use the subdomain approach (SMTP: forest1.domain.com) or anything that does not exist in DNS (SMTP: af39417321095fjlk.random).

2. Repeat the last step for forest2, by modifying the recipient policies to give all GAL objects an additional (secondary) proxyaddress. (SMTP: forest2.domain.com or something random)

3. In each forest, create SMTP connectors with address spaces specific to the opposite forest. Refer to Scenario 1’s screenshots.

Page 509: Exchange Server Admin Troubleshooting

34 Module 8: Cross-Forest/Multi-Forest

4. When you create your GAL Sync Management Agents, on the “Configure GAL” dialog page, add on the forest-specific SMTP namespace the SMTP domain of the secondary proxy address.

The checkbox for “Route mail sent to contacts exported into this forest through the source forest” is meant to cover the scenario where the targetaddress on a source contact is non-SMTP (Groupwise/Notes/x400). This will be propagated to the target contact in forest2. If forest2 does not actually know or can misunderstand the address, then you get an NDR. So checking the box forces the mail to be routed to the source forests, which then decodes the mail from the SMTP proxyaddresses to get the new target address of Groupwise/Notes/etc.

5. When you run the export, GAL Sync will set the targetaddress of the generated contact to this secondary proxy address. Since the targetaddress will not be the same as the authoritative SMTP domain of the sending organization, messages to contacts will be queued for remote delivery.

Possible issues As in any case of a large directory, it may be possible for mail sent to SMTP addresses that do not exist to loop back and forth between organizations’ SMTP connectors because each directory might have an object pointing to the opposite organization. One should check for such contacts in each directory that might cause a loop.

When forests have matching legacyDNs, there may be an issue with how GAL Sync “joins” the two objects in the metaverse. Therefore, one should tread lightly whenever setting-up MIIS between two organizations whose organization names and site names match.

Page 510: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 35

Lesson 4: InterOrg Public Folder Replication

This topic covers Inter-organizational public folder replication. It is designed to replicate public folders between different Exchange organizations. It will also replicate free/busy information between organizations, granted that each organization has either a mail-enabled contact or custom recipient representing the source organization.

History Also known as Exchsync, IO Repl

Released with Exchange 5.5 SP2, and also delivered through Exchange 2000 CD.

Newest incarnation available via Web download (Exchsync package) folder from Web release (ExAllTools) tools. This is also expected to be updated in the future, through Exchange 2003 Service Pack releases.

Components The tool consists of two programs: the Configuration program (exscfg.exe), and the Replication service (exssrv.exe). The Configuration utility creates a configuration file for setting the replication frequency; logging options; folders to be replicated; and accounts to be used. This configuration file is used by the Replication Service to continuously update information between two servers in different organizations. The Exchange Replication Service, using a configuration file created by the Replication Configuration utility, continuously updates information from one server (designated as the Publisher) to one or more Exchange servers (designated as Subscribers). Schedule+ Free/Busy information is replicated from Publisher to Subscriber only. For this reason, you must have two free/busy sessions to bi-directionally update free/busy information. Public folders can be replicated from Publisher to Subscriber or bi-

Page 511: Exchange Server Admin Troubleshooting

36 Module 8: Cross-Forest/Multi-Forest

directionally. You can configure the replication frequency, as well as the logging of message and folder replication, and how much processing power you want devoted to the replication process.

Areas of usage This tool is not supported between two servers that are using different languages. It only supports replication between servers using the same language (for example, English-to-English, French-to-French).

For the two-forest scenario, here are the combinations of organization types where IORepl may be used:

Pure Exchange 5.5 Pure Exchange 2000

Pure Exchange 2003

Exchange 2000/2003

Exchange 5.5/2000/2003

Pure Exchange 5.5 Use earlier version Use earlier version Yes Yes Yes Pure Exchange 2000 Use earlier version Yes Yes Yes Yes Pure Exchange 2003 Yes Yes Yes Yes Yes Exchange 2000/2003 Yes Yes Yes Yes Yes Exchange 5.5/2000/2003 yes Yes Yes Yes Yes

The above is a guideline, but customers choosing not to follow the table above may attempt to use the Exchange 2003 IORepl tool against Exchange 2000 and Exchange 5.5 organizations. Even if replication appears successful, unpredictable behavior might occur as scenarios outside of the above table have not been tested.

Requirements For replication of free/busy information, MIIS and GAL Sync must have already been configured and run so that cross-forest contacts already exist to correspond to external free/busy entries.

The tool must be installed on a computer that has Exchange 2003 Exchange System Manager installed on it. It is possible to install on a workstation/professional Windows platform that only has Exchange System Manager installed. Reference the Exchange 2003 Deployment Tools for more information on installing on creating an Exchange System Manager-only install. Note that Outlook is not supported and must not reside on the same machine on which Exchange System Manager is installed.

A VPN or open-RPC connection must be available between each Exchange organization/Active Directory forest. This is so that MAPI connections may be established.

Page 512: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 37

Although you may use the tool to connect to Exchange 5.5 and Exchange 2000 organizations, you must use the IO Repl versions off the Exchange 2003 Service Packs and Web release. Do not use previous versions.

How to install Installing the InterOrg Replicator Utility for use with Microsoft Exchange Server consists of the following steps:

1. Preparing the Publisher 2. Preparing the Subscribers 3. Installing the InterOrg Replicator utility files 4. Creating a Configuration file 5. Setting up the Replication Service

Preparing the Publisher The first step in configuring the InterOrg Replicator utility is preparing an Exchange server to be a Publisher. The Publisher collects information from an Exchange organization, packages it, and sends it to Subscriber Exchange servers outside the Exchange organization based on a schedule you create. The publisher can be considered as the source server the information is being replicated from.

To prepare the Publisher, you must create a service account and mailbox for the utility to use during the replication process. You also must assign the appropriate permissions to that service account and mailbox, and create a public folder for the utility to use during replication.

It is important to understand that the service account and mailbox that you create must be listed as an owner of each public folder and subfolder you want to replicate, on either the Publisher or the Subscriber. This enables the utility to replicate Anonymous and Default permissions from one organization to the other. Use Microsoft Outlook to change the ownership and the permissions of public folders.

To prepare the Publisher server for InterOrg Replication:

1. Create a Windows NT account and an associated Exchange mailbox for the utility to use as a service account.

2. Using Microsoft Outlook, add the service account mailbox that you created as an owner for every top-level folder and subfolder you want to replicate.

3. Using Microsoft Outlook, create a public folder named ExchsyncSecurityFolder in the root Public Folder and grant Owner permissions to the service account mailbox that you created. Do not specify any Default or Anonymous permissions on this folder; it is used by the Replication Service for additional security and must be present on both the Publisher and Subscriber servers.

Preparing the Subscriber A Subscriber is an Exchange server where you want to replicate information to using the InterOrg Replicator utility. To configure a Subscriber, you must create a Windows NT account and an associated Exchange mailbox that the

Page 513: Exchange Server Admin Troubleshooting

38 Module 8: Cross-Forest/Multi-Forest

utility can use as a service account. Additionally, you must create the public folders that the utility needs for the replication process.

To prepare the Subscriber server for InterOrg Replication:

1. Create a Windows NT account and an associated Exchange mailbox for the utility to use as a service account.

2. Using Microsoft Outlook, create a top-level folder for every portion of the folder hierarchy you are replicating. The utility will create subfolders automatically.

3. Using Outlook, grant Publishing Editor permission for each top-level folder to the service account mailbox that you created.

4. Using Microsoft Outlook, create a public folder named ExchsyncSecurityFolder off of the root public folder and grant Owner permission to the service account mailbox that you created. Do not specify any Default or Anonymous permissions on this folder; it is used by the Replication Service for additional security and must be present on both the Publisher and Subscriber servers.

Note: A Server can be both a publisher and subscriber if you are replicating both ways.

Installing the InterOrg Replicator Utility Files The following files are located in the EXCHSYNC directory on the Web Release download package (available by RTM). Additionally, you may find it on Exchange 2003 Service Pack CD distributions:

Exscfg.exe, the Microsoft Exchange Replication Configuration Program.

Exssrv.exe, the Microsoft Exchange Replication Service.

1. Create a working directory for the utility to use (C:\Exchsync, for example).

2. Copy the files Exssrv.exe and Exscfg.exe into your working directory.

Creating a Configuration File In order to set up replication, you must create a configuration file. The configuration file will contain replication sessions. Each session will be either a free/busy session or a public folder session.

To create a configuration file for free/busy replication:

1. Double-click Exscfg.exe. 2. On the Session menu, click Add. 3. In the Select Session Type dialog box, select Schedule+ Free/Busy

Replication

Page 514: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 39

4. Type a display name for the session.

5. Type the Publisher and Subscriber server names, and the service account mailboxes that you created for each.

Click Advanced and type the Windows NT domain, service account, and password for each of the Publisher and Subscriber accounts.

6. Click on the schedule button and create a replication schedule that fits your needs.

Page 515: Exchange Server Admin Troubleshooting

40 Module 8: Cross-Forest/Multi-Forest

7. Click on the schedule button and create a replication schedule that fits your needs.

8. Choose the sites for which you want replicate free/busy information for. The default is all sites available

9. Click OK to add the session to the configuration file and then save.

NOTE: Log files report when the service starts or stops, any errors it encounters, and statistical information for each session (for example, number of messages and folders replicated).

NOTE: For each mailbox in the Publisher server that you want to replicate free and busy information to, a corresponding custom recipient must exist on the Subscriber server. The SMTP address of the mailbox is the unique key that is used to match mailboxes to custom recipients.

To create a configuration file for public folder replication:

1. Double-click Exscfg.exe.

2. On the Session menu, click Add. 3. In the Select Session Type dialog box, select Public Folder Replication.

Page 516: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 41

4. Type a display name for the session.

5. In the Maximum Task Number dialog box, enter the number of threads to be used for replication. In the Schedule dialog box, enter the time, day, and frequency for the replication session. If you want the utility to write a log during the replication process, click Logging and set the appropriate parameters.

6. Type the Publisher and Subscriber server names, and the service account mailboxes that you created for each.

7. Click Advanced and type the Windows NT domain, service account, and password for each of the Publisher and Subscriber accounts.

8. Click Folder List to choose which folders you want to replicate. In the Session Folder List dialog box, select the folder or folder hierarchy on the Publisher that you want to replicate, and then select the destination folder on the Subscriber.

9. Click the arrow button once to replicate public folder information only from the Publisher to the Subscriber. Click again to toggle bi-directional replication. You can also toggle on if subfolders replicate, deletions replicate, and default or anonymous permissions replicate.

Page 517: Exchange Server Admin Troubleshooting

42 Module 8: Cross-Forest/Multi-Forest

10. Click OK to add the session to the configuration file and save.

To set up the Replication Service: 1. Double-click Exssrv.exe. The first time you run Exssrv.exe, click

Install. 2. Type the Windows NT account name and password for the account that

will run the service. The account should have the rights to log on locally and can run as a service. The account should be entered with domain/username.

3. Type the path and file name of the configuration file you created. 4. Specify whether or not you want the service to automatically start

when you turn on the computer.

Page 518: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 43

5. After you have installed the Service, click Start or start it from the Control Panel.

Page 519: Exchange Server Admin Troubleshooting

44 Module 8: Cross-Forest/Multi-Forest

Troubleshooting/Q&A There are a number of steps you can take if you encounter problems with the InterOrg Replicator tool. 1. Enable logging on the Public Folders or Free/Busy Session Configuration

tabs. 2. Check the Ex00yymmdd.log and Ex01yymmdd.log log files for any errors. 3. Use the application Event Viewer to make sure there are no errors.

Configure the Event Viewer with a large size (> 5MB), and configure it to wrap as needed.

4. Verify that you have granted the appropriate permissions to root folders and the ExchsyncSecurity Folder or the free/busy folder. Verify the folders are visible and that the account used has access to read and write to the folders. If not, you will receive the following error message.

Event ID 116 Source Exchsync Type Error Category None Mailbox user (username) does not have enough security to replicate to server [servername] on session 'Session Name'. WARNING: Create subfolder access is not available to '[SERVERNAME]\Foldername, skipping. WARNING: Write access is not available to '[SERVERNAME]\ Foldername, skipping. WARNING: Write access is not available to '[SERVERNAME]\ Foldername \ Foldername,skipping. WARNING: Read access is not available to '[SERVERNAME]\ Foldername, skipping. WARNING: Read access is not available to '[SERVERNAME]\ Foldername \ Foldername, skipping.

5. Verify that for each mailbox on the publisher server that you want to replicate free and busy information to, a corresponding custom recipient exists on the subscriber server. If none exists, create one. The SMTP address of the mailbox is the unique key that is used to match mailboxes to custom recipients. If a corresponding custom recipient does not exist, no free and busy information will be published.

6. If no free and busy information is displayed after you verify that the custom recipients do exist, quit, and restart your client. This forces your free and busy information to be written to the server. The next replication cycle will then publish your free and busy information.

Other errors: A 115 error event might indicate (in its description) that the ExchsyncSecurityFolder cannot be located. Verify the name matches exactly and there are no trailing or leading spaces in the name. A 118 error event is a communications or authentication error. The tool has

Page 520: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 45

been unable to contact the server in question, or could not connect via MAPI. Check proper name resolution, network connectivity (trace route and ping), and make sure you have the correct version of MAPISVC.INF and that it is not damaged. If it is a permissions error, you may receive this text in the event description: “Unable to create a temporary MAPI profile to server [SERVERNAME] using mailbox [ioreplaccount] under Windows NT username [ioreplaccount] and domain [ADPREPTEST2] on session 'pf gizmos to dark'.”. Even if you were able to connect using the exscfg.exe utility, the service itself will not allow the account specified during exscfg.exe to log on directly to an Exchange server. In this case, grant the privilege of the specified account to log on locally, or add the account to the server operators group.

A 120 error event is a communications error. We have been able to contact the remote server but we failed to make a connection. Again check network connectivity (trace route and ping) making sure of no packet loss. Verify you have the correct user name and password for the service account mailbox.

Question: Can I install the service through terminal services? Answer: No. When you install the service, it creates an exchsyn.ini file in the %systemroot% directory. This variable will not be the same for a user logged into the terminal session as it is for when a service starts. See Q271813 for more information.

Question: The service is using the credentials of the service to login rather then the credentials specified in the configuration file?

Answer: The Public folder replication tool must be running on a machine that has Exchange System Manager installed, but not Outlook. This has to do with the EMSABP32.DLL. During the SP2 to SP3 timeframe of Exchange 5.5 these DLLs were given to Outlook to develop themselves. During this transition time a change was made to this DLL that allows a way for a service to pass a different set of credential when using MAPI. This change never made it to the Outlook version of the DLL and thus only exists in the Exchange version. The interorg tool uses this functionality and since it does not exist in the Outlook version of the DLL it uses the wrong credentials to log in. If you install Outlook 2000 or greater on the server, it will redirect calls to the Outlook version of the DLL and you again will be passing the wrong credentials. In the past this tool has worked on systems without Exchange server install. This is because at the time of this tools release the only versions of Outlook available were 98 and older. If you installed Outlook 98 first, then the Exchange administrator from SP3, the correct DLL would be copied over top in the system32 directory and thus the tool would work. Installing just the admin program is not enough on its own, as it does not update the MAPISVC.INF file with a transport that the tool can use and it will fail with a 118 event instead.

Question: ExchSync tool will not replicate and reports the following error in the log file “ERROR: Unable to import message change [SK= 6A69F6552DD203D5D6000001231] to folder ‘EX:/o=orgname/ou=Sitename’

Page 521: Exchange Server Admin Troubleshooting

46 Module 8: Cross-Forest/Multi-Forest

on server [Servername], message previously existed but has been deleted.” (s0x010904700128)

Answer: At some point the free/busy messages in the subscriber org for the publishing org users were deleted. The error is produced because the publish org still holds a copy of the free/busy message(s). Since we are using the public folder APIs for this replication it will not allow these messages to be replicated back in, as they have the same message ID. New free/busy messages need to be created in the publishing org in order for replication to continue. This can be done by using the fbscrubber utility to clear out all the free/busy information in the publishing org and then making a change to every user’s calendar so it updates the free/busy information again in the publishing org. (The process of making a change to every user’s calendar may be automated through the fbupdate.exe utility.) Then, exchsync finds the new message IDs and pushes them to the subscriber org.

Question: What if free/busy information for new Custom recipient in the subscribing Org does not get updated?

Answer: The tool keeps track of changes and of what users it has replicated information for in the past. This way it does not have to replicate everything every time. If a mailbox in the publishing Org does not match a Custom recipient in the subscribing org it is marked to do not replicate this mailbox again. If a new Custom recipient is created for this user in the subscribing org after this, it still will not replicate, as it was already marked. This information is kept in a “dat” file in the working directory. If you delete this “dat” file, the interorg tool will perform a complete sync the next time and pick up the new custom recipient.

Page 522: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 47

Lab 8.2: Cross Forest Practice 81

Exercise 1: Observing attributes replicated across forests 1. Create a new mailbox-enabled user in forest1: Jason Carlson 2. Wait until this new user obtains its proxyaddresses. (You may speed-up the

Recipient Update Service (RUS) stamping by updating the domain RUS.) 3. On the MMC box, run a delta import for the forest1MA. 4. In "synchronization statistics" window, click on "Adds" 5. Click on Jason Carlson and choose "Properties" 6. Click on "Preview" button, and then "generate preview" 7. To determine the attributes that would be written by MIIS when it is

exported to Forest2, select "Connector Updates" and on the right-hand side, double-click on the entry that has an "added" operation. An entry on the left-hand side will become highlighted, so expand highlighted object and click on "Export attribute flow." The right-hand pane should display the attributes written by MIIS.

When viewing the attribute flow, do not pay attention to "Initial value" as it is either empty, or inaccurate. Instead, the "Final value" is the correct value to observe

8. Proceed with an export by running the forest2MA. 9. Confirm that the contact is created in the "Contacts from other forests OU"

Exercise 2a: Moving a mailbox object between forests. 1. Open migration wizard on forest2dc. (Start programs Microsoft

Exchange Deployment Migration Wizard) Note that this has changed locations from Exchange 2000.

2. Choose the option: migrate from another Exchange server. 3. For the source server, deselect the checkbox for “5.5 server” and specify

forest1dc, with the administrator account “forest1\administrator” and password of “Password1”

This lab has been set up with a conditional forwarder (configured within DNS). Without it, you would receive an error code 1355 with error message "Unable to logon to the server. Please verify the server name, port, account name, and password."

4. Choose the "Users" OU for placement of the target user accounts in forest2. 5. Finish the migration wizard.

Note

Note

Page 523: Exchange Server Admin Troubleshooting

48 Module 8: Cross-Forest/Multi-Forest

6. In forest2, refresh your view on the "Users OU" and confirm that Jason Carlson appears as a mailbox-enabled user account. Do you notice anything unusual about its proxyaddresses?

(The x500 address is added, and it is used for reply-ability)

7. Refresh your view on the "Contacts from other forests" OU, and note that the Jason Carlson contact object no longer exists.

• At this stage, the administrator should delete (or at least mailbox-disable) the object from the source forest (Forest1) and synchronize the Management Agents to populate a mail-enabled contact in forest1 representing the moved mailbox. However, if the deletion is skipped, and instead Management Agents are synchronized, you will receive the error message in the following steps.

8. Run a delta import for the forest2MA. You should see "Synchronization errors" on the lower-left pane.

9. Click on the error and then click on the "Stack Trace" button. Read the error message.

Exercise 2b: Correcting problem with conflicting authoritative objects:

1. Delete Jason Carlson from the source forest. 2. Run a delta import on forest1MA to pick-up the deletion. 3. Run a delta import on forest2MA to get rid of the thrown exception, and

mark the forest2 object as authoritative. 4. Run an export on forest1MA to project the moved object into forest1 as a

contact. The GALs are now consistent.

Exercise 3: Does deleting target forest contacts cause the source contacts to become deleted?

No, this is not like the ADC.

1. To test, delete the “Laura Norman” contact from forest2. 2. Run an import of forest2MA, and you will see that a "delete" for her contact

is detected. This does not mean that the corresponding object is deleted from forest1.

3. Run an export of forest1MA, and you can verify that Laura Norman user account is not deleted because it is authoritative.

4. To re-add any contacts accidentally deleted from forest2, simply run an import of the forest2MA (which we've already done), and an export of forest2MA.

Page 524: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 49

Exercise 4: Testing mailflow 1. Log onto Laura Norman’s mailbox in Forest1 using Outlook Web

Access.

2. Verify that mailflow works to a user in the opposite forest.

Exercise 5: Sharing address spaces 1. To each exchange org's recipient policy, add an additional address

(Contoso-shared.com). In the recipient policy, ensure that Contoso-shared.com is not selected for “This Exchange Organization is responsible for all mail delivery to this address.”

2. Update each forests’ domain RUS. 3. Both Management Agents must now be modified. Open their properties, and

on the "Configure GAL" dialog, edit the SMTP addresses managed by each forest.

4. Run at least two full imports on each connected directory so that the synchronization rules will pick up the new changes to the user objects. Then run exports on the management agents.

5. Configure SMTP connectors as defined by the cross-forest module. 6. While logged-on as Laura Norman, send an SMTP message to

[email protected]. Note that Jason Carlson’s mailbox is in forest2, so Exchange 2003 must resolve and route to the other organization.

Page 525: Exchange Server Admin Troubleshooting

50 Module 8: Cross-Forest/Multi-Forest

Appendix A GAL Sync Log and Error Messages The following table lists the common Microsoft Metadirectory Services 2003 log and error messages associated with GAL synchronization, along with the troubleshooting prescriptions.

Table A.1 Log and Error Messages Messages Action Join messages

Contact in connector space tries to join to two possible objects in the metaverse and neither of them is a provisioned contact: There are multiple objects representing the

same entity in the metaverse, they are

<forest name, DN of object 1, forest name, DN

of object 2…etc>. Microsoft Identity

Integration Services 2003 cannot join, please

examine the object <DN of object that failed

the join, forest name> and the join rules and

determine what attribute(s) caused the

conflict. Please clean up the object to

allow it to be propagated to other forests if

it is authoritative or use Account Joiner to

connect it to a metaverse object to have MIIS

maintain it.

Examine both metaverse objects, and determine which one is the nonauthoritative one. Ensure that the nonauthoritative one, that reports that it is authoritative because it is a user, group or in the authoritative contacts container, is appropriately tagged as nonauthoritative.

After deletion, run import, full synchronization, and export to make the change effective.

If both are authoritative, determine what attribute caused the target object to join to two objects, and appropriately update the target or source objects to remove this join criteria. For example, if the overlapping proxyAddresses caused the problem, delete the proxyAddress that is common to all three objects from the contact that should not contain it.

Contact in connector space tries to join to two possible objects in the metaverse and one of them is a provisioned contact:

There are multiple objects representing the

same entity in the metaverse, they are

<forest name, DN of object 1, forest name, DN

of object 2…etc>. One of these objects:

<forest name, DN of object> was created by

Microsoft Identity Integration Services 2003.

This object will be deleted and the join will

take place with this object <forest name, DN

of object>.

No action required

Contact from the authoritative contacts organizational unit attempts to join when it should be projected:

There are two authoritative objects

representing the same entity in the

metaverse, they are <forest name, DN of

object 1, type of object; forest name, DN of

object 2, type of object…etc>. One of these

objects <forest name, DN of object from

forest where contact in connector space is

Examine both metaverse objects, and determine which one is the nonauthoritative one. Ensure that the nonauthoritative one that reports that it is authoritative because it is a user, group or in the authoritative contacts container is appropriately tagged as nonauthoritative.

After deletion, run import, full synchronization, and export to make the change effective.

Page 526: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 51

from forest indicated by contact forest> will

be propagated to other forests until this

conflict is resolved by removing or modifying

one of the objects so that they no longer

collide.

Provisioning messages

User - A contact for this user object <CN of

Person> object called contact <DN of contact>

already exists in forest <target forest

name>. If you would like to preserve this

contact and have us manage it, please move

the contact into the Synchronization

organizational unit. If you would like us to

create a new contact and manage it, please

delete this one.

User - Multiple contacts for this user object

<CN of Person> object already exist, they

are: contact 1<DN of contact> already exists

in forest <target forest name> | contact 2

<DN of contact> already exists in forest

<target forest name>. If you would like to

preserve a contact and have us manage it,

please move the contact into the

synchronization organizational unit and

delete all others. If you would like us to

create a new contact and manage it, please

delete all other contacts.

Contact - A contact for this contact object

<CN of contact_forest from forest <source

forest name>called contact <DN of contact>

already exists in forest <target forest

name>. If you would like to preserve this

contact and have us manage it, please move

the contact into the synchronization

organizational unit. If you would like us to

create a new contact and manage it, please

delete this one.

Contact - Multiple contacts for this contact

object <CN of contact_forest> from forest

<source forest name> already exist, they are:

contact 1<DN of contact> already exists in

forest <target forest name> | contact 2 <DN

of contact> already exists in forest <target

forest name>. If you would like to preserve

a contact and have us manage it, please move

the contact into the synchronization

organizational unit and delete all others.

If you would like us to create a new contact

and manage it, please delete all other

contacts.

Group - A contact for this Group object <CN

of Person> object called contact <DN of

contact> already exists in forest <target

Page 527: Exchange Server Admin Troubleshooting

52 Module 8: Cross-Forest/Multi-Forest

forest name>. If you would like to preserve

this contact and have us manage it, please

move the contact into the synchronization

organizational unit. If you would like us to

create a new contact and manage it, please

delete this one.

Group - Multiple contacts for this Group

object <CN of Person> object already exist,

they are: contact 1<DN of contact> already

exists in forest <target forest name> |

contact 2 <DN of contact> already exists in

forest <target forest name>. If you would

like to preserve a contact and have us manage

it, please move the contact into the

synchronization organizational unit and

delete all others. If you would like us to

create a new contact and manage it, please

delete all other contacts.

Deprovisioning messages

When any object is deprovisioned that is not in the Synchronization organizational unit, the following message is written to the log:

The source object <source object CN>

associated with this target object <target

object name> from forest <target forest name>

has been deleted, the target object should

also be deleted but it is outside the

SYNCHRONIZATION ORGANIZATIONAL UNIT. Please

delete the object manually or move it into

the SYNCHRONIZATION ORGANIZATIONAL UNIT.

Projection messages

User found in the Synchronization organizational unit:

A user object <connector space entry of

object = DN of object, MA name> has been

found in the synchronization organizational

unit, this will not be projected, please move

it out of the synchronization organizational

unit if you want this object to propagate to

other forests.

Group found in the synchronization organizational unit:

A group object <connector space entry of

object = DN of object, MA name> has been

found in the synchronization organizational

unit, this will not be projected, please move

it out of the synchronization organizational

unit if you want this object to propagate to

other forests.

Attribute Flow Messages

When target address cannot be created for a user or a group because there is no match between proxyAddresses and the first SMTP mail domain suffix provided by the administrator, Microsoft Identity Integration Services

Page 528: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 53

2003 throws an exception and logs an error.

Other messages

If changes are made to the XML configuration files to include new objects but not the custom extensions, Microsoft Identity Integration Services 2003 throws an UnexpectedDataException and logs it.

If a user adds a label for a scripted flow rule, join rule, but does not update the custom extensions, Microsoft Identity Integration Services 2003 throws an extension.

Adding an object and using existing attribute flow rules will throw an exception because the administrator must confirm that the flow rules apply to this object type.

Page 529: Exchange Server Admin Troubleshooting

54 Module 8: Cross-Forest/Multi-Forest

Appendix B: GAL Sync Mapping Types

Attribute mapping is defined in one of two types. The following table describes the three attribute mapping types.

Table A.2 Attribute Mapping Types Mapping Type Description Direct Defines a direct relationship between a

single source attribute and a single destination attribute. The destination attribute is assigned the value of the source attribute which cannot be modified by a custom extension. An example would be to map employeeID to userID

Custom Defines a direct relationship between multiple source attributes and a single destination attribute. An example would be to map firstName and lastName to fullName

Constant Defines a single destination attribute and the constant string value that the attribute will have.

Import Attribute flow: Source Forest User to Metaverse Person The following table illustrates the mapping of the Active Directory user object attributes to the metaverse person object attributes.

Source Forest User Object Attribute Metaverse Person Object Attribute Mapping Type

LegacyExchangeDN LegacyExchangeDN Direct

Mail Mail Direct

MailNickname MailNickname Direct

CN CN Direct

TelephoneNumber TelephoneNumber Direct

TargetAddress TargetAddress Rules extension

MSExchMasterAccountSID MSExchMasterAccountSID Direct

MSExchHideFromAddressLists MSExchHideFromAddressLists Direct

HomeMDB homeMDB Direct

ProxyAddresses proxyAddresses Direct

DisplayName displayName Direct

DistinguishedName distinguishedName Direct

SN SN Direct

Page 530: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 55

C C Direct

Company Company Direct

Department Department Direct

Division Division Direct

EmployeeID employeeID Direct

EmployeeType employeeType Direct

GivenName givenName Direct

Manager Manager Direct

O O Direct

Title Title Direct

UserCertificate UserCertificate Direct

UserSMIMECertificate UserSMIMECertificate Direct

streetAddress streetAddress Direct

St St Direct

postalCode postalCode Direct

Co Co Direct

physicalDeliveryOfficeName physicalDeliveryOfficeName Direct

msExchAssistantName msExchAssistantName Direct

otherTelephone otherTelephone (multi-valued) Direct

homePhone homePhone Direct

otherHomePhone otherHomePhone (multi-valued) Direct

facsimileTelephoneNumber facsimileTelephoneNumber Direct

Mobile Mobile Direct

telephoneAssistant telephoneAssistant Direct

Pager Pager Direct

Info Info Direct

L (Locality-Name) L (Locality-Name) Direct

MapiRecipient MapiRecipient Rules extension

Initials Initials Direct

—- MSExchOriginatingForest Rules extension

The following logic is applied to the import attribute flow mapping from the source forest user object to the metaverse person object:

1. If a user object has HomeMDB specified, then Microsoft Identity Integration Server 2003 treats it as a mailbox-enabled user, and if does not have it specified, then it is a mail-enabled user.

a. If it is a mailbox-enabled user, the TargetAddress is constructed by matching the value in proxyAddresses with the first value of SMTP mail domain suffixes managed by that forest. If no match is found, Microsoft Identity Integration Server 2003 logs an error.

Page 531: Exchange Server Admin Troubleshooting

56 Module 8: Cross-Forest/Multi-Forest

b. If it is a mail-enabled user and the administrator of the source forest required that all mail to contacts flow through the source forest, then Microsoft Identity Integration Server 2003 constructs a targetAddress based on the matches between proxyAddresses and SMTP mail domain suffixes managed by the source forest, or else Microsoft Identity Integration Server 2003 flows the targetAddress directly. If there are multiple matches, one is selected. If no match is found irrespective of the administrator’s setting, Microsoft Identity Integration Server 2003 flows the targetAddress directly.

2. ProxyAddresses flow directly because Microsoft Identity Integration Server 2003 assumes that there will only be one primary SMTP proxyAddress from the authoritative source object.

3. MapiRecipient: If there is a mailbox associated with the user account, then Microsoft Identity Integration Server 2003 sets the value of the MapiRecipient attribute of the metaverse person object to TRUE, or if it is present, then Microsoft Identity Integration Server 2003 sets the value of the MapiRecipient attribute of the metaverse person object to the same value as the MapiRecipient attribute of the mailbox.

4. MSExchOriginatingForest: This value is generated by using the DN to construct a DNS forest name. Therefore CN=MyName,CN=Users,DC=Fabrikam,DC=com could be translated as ‘Fabrikam.com,’ being the DNS name of the forest and the attribute populated in the metaverse person object attribute and flowed out to the contact in the target forest. The process is as follows:

a. Parse leftwards until the first Domain Congroller

b. Extract out everything after the ‘=’ up to the separator ‘,’

c. Put a dot after the extracted string and check if any other “DC” appears to the left. Return to step a if one does or else return the string.

Page 532: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 57

Import Attribute Flow: Source Forest Contact to Metaverse Contact_Forest

The following table describes the mapping of the Active Directory contact object attributes to the metaverse contact_Forest object attributes.

Source Forest Active Directory Contact Object Attribute

Metaverse contact_forest Attribute

Mapping Type

LegacyExchangeDN LegacyExchangeDN,

Forest_LegacyExchangeDN Rules extension

Mail Mail Direct

MailNickname MailNickname Direct

CN CN Direct

TelephoneNumber TelephoneNumber Direct

targetAddress targetAddress Rules extension

Secretary Secretary Direct

MSExchHideFromAddressLists MSExchHideFromAddressLists Direct

proxyAddresses proxyAddresses Direct

displayName displayName Direct

distinguishedName distinguishedName Direct

Name Name Direct

SN SN Direct

C C Direct

Company Company Direct

Department Department Direct

Division Division Direct

employeeID employeeID Direct

employeeType employeeType Direct

givenName givenName Direct

Manager Manager Direct

O O Direct

Title Title Direct

streetAddress streetAddress Direct

St St Direct

postalCode postalCode Direct

Co Co Direct

physicalDeliveryOfficeName physicalDeliveryOfficeName Direct

msExchAssistantName msExchAssistantName Direct

otherTelephone otherTelephone Direct

Page 533: Exchange Server Admin Troubleshooting

58 Module 8: Cross-Forest/Multi-Forest

homePhone homePhone Direct

otherHomePhone otherHomePhone Direct

facsimileTelephoneNumber facsimileTelephoneNumber Direct

mobile mobile Direct

telephoneAssistant telephoneAssistant Direct

Pager Pager Direct

Info Info Direct

L (Locality-Name) L (Locality-Name) Direct

MapiRecipient MapiRecipient Direct

Initials Initials Direct

--- MSExchOriginatingForest SCRIPTED

The following logic is applied to the import attribute flow mapping from the source forest contact object to the metaverse contact_forest object:

1. If the contact is from the forest specified in the contact_forest object type, then the import attribute flow is generated as follows: the multivalued LegacyExchangeDN attribute of the contact object in Forest1 flows directly to the multivalued Forest1_LegacyExchangeDN attribute of the contact_Forest1 object. This is a direct import attribute flow that is not part of a rules extension.

2. If the contact is not from the forest specified in the contact_forest object type, then the import attribute flow generated is as follows: the multivalued LegacyExchangeDN attribute of the contact object in Forest2 flows directly to the multivalued Forest2_LegacyExchangeDN attribute of the contact_Forest2 object. This is a direct import attribute flow that is not part of a rules extension.

3. If the administrator of the forest specified in contact_forest requires that all mail to contacts flow through the source forest, then Microsoft Identity Integration Server 2003 constructs a targetAddress based on the matches between proxyAddresses and SMTP mail domain suffixes managed by the source forest; otherwise Microsoft Identity Integration Server 2003 flows the targetAddress directly. If there are multiple matches, Microsoft Identity Integration Server 2003 will select one. If no match is found irrespective of the administrator’s setting, then Microsoft Identity Integration Server 2003 should flow targetAddress directly.

4. ProxyAddresses flow directly because Microsoft Identity Integration Server 2003 assumes that there will only be one primary SMTP proxyAddress from the authoritative source object.

5. MSExchOriginatingForest: This value is generated by using the DN to construct a DNS forest name. Therefore: CN=MyName,CN=Users,DC=Fabrikam,DC=com could be translated into ‘Fabrikam.com,’ being the DNS name of the forest and the attribute populated in the metaverse person object attribute, and flowed out to the contact in the target forest. The process would be as follows:

d. Parse leftwards until the first DC

e. Extract out everything after the ‘=’ up to the separator ‘,’

f. Put a dot after the extracted string and check if any other “DC” appears to the left. Return to step a if one does or else return the string.

Page 534: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 59

Import Attribute Flow: Source Forest Group to Metaverse Group The following table describes the mapping of the Active Directory group object attributes to the metaverse group object attributes.

Source Forest Active Directory Group Object Attribute Metaverse Group Object Attribute Mapping Type

LegacyExchangeDN LegacyExchangeDN Direct

Mail Mail Direct

MailNickname MailNickname Direct

CN CN Direct

ProxyAddresses targetAddress Rules extension

MSExchHideFromAddressLists MSExchHideFromAddressLists Direct

ProxyAddresses proxyAddresses Direct

DisplayName DisplayName Direct

MapiRecipient MapiRecipient Direct

—- MSExchOriginatingForest Rules extension

The following logic is applied to the import attribute flow mapping from the source forest group object to the metaverse group object:

7. Microsoft Identity Integration Server 2003 constructs a targetAddress based on the matches between proxyAddresses and the first SMTP mail domain suffix managed by the source forest. If no match is found, Microsoft Identity Integration Server 2003 logs an error.

8. ProxyAddresses flow directly because Microsoft Metadirectory Services 2003 assumes that there will only be one primary SMTP proxyAddress from the authoritative source object.

9. MSExchOriginatingForest: This value is generated by using the DN to construct a DNS forest name. Therefore CN=MyName,CN=Users,DC=Fabrikam,DC=com could be translated into ‘Fabrikam.com,’ being the DNS name of the forest and the attribute populated in the metaverse person object attribute and flowed out to the contact in the target forest. The process is as follows:

a. Parse leftwards until the first DC

b. Extract out everything after the ‘=’ up to the separator ‘,’

c. Put a dot after the extracted string and check if any other “DC” appears to the left. Return to step a if one does or else return the string.

Import Attribute Flow: Target Forest Contact to Metaverse Person The following table describes the mapping of the target forest contact object to the metaverse person object.

Page 535: Exchange Server Admin Troubleshooting

60 Module 8: Cross-Forest/Multi-Forest

Target Forest Contact Object Attribute

Metaverse Person Object Attribute Mapping Type

LegacyExchangeDN Forest_LegacyExchangeD

N Direct.

The value of LegacyExchangeDN of the contact object is assigned from the target forest to the multiple attributes called “Forest”_LegacyExchangeDN.

Import Attribute Flow: Target Forest Contact to Metaverse Group The following table describes the mapping of the target forest contact object to the metaverse group object.

Target Forest Contact Object Attribute

Metaverse Group Object Attribute Mapping Type

LegacyExchangeDN Forest_LegacyExchangeD

N Direct.

The value of LegacyExchangeDN of the contact object is assigned to the multiple attributes called “Forest”_LegacyExchangeDN.

Page 536: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 61

Export Attribute Flow: Metaverse Person to Target Forest Contact The following table describes the mapping of the metaverse person object attributes to the target forest contact object.

Metaverse Person Object Attribute

Target Active Directory Contact Object Attribute

Mapping Type

LegacyExchangeDN Added to ProxyAddresses Rules

extension

Forest_LegacyExchangeDN(s) Not propagated Not propagated

Mail Mail Direct

MailNickname MailNickname Direct

CN CN Direct

TelephoneNumber TelephoneNumber Direct

TargetAddress targetAddress Direct

Secretary Secretary Direct

MSExchMasterAccountSID Not propagated Not propagated

MSExchHideFromAddressLists MSExchHideFromAddressLists Direct

HomeMDB Not propagated Not propagated

ProxyAddresses Added to proxyAddresses Rules extension

displayName displayName Direct

distinguishedName distinguishedName Direct

Name Name Direct

SN SN Direct

C C Direct

Company Company Direct

Department Department Direct

Division Division Direct

employeeID employeeID Direct

employeeType employeeType Direct

givenName givenName Direct

Manager Manager Direct

O O Direct

Title Title Direct

MapiRecipient MAPI_RECIPIENT Direct

UserCertificate UserCertificate Direct

UserSMIMECertificate UserSMIMECertificate Direct

streetAddress streetAddress Direct

St St Direct

Page 537: Exchange Server Admin Troubleshooting

62 Module 8: Cross-Forest/Multi-Forest

postalCode postalCode Direct

Co Co Direct

physicalDeliveryOfficeName physicalDeliveryOfficeName Direct

msExchAssistantName msExchAssistantName Direct

otherTelephone otherTelephone Direct

homePhone homePhone Direct

otherHomePhone otherHomePhone Direct

facsimileTelephoneNumber facsimileTelephoneNumber Direct

mobile mobile Direct

telephoneAssistant telephoneAssistant Direct

Pager Pager Direct

Info Info Direct

L (Locality-Name) L (Locality-Name) Direct

Initials Initials Direct

MSExchOriginatingForest MSExchOriginatingForest Direct

The following logic is applied to the export attribute flow mapping from the metaverse person object to the target forest contact object:

1. The legacyExchangeDN of the contact object is generated and assigned as follows:

a. It must equal the legacyExchangeDN of the msExchAdminGroup object in Active Directory at “CN=Administrative Groups,CN=Exchange Org Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=...”.

b. All separators in the legacyExchangeDN must be lower case (/o=, /ou=, /cn=).

c. It must have a GUID generated and appended to it.

2. The multivalued LegacyExchangeDN of the person object is added to the proxyAddresses of the contact.

3. The primary SMTP proxyAddress of the source object, propogated through the metaverse object, is the one that Microsoft Identity Integration Server 2003 wants to propagate as the primary SMTP on all target objects in all the forests. The proxyAddress attribute is cleared and all the values in the metaverse are flowed to the target value. Therefore, whatever the primary SMTP proxyAddress is on the source, it is flowed from the source object to the metaverse to the target.

Export Attribute Flow: Metaverse Person to Target Forest User The following table describes the mapping of the metaverse person object attributes to the target Active Directory forest user object.

Note In addition to the above, an Exchange administrative group maps to the LegacyExchangeDN target contact object attribute.

Page 538: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 63

Metaverse Person Attribute Target Active Directory User Object Attribute

Mapping Type

Forest_LegacyExchangeDN(s) ProxyAddresses Rules extension

Microsoft Metadirectory Services 2003 ensures that ProxyAddresses of Active Directory user objects contain all the Forest_LegacyExchangeDN attributes, where Forest is not the same as the forest from which the user account is coming.

Export Attribute flow: Metaverse contact_forest to Target Forest Contact

The following table describes the mapping of the metaverse contact_Forest object attributes to the target Active Directory forest contact object.

Metaverse contact_forest Object Attribute

Target Active Directory Contact Object Attribute Mapping Type

LegacyExchangeDN Added to ProxyAddresses Rules extension

Forest_LegacyExchangeDN(s) Added to ProxyAddresses Rules extension

Mail Mail Direct

MailNickname MailNickname Direct

CN CN Direct

TelephoneNumber TelephoneNumber Direct

TargetAddress targetAddress Direct

Secretary Secretary Direct

MSExchHideFromAddressLists MSExchHideFromAddressLists Direct

proxyAddresses proxyAddresses Direct

displayName displayName Direct

distinguishedName distinguishedName Direct

Name Name Direct

SN SN Direct

C C Direct

Company Company Direct

Department Department Direct

Division Division Direct

employeeID employeeID Direct

employeeType employeeType Direct

givenName givenName Direct

Manager Manager Direct

O O Direct

Title Title Direct

MapiRecipient MapiRecipient Direct

StreetAddress streetAddress Direct

Page 539: Exchange Server Admin Troubleshooting

64 Module 8: Cross-Forest/Multi-Forest

St St Direct

PostalCode postalCode Direct

Co Co Direct

physicalDeliveryOfficeName physicalDeliveryOfficeName Direct

msExchAssistantName msExchAssistantName Direct

otherTelephone otherTelephone Direct

homePhone homePhone Direct

otherHomePhone otherHomePhone Direct

facsimileTelephoneNumber facsimileTelephoneNumber Direct

mobile mobile Direct

telephoneAssistant telephoneAssistant Direct

Pager Pager Direct

Info Info Direct

L (Locality-Name) L (Locality-Name) Direct

Initials Initials Direct

MSExchOriginatingForest MSExchOriginatingForest Direct

The following logic is applied to the export attribute flow mapping from the metaverse contact_forest object to the target forest contact object:

1. The legacyExchangeDN of the contact object is generated and assigned as follows:

It must equal the legacyExchangeDN of the msExchAdminGroup object in Active Directory at “CN=Administrative Groups,CN=Exchange Org Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=...”.

All separators in the legacyExchangeDN must be lower case (/o=, /ou=, /cn=).

It must have a GUID generated and appended to it.

2. The values of the multivalued LegacyExchangeDN of the contact_forest object are added to the proxyAddresses attribute of the contact.

3. The primary SMTP proxyAddress of the source object, through the metaverse object, is the one that Microsoft Identity Integration Server 2003 wants to propagate as the primary SMTP on all target objects in all the forests. The proxyAddress attribute is cleared and all the values in the metaverse are flowed to the target value. Therefore, whatever the primary SMTP proxyAddress is on the source, it is flowed from the source object to the metaverse to the target.

4. Ensure that ProxyAddresses of Active Directory contact_forest contain all the Forest_LegacyExchangeDN attributes where Forest is not the same as the forest from which the user account is coming. This is done as part of a rules extension.

If the contact is from the forest specified in the contact_forest object type, then the values of the multivalued LegacyExchangeDN attribute for the contact_forest object flow to the proxyAddresses attribute of the target forest contact object.

If the contact is not from the forest specified in the contact_forest object type, then there is no export attribute flow generated for the Forest(s)_LegacyExchangeDN attribute.

Page 540: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 65

Export Attribute flow: Metaverse Group to Target Forest Group The following table describes the mapping of the metaverse group object attributes to the target Active Directory forest group object attribute.

MV Group Attribute AD Group Attribute Mapping Type Forest_LegacyExchangeDN(s) ProxyAddresses Rules extension

Microsoft Identity Integration Services 2003 ensures that ProxyAddresses of Active Directory user objects contains all the Forest_LegacyExchangeDN attributes where Forest is not the same as the forest from which the user account is coming.

Page 541: Exchange Server Admin Troubleshooting

66 Module 8: Cross-Forest/Multi-Forest

Export Attribute Flow: Metaverse Group to Target Forest Contact

The following table describes the mapping of the metaverse group object attribute to the target Active Directory forest contact object attribute.

Metaverse Group Object Attribute Target Active Directory Contact Object Attribute Mapping Type

(NOT an metaverse attribute) Administrator user interface setting: Exchange Administrative Group

LegacyExchangeDN Rules extension

LegacyExchangeDN Added to ProxyAddresses Rules extension

Mail Mail Direct

MailNickname MailNickname Direct

CN CN Direct

TargetAddress TargetAddress Direct

MSExchHideFromAddressLists MSExchHideFromAddressLists Direct

proxyAddresses ProxyAddresses Rules extension

DisplayName DisplayName Direct

distinguishedName DistinguishedName Direct

MapiRecipient MapiRecipient Direct

MSExchOriginatingForest MSExchOriginatingForest Direct

The following logic is applied to the export attribute flow mapping from the metaverse contact_forest object to the target forest contact object:

1. The LegacyExchangeDN of the contact object is generated and assigned as follows:

a. It must equal the LegacyExchangeDN of the msExchAdminGroup object in Active Directory at “CN=Administrative Groups,CN=Exchange Org Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=...”.

b. All separators in the legacyExchangeDN must be lower case so (/o=, /ou=, /cn=).

c. It must have a GUID generated and appended to it.

2. The values of the multivalued LegacyExchangeDN attribute of the group object are added to the proxyAddresses attribute of the contact.

3. The primary SMTP proxyAddress of the source object is the one that Microsoft Identity Integration Server 2003 wants to propagate as the primary SMTP on all target objects in all the forests. The proxyAddress attribute is cleared and all the values in the metaverse are flowed to the target value. Therefore, whatever the primary SMTP proxyAddress is on the source, it is flowed from the source object to the metaverse to the target.

Page 542: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 67

Appendix C: GAL Sync Provisioning Rules

The changes made to the metaverse during the import of source Active Directory forest data initiate the provision operation, which creates, connects, and disconnects objects in the connector space based on the changes to the metaverse.

There are three objects that are affected by provisioning rules:

User

Group

Contact

For all objects, the provisioning rule depends on the scenario for that object and occurs within the identity integration process shown in Figure 4.3. Some of the rules shown in Figure 4.3 do not apply to all scenarios.

Identity Integration Process

User Object Provisioning Rules The following table lists various scenarios in which a user object is provisioned and the specific provisioning logic that applies to that scenario. The first run profile is a full import with the staging only option and provisioning enabled. The next two run profiles are delta synchronization only and the final run profile is export.

Page 543: Exchange Server Admin Troubleshooting

68 Module 8: Cross-Forest/Multi-Forest

User Object Provisioning Rules

Scenario Provisioning Logic

For a mail-enabled or mailbox-enabled user in the source forest, no corresponding contact exists in the target forest.

Provisioning case 1: If an object is a person and it does not have a connector to any contact, then a connector is added and creates a contact in the target forest in the target Active Directory forest organizational unit with the same name. If a contact with that name already exists, create a contact with the following string concatenation logic: target forest contact -> common name = Person -> common name + Person->department.

For example, if the target forest contact common name is Mike Danseglio, this name is concatenated with the department name Fabrikam, and propagated to the metaverse as Mike Danseglio (Fabrikam).

Mail-enabled or mailbox-enabled user in the source forest and there is a contact in the target forest corresponding to this user in the synchronization OU.

Provisioning case 2: For a metaverse person entry for both Management Agents, if the number of connectors is equal to 1, and the connector for the contact is inside the target forest organizational unit:

• If common name (cn) of connector equals the cn of a metaverse object concatenated with a suffix (the suffix may have been added to avoid name collisions), no provisioning occurs.

• If cn of connector does not equal the cn of a metaverse object concatenated with a suffix, the source object has been renamed. Consequently, the connector must also be renamed.

Mail-enabled or mailbox-enabled user in the source forest and there is a contact in the target forest corresponding to this user who is not in the synchronization OU.

Provisioning case 3: For a metaverse person object for both Management Agents, if the number of connectors is equal to 1 and the distinguished name (DN) of that connector indicates that it is outside the target forest organizational unit, an error is logged to indicate that the contact already exists and must be moved or deleted.

Mail-enabled or mailbox-enabled user in the source forest and there is more than one contact in the target forest corresponding to this user in the synchronization OU.

An error is logged indicating that you must move the incorrect data, contact to the synchronization OU, or delete the data because contacts that are outside the synchronization OU cannot be managed by Microsoft Identity Integration Services 2003.

Provisioning case 4: For a metaverse person object for both Management Agents, if the number of connectors is greater than 1, the following rules are applied to every connector:

• If the connector was created by provisioning (that is, by Microsoft Identity Integration Server 2003), it is disconnected.

• If the connector was created by another process and joined (it may or may not be in the Microsoft Identity Integration Server 2003 target forest organizational unit, no operation is performed and an error is logged describing that there are duplicate contacts, that all except one must be deleted, and that one must be moved into the Microsoft Identity Integration Services 2003 target forest organizational unit to be managed by Microsoft Identity Integration Services 2003.

Page 544: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 69

Mail-enabled or mailbox-enabled user exists and there is more than one contact in the target forest corresponding to this user. The additional contacts may or may not be in the synchronization OU.

An error is logged indicating that you must move the incorrect data, contact to the synchronization OU, or delete the data because contacts that are outside the synchronization OU cannot be managed by Microsoft Identity Integration Server 2003, and it does not allow duplicate contacts in the same forest representing the same source object.

For a metaverse person object for both Management Agreements, if the number of connectors is equal to 1 and the distinguished name (DN) of that connector indicates that it is outside the target forest organizational unit, an error is logged to indicate that the contact already exists and must be moved or deleted.

User is deleted. When a deletion is received, the following process occurs:

1. Connector is disconnected from metaverse object, (connector space and metaverse link tables lose an entry).

2. Disconnector is deleted. (connector space entry table loses an entry).

3. Metaverse deletion is called and the rule determines if it is a master/source object, and deletes the object from the metaverse if is a master.

4. When the object from metaverse is deleted, deprovisioning is called and deletes connector objects for all contact objects linked to that metaverse person object.

5. After export, Contacts are deleted from Active Directory if they are in the synchronization OU.

User is changed. When a change is received, the following process occurs:

1. Connector space filter rules will fail.

2. Connector is disconnected from metaverse object. Connector space and metaverse link tables lose an entry.

3. Disconnector is deleted. Connector space entry tables loses an entry.

4. Metaverse deletion is called and the rule determines if it is a master/source object, and deletes the object from the metaverse if is a master.

5. When object from metaverse is deleted, deprovisioning is called and deletes connectors for all contact objects linked to that metaverse person object.

6. Connector space filter rules succeed.

7. Connector changes.

8. Import attribute flow is reevaluated.

9. Provisioning case 2 succeeds and objects in connector space are updated.

Contact Object Provisioning Rules The following lists various scenarios in which a contact object is provisioned and the specific provisioning logic that applies to that scenario. The first run profile is a full import with the staging only option and provisioning is enabled. The next two run profiles are delta synchronization only and the final run profile is export.

Contact Object Provisioning Rules

Page 545: Exchange Server Admin Troubleshooting

70 Module 8: Cross-Forest/Multi-Forest

Scenario Provisioning Logic

A contact object in the source forest, but there is no contact in the target forest corresponding to this contact; and the contact meets the projection requirements outlined in the projection section.

Provisioning case 1: For the metaverse contact_forest entry for both Management Agents, if there is no connector, add a connector for a contact in the Microsoft Identity Integration Services 2003 target forest organizational unit.

If an object is in the contact_forest and it does not have a connector to any contact, then add a connector to create a contact in the Microsoft Identity Integration Services 2003 target forest organizational unit with the same name. If a contact with that name already exists, create a contact with the following string concatenation logic: target forest Active Directory contact -> common name = contact_forest -> common name + contact_forest->department.

A contact object exists in the source forest, but there is no contact in the target forest corresponding to this contact; and the contact does not meet the projection requirements outlined in the projection section of this document.

If projection rules are not satisfied, an exception is identified and that object is not propagated into any other forest. Also errors are logged.

A contact object in the source forest and there is a contact in the target forest corresponding to this contact in the Microsoft Identity Integration Services 2003 synchronization OU.

Provisioning case 2: For the metaverse contact_forest entry for both Management Agents, if the number of connectors is 1, and the connector for the contact is inside the target forest organizational unit:

• If the cn of a connector equals the cn of any metaverse object concatenated with a suffix (the suffix may have been added to avoid name collisions), no operation is performed.

• If the cn of a connector does not equal the cn of any metaverse object concatenated with a suffix, then the source object has been renamed. Consequently, the connector must also be renamed, and if there is a name collision as a result of this rename, resolve the collision using the same logic as in the first contact object provisioning rule.

A contact object exists and there is a contact in the target forest corresponding to this contact, but not in the synchronization OU.

An error is logged, informing administrators repair the discrepancy and either move incorrect data (contact) to the synchronization OU, or delete it, because contacts that are outside the synchronization OU are not managed by this process.

Provisioning case 3: For the metaverse contact_forest entry for both Management Agents, if the number of connectors is 1 and the DN of that connector indicates that it is outside the Microsoft Identity Integration Services 2003 target forest organizational unit, an error is logged, describing that the contact already exists and must be moved or deleted.

Page 546: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 71

A contact object exists and there is more than one contact in the target forest corresponding to this contact. (The additional contacts may or may not be in the synchronization OU.

An error is logged, informing administrators that they must repair the data and either move the contact data to the synchronization OU or delete it because contacts outside the synchronization OU are not managed and duplicate contacts in the same forest representing the same source object are not allowed.

Provisioning case 4: For the metaverse contact_forest entry for both Management Agents, if the number of connectors is greater than 1, the following rules are applied to every connector:

• If the connector was created by provisioning (i.e. by Microsoft Identity Integration Services 2003), it is disconnected.

• If the connector was created by another process and joined (may or may not be in the Microsoft Identity Integration Services 2003 target forest organizational unit), no operation is performed and an error is logged describing that there are duplicate contacts, and that all except one must be deleted, and that one must be moved into the Microsoft Identity Integration Services 2003 target forest organizational unit to be managed by Microsoft Identity Integration Services 2003.

Contact object outside the synchronization OU, in authoritative contacts container, is deleted.

When the deletion is received, the following process occurs:

1. Connector is disconnected from metaverse object, (connector space and metaverse link tables lose an entry).

2. Disconnector is deleted. (Connector space entry tables lose an entry.)

3. Metaverse deletion is called and the rule checks if it is a master/source object. The contact_forest object will be deleted if the source Active Directory object deleted was in the authoritative contacts OU and if the contact is from the forest in contact_forest.

4. Deprovisioning is called, and because the management agent is not for the forest mentioned in contact_forest, and the connector is for an object in the synchronization OU, then it deprovisions to delete the contact in the Active Directory.

Contact object outside the synchronization OU, or the outside authoritative contacts container, is deleted.

When the deletion is received, the following process occurs:

1. Connector is disconnected from metaverse object, (connector space metaverse link tables lose an entry).

2. Disconnector is deleted. (Connector space entry tables lose an entry.)

3. Metaverse deletion is called and the rule checks if it is a master/source object. The contact_forest object is deleted if the source Active Directory object deleted was in the authoritative contacts OU and if the contact is from forest mentioned in contact_forest. In this scenario, it was not in the authoritative OU, and therefore it is not deleted.

Page 547: Exchange Server Admin Troubleshooting

72 Module 8: Cross-Forest/Multi-Forest

Contact outside the synchronization OU, or in the authoritative contacts OU, is changed.

The following process occurs:

1. Import with provisioning enabled.

2. Connector space filter rules fail.

3. Deletion is received.

4. Connector is disconnected from metaverse object, (connector space and metaverse link tables lose an entry).

5. Disconnector is deleted. (connector space entry tables lose an entry.)

6. Metaverse deletion is called and the rule will check if it is a master/source object. The contact_forest object will be deleted if the source Active Directory object deleted was in the authoritative contacts OU and if the contact is from forest in contact_forest.

7. Deprovisioning is called, and because the management agent is not for the forest mentioned in contact_forest, and the connector is for an object in the synchronization OU, then it is deprovisioned to delete the contact in the Active Directory.

8. Connector space filter rules succeed.

9. Connector changes.

10. Import attribute flow is reevaluated.

Provisioning case 2 succeeds and objects in the connector space are updated.

Contact object is outside the synchronization OU, or the outside authoritative contacts OU is changed.

The following process occurs:

1. Import with provisioning enabled.

2. Connector space filter rules fail.

3. Deletion is received.

4. Connector is disconnected from metaverse object, (connector space and metaverse link tables lose an entry).

5. Disconnector is deleted. (connector space entry tables loses an entry.)

6. Metaverse deletion is called and the rule checks if it is a master/source object. It is not, because it is a contact outside of the authoritative contacts OU, therefore no further processing occurs.

7. Connector space filter rules succeed.

8. Connector changes.

9. Import attribute flow is reevaluated.

10. Provisioning case 3 or 4 succeed.

Contact object in the synchronization OU is deleted (not by Microsoft Identity Integration Services 2003).

The following process occurs:

1. Connector disappears, provisioning is called, and the next export creates the object again.

2. Provisioning case 1 applies.

Contact object in the synchronization OU is created (not by Microsoft Identity Integration Services 2003).

The following process occurs:

1. Joins to something and, when provisioning is called, it will log an event.

2. Provisioning cases 2, 3, or 4 apply.

Page 548: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 73

Contact object is modified in synchronization OU (not by Microsoft Identity Integration Services 2003).

If there is an import attribute flow defined for contact to person, then it will flow back to the source object to which that contact is joined. Nothing on the source object is overwritten, only the LegacyExchangeDN is added to the proxyAddresses of this attribute.

If there is no import attribute flow defined for contact to source object, it is rewritten during the next export so that no synchronized information is flowed back, just data from that forest. Exceptions are LegacyExchangeDN and proxyAddresses, because proxyAddresses of any object should not be overwritten and because legacyExchangeDN is assigned by the target forest.

Group Object Provisioning Rules The following table lists various scenarios in which a group object is provisioned and the specific provisioning logic that applies to that scenario. The first run profile is a full import with the staging only option and provisioning is enabled. The next two run profiles are delta synchronization only and the final run profile is export.

Group Object Provisioning Rules

Scenario Provisioning Logic

Mail enabled or mailbox enabled group object in the source forest and there is no contact in the target forest corresponding to this group.

Provisioning case 1: For the metaverse group entry for both Management Agents, if there is no connector, a connector is added for a contact in the Microsoft Identity Integration Services 2003 target forest organizational unit.

If the object is a group and it does not have a connector to any contact, then a connector is added in the Microsoft Identity Integration Services 2003 target forest organizational unit to create a contact in the target forest with the same name. If a contact with that name already exists, a contact is created with the following string concatenation: target forest AD contact -> common name = Group -> common name + Group+1.

Mail enabled or mailbox enabled group in the source forest and there is a contact in the target forest corresponding to this group in the synchronization OU.

Provisioning case 2: For the metaverse group entry for both Management Agents, if the number of connectors is 1, and the connector for the contact is inside the Microsoft Identity Integration Services 2003 target forest organizational unit:

• If the cn of the connector equals the cn of metaverse object concatenated with a suffix (for potential name collision logic renames), then no operation is performed.

• If the cn of the connector does not equal the cn of metaverse object concatenated with a suffix, then the source object has been renamed. Consequently, the connector must also be renamed and if there is a name collision as a result of this rename, it must be resolved using the same logic as in the first group object provisioning rule.

Mail enabled or mailbox enabled group in the source forest and there is a contact in the target forest corresponding to this group is NOT in the synchronization OU.

An error is logged to inform the administrator to repair this situation and either move the corrupt data/contact to the synchronization OU or delete it because contacts that are outside the synchronization OU are not managed by Microsoft Identity Integration Services 2003.

For the metaverse group entry for both Management Agents, if the number of connectors is 1, and the DN of that connector indicates that it is outside the Microsoft Identity Integration Services 2003 target forest organizational unit, an error is logged that describes that the contact already exists and must be moved or deleted.

Page 549: Exchange Server Admin Troubleshooting

74 Module 8: Cross-Forest/Multi-Forest

Mail enabled or mailbox enabled group in the source forest and there is more than one contact in the target forest corresponding to this group. (The additional contacts may or may not be in the synchronization OU.)

An error is logged to inform the administrator to repair this situation and either move the corrupt data/contact to the synchronization OU or delete it because contacts outside the synchronization OU are not managed and duplicate contacts in the same forest representing the same source object are not allowed.

Provisioning case 4: For the metaverse person entry for both Management Agents, if the number of connectors is greater than 1, then for every connector:

• If the connector was created by provisioning (that is, by Microsoft Identity Integration Services 2003), it is disconnected.

• If the connector was created by another process and joined (may or may not be in the Microsoft Identity Integration Services 2003 target forest organizational unit), no operation is performed and an error is logged, describing that there are duplicate contacts, and that all except one must be deleted, and that one must be moved into the Microsoft Identity Integration Services 2003 target forest organizational unit to be managed by Microsoft Identity Integration Services 2003.

Group object is deleted. When a deletion is received, the following process occurs:

1. Connector is disconnected from metaverse object, (connector space and metaverse link tables lose an entry).

2. Disconnector is deleted. (connector space entry tables lose an entry.)

3. Metaverse deletion is called and the rule checks if it is a master/source object, deletes the object from the metaverse if it is a master.

4. When object from metaverse is deleted, deprovisioning is called and deletes connectors for all contact objects linked to that metaverse group object.

Group object is changed.

When a change is received, the following process occurs:

1. Connector space filter rules fail.

2. Connector is disconnected from metaverse object, (connector space and metaverse link tables lose an entry).

3. Disconnector is deleted. (connector space entry table loses an entry).

4. Metaverse deletion is called and the rule checks if it is a master/source object, then deletes the object from the metaverse if is a master.

5. When object from metaverse is deleted, deprovisioning is called and deletes connectors for all contact objects linked to that metaverse group object.

6. Connector space filter rules succeed.

7. Connector changes.

8. Import attribute flow is reevaluated.

Provisioning case 2 succeeds and objects in connector space are updated.

Metaverse Object Deletion Rules The next table lists the three metaverse object deletion rules extensions.

Page 550: Exchange Server Admin Troubleshooting

Module 8: Cross-Forest/Multi-Forest 75

Metaverse Deletion Rules

Rule Description

User If the object that was deleted is a person, then it is deleted from the metaverse.

Contact If the Management Agent where the connector disappeared is from contact_forest, then it is deleted.

Group If the object that was deleted is a group, then it is deleted from the metaverse.

Deprovisioning Rules The next table lists the three deprovisioning rules.

TDeprovisioning Rules

Rule Description

User When the metaverse person object is deleted, the contact object is deleted if it is in the synchronization organizational unit. If not, it is logged.

Contact When the metaverse contact_forest object is deleted, the contact object is deleted if it is in the synchronization organizational unit. If not, it is logged.

Group When the metaverse group object is deleted, the contact object is deleted if it is in the synchronization organizational unit. If not, it is logged.

Page 551: Exchange Server Admin Troubleshooting

76 Module 8: Cross-Forest/Multi-Forest

Acknowledgments Microsoft Employee

Vincent Yim

External Websites:

http://www.microsoft.com/windowsserver2003/technologies/directory/miis/default.mspx