Issue : getting event id 1988 on the server

Issue :  getting event id 1988 on the server  LONSRV-IIS-01 & on EDISRV-DC-01
Scope : To fix the AD replication between  LONSRV-IIS-01 & on EDISRV-DC-01


You have 2 Sites with 2 DCs in a particular site and 1 DC in a remote site.
+ The site had VPN link between them.

+ initially you had following site structure

I Ste -- London:
LONSRV-DC-01 - All FSMO Roles
LONSRV-DC-02

II Site -- Endinbrough:
EDISRV-DC-01

+ LONSRV-DC-02 was decommisioned & installed LONSRV-IIS-01 about a month ago

+OS on LONSRV-DC-01 -- Windows Server 2003 Standard SP2, x64 , IP address : 172.22.242.5
+OS on LONSRV-IIS-01 -- Windows Server 2003 Standard SP2, x86 , IP address : 172.22.242.94
+OS on EDISRV-DC-01 -- Windows Server 2003 Standard SP2, x86 , IP address : 192.168.0.3




+ getting event id 1864 & 1988 on LONSRV-IIS-01 & on EDISRV-DC-01

C:\Program Files\Support Tools>netdom query dc
List of domain controllers with accounts in the domain:

LONSRV-DC-01
EDISRV-DC-01
LONSRV-IIS-01
The command completed successfully.


C:\Program Files\Support Tools>netdom query fsmo
Schema owner                LONSRV-DC-01.eurowelcome.co.uk

Domain role owner           LONSRV-DC-01.eurowelcome.co.uk

PDC role                    LONSRV-DC-01.eurowelcome.co.uk

RID pool manager            LONSRV-DC-01.eurowelcome.co.uk

Infrastructure owner        LONSRV-DC-01.eurowelcome.co.uk

The command completed successfully.


C:\Documents and Settings\Administrator.EUROWELCOME>net config rdr
Computer name                        \\LONSRV-IIS-01
Full Computer name                   LONSRV-IIS-01.eurowelcome.co.uk
User name                            Administrator

Workstation active on
NetbiosSmb (000000000000)
NetBT_Tcpip_{EFE55030-413F-441A-BB94-4E87C93C2719} (00155DF20B03)

Software version                     Microsoft Windows Server 2003

Workstation domain                   EUROWELCOME
Workstation Domain DNS Name          eurowelcome.co.uk
Logon domain                         EUROWELCOME

COM Open Timeout (sec)               0
COM Send Count (byte)                16
COM Send Timeout (msec)              250
The command completed successfully.


C:\Documents and Settings\Administrator.EUROWELCOME>ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : LONSRV-IIS-01
Primary Dns Suffix  . . . . . . . : eurowelcome.co.uk
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : eurowelcome.co.uk
co.uk

Ethernet adapter Local Area Connection 3:

Connection-specific DNS Suffix  . :
Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Ada
pter #2
Physical Address. . . . . . . . . : 00-15-5D-F2-0B-03
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 172.22.242.94
Subnet Mask . . . . . . . . . . . : 255.255.254.0
Default Gateway . . . . . . . . . : 172.22.242.9
DNS Servers . . . . . . . . . . . : 172.22.242.94
172.22.242.5


+ getting following:

Event Type:        Error
Event Source:    NTDS Replication
Event Category:                Replication
Event ID:              1988
Date:                     23/07/2010
Time:                     09:28:28
User:                     NT AUTHORITY\ANONYMOUS LOGON
Computer:          LONSRV-IIS-01
Description:
Active Directory Replication encountered the existence of objects in the following partition that have been deleted from the local domain controllers (DCs) Active Directory database.  Not all direct or transitive replication partners replicated in the deletion before the tombstone lifetime number of days passed.  Objects that have been deleted and garbage collected from an Active Directory partition but still exist in the writable partitions of other DCs in the same domain, or read-only partitions of global catalog servers in other domains in the forest are known as "lingering objects".

This event is being logged because the source DC contains a lingering object which does not exist on the local DCs Active Directory database.  This replication attempt has been blocked.

The best solution to this problem is to identify and remove all lingering objects in the forest.


Source DC (Transport-specific network address):
8d62209c-f446-4cfd-9ac3-27bfd64f9809._msdcs.eurowelcome.co.uk
Object:
CN=EWEDI08-Canon i250\0ACNF:46afd7b3-7f31-4252-8fd8-ea5a8277bef7,CN=EWEDI08,OU=Edinburgh Computers,OU=All Computers,DC=eurowelcome,DC=co,DC=uk
Object GUID:
46afd7b3-7f31-4252-8fd8-ea5a8277bef7

User Action:

Remove Lingering Objects:

The action plan to recover from this error can be found at http://support.microsoft.com/?id=314282.

If both the source and destination DCs are Windows Server 2003 DCs, then install the support tools included on the installation CD.  To see which objects would be deleted without actually performing the deletion run "repadmin /removelingeringobjects <Source DC> <Destination DC DSA GUID> <NC> /ADVISORY_MODE". The eventlogs on the source DC will enumerate all lingering objects.  To remove lingering objects from a source domain controller run "repadmin /removelingeringobjects <Source DC> <Destination DC DSA GUID> <NC>".

If either source or destination DC is a Windows 2000 Server DC, then more information on how to remove lingering objects on the source DC can be found at http://support.microsoft.com/?id=314282 or from your Microsoft support personnel.

If you need Active Directory replication to function immediately at all costs and don't have time to remove lingering objects, enable loose replication consistency by unsetting the following registry key:

Registry Key:
HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replication Consistency

Replication errors between DCs sharing a common partition can prevent user and compter acounts, trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data to vary between DCs, affecting the ability to log on, find objects of interest and perform other critical operations. These inconsistencies are resolved once replication errors are resolved.  DCs that fail to inbound replicate deleted objects within tombstone lifetime number of days will remain inconsistent until lingering objects are manually removed by an administrator from each local DC.

Lingering objects may be prevented by ensuring that all domain controllers in the forest are running Active Directory, are connected by a spanning tree connection topology and perform inbound replication before Tombstone Live number of days pass.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

***************** Also getting**************


Event Type:        Error
Event Source:    NTDS Replication
Event Category:                Replication
Event ID:              1864
Date:                     23/07/2010
Time:                     09:27:48
User:                     NT AUTHORITY\ANONYMOUS LOGON
Computer:          LONSRV-IIS-01
Description:
This is the replication status for the following directory partition on the local domain controller.

Directory partition:
DC=eurowelcome,DC=co,DC=uk

The local domain controller has not recently received replication information from a number of domain controllers.   The count of domain controllers is shown, divided into the following intervals.

More than 24 hours:
2
More than a week:
2
More than one month:
0
More than two months:
0
More than a tombstone lifetime:
0
Tombstone lifetime (days):
60
Domain controllers that do not replicate in a timely manner may encounter errors. It may miss password changes and be unable to authenticate. A DC that has not replicated in a tombstone lifetime may have missed the deletion of some objects, and may be automatically blocked from future replication until it is reconciled.

To identify the domain controllers by name, install the support tools included on the installation  CD and run dcdiag.exe.
You can also use the support tool repadmin.exe to display the replication latencies of the domain controllers in the forest.   The command is "repadmin /showvector /latency <partition-dn>".

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


+ ping 8d62209c-f446-4cfd-9ac3-27bfd64f9809._msdcs.eurowelcome.co.uk

Pinging lonsrv-dc-01.eurowelcome.co.uk [172.22.242.5] with 32 bytes of data:

Reply from 172.22.242.5: bytes=32 time=1ms TTL=128
Reply from 172.22.242.5: bytes=32 time<1ms TTL=128
Reply from 172.22.242.5: bytes=32 time<1ms TTL=128
Reply from 172.22.242.5: bytes=32 time<1ms TTL=128

Ping statistics for 172.22.242.5:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms



+ ALSO GETTING EVENT ID 1988 ON THE SERVER EDISRV-DC-01


Event Type:        Error
Event Source:    NTDS Replication
Event Category:                Replication
Event ID:              1988
Date:                     30/06/2010
Time:                     15:16:10
User:                     NT AUTHORITY\ANONYMOUS LOGON
Computer:          EDISRV-DC-01
Description:
Active Directory Replication encountered the existence of objects in the following partition that have been deleted from the local domain controllers (DCs) Active Directory database.  Not all direct or transitive replication partners replicated in the deletion before the tombstone lifetime number of days passed.  Objects that have been deleted and garbage collected from an Active Directory partition but still exist in the writable partitions of other DCs in the same domain, or read-only partitions of global catalog servers in other domains in the forest are known as "lingering objects".

This event is being logged because the source DC contains a lingering object which does not exist on the local DCs Active Directory database.  This replication attempt has been blocked.

The best solution to this problem is to identify and remove all lingering objects in the forest.


Source DC (Transport-specific network address):
8d62209c-f446-4cfd-9ac3-27bfd64f9809._msdcs.eurowelcome.co.uk
Object:
CN=EWEDI08-Canon i250\0ACNF:46afd7b3-7f31-4252-8fd8-ea5a8277bef7,CN=EWEDI08,OU=Edinburgh Computers,OU=All Computers,DC=eurowelcome,DC=co,DC=uk
Object GUID:
46afd7b3-7f31-4252-8fd8-ea5a8277bef7

User Action:

Remove Lingering Objects:

The action plan to recover from this error can be found at http://support.microsoft.com/?id=314282.

If both the source and destination DCs are Windows Server 2003 DCs, then install the support tools included on the installation CD.  To see which objects would be deleted without actually performing the deletion run "repadmin /removelingeringobjects <Source DC> <Destination DC DSA GUID> <NC> /ADVISORY_MODE". The eventlogs on the source DC will enumerate all lingering objects.  To remove lingering objects from a source domain controller run "repadmin /removelingeringobjects <Source DC> <Destination DC DSA GUID> <NC>".

If either source or destination DC is a Windows 2000 Server DC, then more information on how to remove lingering objects on the source DC can be found at http://support.microsoft.com/?id=314282 or from your Microsoft support personnel.

If you need Active Directory replication to function immediately at all costs and don't have time to remove lingering objects, enable loose replication consistency by unsetting the following registry key:

Registry Key:
HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replication Consistency

Replication errors between DCs sharing a common partition can prevent user and compter acounts, trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data to vary between DCs, affecting the ability to log on, find objects of interest and perform other critical operations. These inconsistencies are resolved once replication errors are resolved.  DCs that fail to inbound replicate deleted objects within tombstone lifetime number of days will remain inconsistent until lingering objects are manually removed by an administrator from each local DC.

Lingering objects may be prevented by ensuring that all domain controllers in the forest are running Active Directory, are connected by a spanning tree connection topology and perform inbound replication before Tombstone Live number of days pass.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


+ followed http://technet.microsoft.com/en-us/library/cc785298(WS.10).aspx -- Use Repadmin to remove lingering objects
+ ran following on the server LONSRV-IIS-01

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "CN=Schema,CN=Configuration,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "DC=ForestDnsZones,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "DC=DomainDnsZones,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "CN=Configuration,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode


repadmin /removelingeringobjects lonsrv-dc-01 ec051db8-b14d-47db-b063-547745b3ad6a "CN=Schema,CN=Configuration,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 ec051db8-b14d-47db-b063-547745b3ad6a "DC=ForestDnsZones,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 ec051db8-b14d-47db-b063-547745b3ad6a "DC=DomainDnsZones,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 ec051db8-b14d-47db-b063-547745b3ad6a "CN=Configuration,DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

repadmin /removelingeringobjects lonsrv-dc-01 ec051db8-b14d-47db-b063-547745b3ad6a "DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode

+ The output of Advisory mode is as below:


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 2ab
142b5-a557-4d26-a2d5-e41f2a959efb "CN=Schema,CN=Configuration,DC=eurowelcome,DC=
co,DC=uk" /Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 2ab
142b5-a557-4d26-a2d5-e41f2a959efb "DC=ForestDnsZones,DC=eurowelcome,DC=co,DC=uk"
/Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 2ab
142b5-a557-4d26-a2d5-e41f2a959efb "DC=DomainDnsZones,DC=eurowelcome,DC=co,DC=uk"
/Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 2ab
142b5-a557-4d26-a2d5-e41f2a959efb "CN=Configuration,DC=eurowelcome,DC=co,DC=uk"
/Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 2ab
142b5-a557-4d26-a2d5-e41f2a959efb "DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 ec0
51db8-b14d-47db-b063-547745b3ad6a "CN=Schema,CN=Configuration,DC=eurowelcome,DC=
co,DC=uk" /Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 ec0
51db8-b14d-47db-b063-547745b3ad6a "DC=ForestDnsZones,DC=eurowelcome,DC=co,DC=uk"
/Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 ec0
51db8-b14d-47db-b063-547745b3ad6a "DC=DomainDnsZones,DC=eurowelcome,DC=co,DC=uk"
/Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 ec0
51db8-b14d-47db-b063-547745b3ad6a "CN=Configuration,DC=eurowelcome,DC=co,DC=uk"
/Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


C:\Program Files\Support Tools>repadmin /removelingeringobjects lonsrv-dc-01 ec0
51db8-b14d-47db-b063-547745b3ad6a "DC=eurowelcome,DC=co,DC=uk" /Advisory_Mode
RemoveLingeringObjects sucessfull on lonsrv-dc-01.


+ went to server lonsrv-dc-01
+ open adsiedit.msc
+ went to CN=EWEDI08,OU=Edinburgh Computers,OU=All Computers,DC=eurowelcome,DC=co,DC=uk
+ deleted the CNF -- CN=EWEDI08-Canon i250\0ACNF:46afd7b3-7f31-4252-8fd8-ea5a8277bef7
+ checked AD replication & it gives following error:

---------------------------
Replicate Now
---------------------------
The following error occurred during the attempt to synchronize naming context eurowelcome.co.uk from domain controller LONSRV-DC-01 to domain controller LONSRV-IIS-01: Insufficient attributes were given to create an object.  This object may not exist because it may have been deleted and already garbage collected.  This operation will not continue.
---------------------------
OK  
---------------------------

+ ran repadmin /syncall /eAPd > rep.txx & rep.txt
the error is:
SyncAll reported the following errors: Error issuing replication: 8606 (0x219e):     Can't retrieve message string 8606 (0x219e), error 1815.     From: CN=NTDS Settings,CN=LONSRV-DC-01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=eurowelcome,DC=co,DC=uk     To  : CN=NTDS Settings,CN=LONSRV-IIS-01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=eurowelcome,DC=co,DC=uk
+ ran repadmin /showrepl * > a.txt & a.txt
the error is :
Source: Default-First-Site-Name\LONSRV-DC-01 ******* 37172 CONSECUTIVE FAILURES since 2010-06-30 13:13:38 Last error: 8606 (0x219e):             Can't retrieve message string 8606 (0x219e), error 1815.

+ checked event viewer on LONSRV-DC-01 & noted following event 1942, 1946, 1938

Event Type:        Information
Event Source:    NTDS Replication
Event Category:                Replication
Event ID:              1942
Date:                     23/07/2010
Time:                     13:12:26
User:                     EUROWELCOME\administrator
Computer:          LONSRV-DC-01
Description:
Active Directory has completed the verification of lingering objects on the local domain controller in advisory mode. All objects on this domain controller have had their existence verified on the following source domain controller. 

Source domain controller:
2ab142b5-a557-4d26-a2d5-e41f2a959efb._msdcs.eurowelcome.co.uk
Number of objects examined and verified:
1 

Objects that have been deleted and garbage collected on the source domain controller yet still exist on this domain controller have been listed in past event log entries. To permanently delete the lingering objects, restart this procedure without using the advisory mode option.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

+ ran following command on the server  LONSRV-IIS-01

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "CN=Configuration,DC=eurowelcome,DC=co,DC=uk"

repadmin /removelingeringobjects lonsrv-dc-01 2ab142b5-a557-4d26-a2d5-e41f2a959efb "DC=eurowelcome,DC=co,DC=uk"

+ got following event 1939 on lonsrv-dc-01
Event Type:        Information
Event Source:    NTDS Replication
Event Category:                Replication
Event ID:              1939
Date:                     23/07/2010
Time:                     13:15:09
User:                     EUROWELCOME\administrator
Computer:          LONSRV-DC-01
Description:
Active Directory has completed the removal of lingering objects on the local domain controller. All objects on this domain controller have had their existence verified on the following source domain controller.

Source domain controller:
2ab142b5-a557-4d26-a2d5-e41f2a959efb._msdcs.eurowelcome.co.uk 
Number of objects deleted:
1 

Objects that were deleted and garbage collected on the source domain controller yet existed on the local domain controller were deleted from the local domain controller. Past event log entries list these deleted objects.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

+ ran repadmin /syncall /eAPd – success,  Issue resolved