Set of files seems unable to replicate between two DFS member serv

Discussion:

(too old to reply)

Marc

2009-01-02 01:48:23 UTC

Hello,

I've set up a DFS between two servers, pre-staged data on the second server,
and set up DFS replication between the two servers. Now after replication
has completed the second (non-primary) server has a backlog of a couple
thousand files which won't seem to go away, and wont replicate on just one of
several replicated folders (albeit the largest one). Other files created on
the second server do replicate, but after quite a delay. Files from the
first server to the second replicate with no delay.

I've found the below errors in the debug logs which seem to relate to the
files in question, but can't find much guidance on the internet about how to
correct them. If anyone has any suggestions they would be most appreciated:

On the "primary" server, which has no backlog to the secondary these
messages appear for all of the backlogged files from the secondary:

20090101 19:29:03.511 2684 MEET 1207 Meet::Install Retries:0
updateName:05-08 uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT updateType:missing
parent
20090101 19:29:03.511 2684 INCO 5610 InConnection::LogTransferActivity
Received RAWGET uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673 fileName:05-08
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3} stagedSize:368
20090101 19:29:03.511 2684 ASYN 510 AsyncUnbufferedFileWriter::Close Async
WRITE Statistics:
20090101 19:29:03.511 2684 STAG 799 StageWriter::CompleteDownloadStage
Completed download or stage file
1603673-{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673-{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673-Downloaded.frx
20090101 19:29:03.511 2684 MEET 2032 Meet::Download Download Succeeded :
true updateName:05-08 uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
20090101 19:29:03.511 2684 ASYN 1291 AsyncUnbufferedFileReader::Close Async
READ Statistics:
20090101 19:29:03.511 2684 MEET 4092 Meet::GetParent -> BLOCKED Could not
find parent. parentUid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
updateName:05-08 uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
20090101 19:29:35.523 2684 UPMG 418 UpdateWorker::ConsumeUpdates No
pending updates. connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
20090101 19:29:35.523 1288 MEET 1207 Meet::Install Retries:6
updateName:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
gvsn:{00000000-0000-0000-0000-000000000000}-v0
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT updateType:missing
parent
20090101 19:29:35.523 1288 DOWN 5186 [ERROR] DownstreamTransport::RdcGet
Failed on connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3} rgName:bopnet.local\files\dept
update:

the "Meet::GetParent -> BLOCKED Could not find parent." messages seem most
telling and I couldn't find anything about how to correct an error of this
type.

On the "second" server, which does have the backlog to the first I see:

20090101 19:13:32.393 3152 OUTC 784 OutConnection::OpenFile Received
request for update:
+ present 1
+ nameConflict 0
+ attributes 0x10
+ ghostedHeader 0
+ data 0
+ gvsn
{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
+ uid
{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
+ parent
{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
+ fence 16010101 00:00:00.000 (I)
+ clockDecrementedInDirtyShutdown 0
+ clock 20081209 17:44:52.317 GMT
(0x1c95a25d6f0e449)
+ createTime 20081209 04:04:45.170 GMT
+ csId {DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
+ hash 3CE4FAA8-104E9066-9FB75CF8-864E5F41
+ similarity 01163A31-07152A00-0F351C30-1634390A
+ name 05-08
+ rdcDesired:1 connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
rgName:bopnet.local\files\dept
20090101 19:13:32.393 2712 OUTC 1534 OutConnection::OpenFile Sent file
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674 name:06-08 fileSize:372
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} rgName:bopnet.local\files\dept
20090101 19:13:32.393 2712 MRSH 4615 Marshaller::Marshal FileAttrs in
metadata : 0x10
20090101 19:13:32.393 2712 SRTR 2357
InitializeFileTransferAsyncState::ProcessIoCompletion Initialized
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} rdc:1
context:0000000000000000,0000000000000000,0000000000000000
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674
gvsn{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674 ptr:0000000002D510D0
20090101 19:13:32.393 3152 OUTC 1534 OutConnection::OpenFile Sent file
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673 name:05-08 fileSize:372
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} rgName:bopnet.local\files\dept
20090101 19:13:32.393 3152 MRSH 4615 Marshaller::Marshal FileAttrs in
metadata : 0x10
20090101 19:13:32.393 3152 SRTR 2357
InitializeFileTransferAsyncState::ProcessIoCompletion Initialized
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} rdc:1
context:0000000000000000,0000000000000000,0000000000000000
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673 ptr:000000000D838140
20090101 19:13:56.121 3508 SRTR 784 [WARN] SERVER_EstablishSession Failed
on connId:{75CA278C-01B4-443B-848B-445C70765BFB}
csId:{3B02FF3D-F84B-47BC-B6DF-16DA6441709C} Error:
+ [Error:9051(0x235b) UpstreamTransport::EstablishSession
upstreamtransport.cpp:707 3508 C The content set is not ready]
+ [Error:9051(0x235b) OutConnection::EstablishSession outconnection.cpp:2623
3508 C The content set is not ready]
20090101 19:15:56.133 3508 SRTR 784 [WARN] SERVER_EstablishSession Failed
on connId:{75CA278C-01B4-443B-848B-445C70765BFB}
csId:{3B02FF3D-F84B-47BC-B6DF-16DA6441709C} Error:
+ [Error:9051(0x235b) UpstreamTransport::EstablishSession
upstreamtransport.cpp:707 3508 C The content set is not ready]
+ [Error:9051(0x235b) OutConnection::EstablishSession outconnection.cpp:2623
3508 C The content set is not ready]

I know the "Content set is not ready" messages relate to initial
replication, but initial replication completed successfully and querying WMI
on both systems states that everything is normal.

Is there anything else I should check or try to do to get this backlog
cleared between the two servers and bi-directional replication working with
the speed it is supposed to?

Marc

2009-01-02 01:50:18 UTC

Permalink

Also, both servers are running Windows Server 2008.

Mark Parris [ADUG]

2009-01-05 22:49:31 UTC

Permalink

What are the file types?

a.. NTFS mounted drives within the local path of the replicated folder.
(However, the local path of a replicated folder can be at or under a mounted
drive.)

a.. Files encrypted by using encrypting file system (EFS).

a.. Any reparse points except those associated with DFS Namespaces. If a
file has a reparse point used for Hierarchical Storage Management (HSM) or
Single Instance Store (SIS), DFS Replication replicates the underlying file
but not the reparse point.

a.. Files on which the temporary attribute has been set.

a.. Files with case-sensitive names created by using a UNIX-compatible
application and saved to a Network File System (NFS) shared folder. (NTFS
supports the Portable Operating System Interface [POSIX] standard, which
provides the ability to create two files that have the same name but
different capitalization.) For example, if a user creates two files,
FILE.doc and File.doc, on an NFS shared folder, DFS Replication will
replicate the first but not the second.
--
[ADUG] UK Active Active Directory User Group
http://adug.co.uk

Post by Marc
Hello,
I've set up a DFS between two servers, pre-staged data on the second server,
and set up DFS replication between the two servers. Now after replication
has completed the second (non-primary) server has a backlog of a couple
thousand files which won't seem to go away, and wont replicate on just one of
several replicated folders (albeit the largest one). Other files created on
the second server do replicate, but after quite a delay. Files from the
first server to the second replicate with no delay.
I've found the below errors in the debug logs which seem to relate to the
files in question, but can't find much guidance on the internet about how to
On the "primary" server, which has no backlog to the secondary these
20090101 19:29:03.511 2684 MEET 1207 Meet::Install Retries:0
updateName:05-08 uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
updateType:missing
parent
20090101 19:29:03.511 2684 INCO 5610 InConnection::LogTransferActivity
Received RAWGET uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673 fileName:05-08
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3} stagedSize:368
20090101 19:29:03.511 2684 ASYN 510 AsyncUnbufferedFileWriter::Close Async
20090101 19:29:03.511 2684 STAG 799 StageWriter::CompleteDownloadStage
Completed download or stage file
1603673-{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673-{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673-Downloaded.frx
true updateName:05-08 uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
20090101 19:29:03.511 2684 ASYN 1291 AsyncUnbufferedFileReader::Close Async
20090101 19:29:03.511 2684 MEET 4092 Meet::GetParent -> BLOCKED Could not
find parent. parentUid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
updateName:05-08 uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
20090101 19:29:35.523 2684 UPMG 418 UpdateWorker::ConsumeUpdates No
pending updates. connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
20090101 19:29:35.523 1288 MEET 1207 Meet::Install Retries:6
updateName:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
gvsn:{00000000-0000-0000-0000-000000000000}-v0
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} csName:DEPT
updateType:missing
parent
20090101 19:29:35.523 1288 DOWN 5186 [ERROR] DownstreamTransport::RdcGet
Failed on connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
csId:{DB66BBF6-0B60-4EDB-A697-6E102299E6C3} rgName:bopnet.local\files\dept
the "Meet::GetParent -> BLOCKED Could not find parent." messages seem most
telling and I couldn't find anything about how to correct an error of this
type.
20090101 19:13:32.393 3152 OUTC 784 OutConnection::OpenFile Received
+ present 1
+ nameConflict 0
+ attributes 0x10
+ ghostedHeader 0
+ data 0
+ gvsn
{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
+ uid
{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
+ parent
{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1602371
+ fence 16010101 00:00:00.000 (I)
+ clockDecrementedInDirtyShutdown 0
+ clock 20081209 17:44:52.317 GMT
(0x1c95a25d6f0e449)
+ createTime 20081209 04:04:45.170 GMT
+ csId {DB66BBF6-0B60-4EDB-A697-6E102299E6C3}
+ hash 3CE4FAA8-104E9066-9FB75CF8-864E5F41
+ similarity 01163A31-07152A00-0F351C30-1634390A
+ name 05-08
+ rdcDesired:1 connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
rgName:bopnet.local\files\dept
20090101 19:13:32.393 2712 OUTC 1534 OutConnection::OpenFile Sent file
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674 name:06-08 fileSize:372
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
rgName:bopnet.local\files\dept
20090101 19:13:32.393 2712 MRSH 4615 Marshaller::Marshal FileAttrs in
metadata : 0x10
20090101 19:13:32.393 2712 SRTR 2357
InitializeFileTransferAsyncState::ProcessIoCompletion Initialized
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} rdc:1
context:0000000000000000,0000000000000000,0000000000000000
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674
gvsn{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603674 ptr:0000000002D510D0
20090101 19:13:32.393 3152 OUTC 1534 OutConnection::OpenFile Sent file
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673 name:05-08 fileSize:372
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2}
rgName:bopnet.local\files\dept
20090101 19:13:32.393 3152 MRSH 4615 Marshaller::Marshal FileAttrs in
metadata : 0x10
20090101 19:13:32.393 3152 SRTR 2357
InitializeFileTransferAsyncState::ProcessIoCompletion Initialized
connId:{9472AC6C-5C29-41DD-A2A2-DEDBA37BF2E2} rdc:1
context:0000000000000000,0000000000000000,0000000000000000
uid:{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673
gvsn{167B913C-FAB5-425A-A0E2-CCE2DE10B292}-v1603673 ptr:000000000D838140
20090101 19:13:56.121 3508 SRTR 784 [WARN] SERVER_EstablishSession Failed
on connId:{75CA278C-01B4-443B-848B-445C70765BFB}
+ [Error:9051(0x235b) UpstreamTransport::EstablishSession
upstreamtransport.cpp:707 3508 C The content set is not ready]
+ [Error:9051(0x235b) OutConnection::EstablishSession
outconnection.cpp:2623
3508 C The content set is not ready]
20090101 19:15:56.133 3508 SRTR 784 [WARN] SERVER_EstablishSession Failed
on connId:{75CA278C-01B4-443B-848B-445C70765BFB}
+ [Error:9051(0x235b) UpstreamTransport::EstablishSession
upstreamtransport.cpp:707 3508 C The content set is not ready]
+ [Error:9051(0x235b) OutConnection::EstablishSession
outconnection.cpp:2623
3508 C The content set is not ready]
I know the "Content set is not ready" messages relate to initial
replication, but initial replication completed successfully and querying WMI
on both systems states that everything is normal.
Is there anything else I should check or try to do to get this backlog
cleared between the two servers and bi-directional replication working with
the speed it is supposed to?

Marc

2009-01-08 21:44:27 UTC

Permalink

All the files are regular file. Mostly JPEGs and the like.

I've looked more at the DFSR database as queried through WMI, and discovered
something interesting.

The files that won't replicate have two entries in the DfsrIdRecordInfo
table, one with a DatabaseGuid in the UID that indicates it was modified on
the secondary member, and another indicating that it was modified on the
primary. The files that won't replicate have a Fence value of 1 (initial
sync secondary). Presumably these second files don't actually exist, and
thus can't replicate (the member that can't replicate them does have errors
stating "The file meta data is not synchronized with the file system" in the
dfsr log). I haven't quite figured out how to query the NTFS volume by file
ID to see if the file ids in the dfsr database exist on the file system or
not, but I suspect that they do not.

So what I appear to have is a dfsr database with some erroneous data in it,
but don't know what the proper way to have it repair itself is.

-Marc

Post by Mark Parris [ADUG]
What are the file types?
a.. NTFS mounted drives within the local path of the replicated folder.
(However, the local path of a replicated folder can be at or under a mounted
drive.)
a.. Files encrypted by using encrypting file system (EFS).
a.. Any reparse points except those associated with DFS Namespaces. If a
file has a reparse point used for Hierarchical Storage Management (HSM) or
Single Instance Store (SIS), DFS Replication replicates the underlying file
but not the reparse point.
a.. Files on which the temporary attribute has been set.
a.. Files with case-sensitive names created by using a UNIX-compatible
application and saved to a Network File System (NFS) shared folder. (NTFS
supports the Portable Operating System Interface [POSIX] standard, which
provides the ability to create two files that have the same name but
different capitalization.) For example, if a user creates two files,
FILE.doc and File.doc, on an NFS shared folder, DFS Replication will
replicate the first but not the second.
--
[ADUG] UK Active Active Directory User Group
http://adug.co.uk